SJCIT Jun 2019

SVD Toy Examples

1. Introduction.
We can view a matrix A mxn (m rows and n columns) as a tranformation from
Row space (nxn) into Column space (mxm) or viceversa . This is illustrated in the figure below.

mxm col space

A mrowxncol =A*At

nxn
row space
=At*A

We can go from rowspace into col space with A and return back into row space with At*A.
So At A describes the row space with the col space effects. Similarly A*At describes
the col space with row effects.

SVD is the decomposition of matrix A as given below:

Here U is the eigen basis of col space space , V is the col space and Sig connects these spaces
via a diagonal matrix .

First we consider a simple numerical 5x4 A matrix to relate to above concepts.

A(5x4 ) =

12. 12. 18. 36.

11. 11. 13. 40.
17. 17. 25. 52.
2. 2. 4. 4.
8. 8. 10. 28

The col space is U given by 5 orthogonal eigen basis vector.In other words
any vector in the column space can be expressed as a linear combination of
the orthogonal basis cols of U below.

U(5x5) = - 0.4639438 0.3694793 0.0623360 0.7586594 0.2622811

- 0.4748203 - 0.6914212 0.390426 - 0.1111717 0.3628923
0.6643306 0.3762704 0.1423082 - 0.4481469 - 0.4427167
- 0.0631701 0.3558972 - 0.2852295 - 0.4525373 0.7636759
- 0.3376036 - 0.3423150 - 0.8614373 0.0803186 - 0.1425443

The row space is V . Any vector in row space is a linear combination of

the ortogonal basis cols of V .

V(4x4)= - 0.2649413 0.1962187 0.4640557 0.8221658

- 0.2649413 0.1962187 - 0.8726214 0.3603280
- 0.3696952 0.8303086 0.1361886 - 0.3941646
- 0.8502574 - 0.4833053 0.0680943 - 0.1970823

The sigma matrix is able to take any row vextor (4x1) into col space (5x1)

S(5x4) = 94.025126 0. 0. 0.
0. 6.1053811 0. 0.
0. 0. 3.221D-16 0.
0. 0. 0. 5.804D-17
0. 0. 0. 0.

2. Toy application of above matrix.

Let us assume 5 girls rate 4 occupations Scientist, Teacher, Dancer, Actor.

Then the above matrix A can be viwed as their ratings simply by labeling the cols and rows as
below. A(5x4 ) = Snt Te Da Ac
G1 12. 12. 18. 36.
G2 11. 11. 13. 40.
G3 17. 17. 25. 52.
G4 2. 2. 4. 4.
G5 8. 8. 10. 28
Possible interpretation of SVD.

The occupation space is obtained from Oc= At*A as

Oc = Snt Te Da Ac
Snt 622. 622. 872. 1988.
Te 622. 622. 872. 1988.
Da 872. 872. 1234. 2764.
Ac 1988. 1988. 2764. 6400.

Now this Matrix Oc can be seen in the eigen basis V

with corresponding svd values as:

V(4x4)= 94.025126

S(5x4) = 94.025126 0. 0. 0.
0. 6.1053811 0. 0.
0. 0. 3.221D-16 0.
0. 0. 0. 5.804D-17
0. 0. 0. 0.

The S matrix tells us that there are really only two impt sets of ratings given
by the girls viz (ignore neg sign)

S(1,1)= 94.025126 this is the max weightage ,given to first vector of ratings of the girls:

V(:,1) = Snt
Snt - 0.2649413
Te - 0.2649413
Da - 0.3696952
Ac - 0.8502574

From the vector V(:,1) we can infer that the girls preferred occupation is in descending
order :
Actor , Dancer Teacher and Scientist equal preference.

Similarly as second choice the ratings are obtained from

V(:,2)
0.1962187
0.1962187
0.8303086
- 0.4833053
Here dancing is chosen over Actor
Now let us check the Girl space

U(5x5) = G1 G2 G3 G4 G5
G1 - 0.4639438 0.3694793 0.0623360 0.7586594 0.2622811
G2 - 0.4748203 - 0.6914212 0.390426 - 0.1111717 0.3628923
G3 0.6643306 0.3762704 0.1423082 - 0.4481469 - 0.4427167
G4 - 0.0631701 0.3558972 - 0.2852295 - 0.4525373 0.7636759
G5 - 0.3376036 - 0.3423150 - 0.8614373 0.0803186 - 0.1425443

This tells us that there are 5 sets of girls corresponding to 5 cols of U.

However only the first coloumn girl set U(:,1) is impt because its associated
sig value = 94 is the highest.
Moreover this set has corresponding pref of occupations as given by V(:,1)

V(:,1) =
- 0.2649413 Snt
- 0.2649413 Ta
- 0.3696952 Da
- 0.8502574 Ac

We can obtain the rating of each girl for this set with A*V(:,1) as below:

->A*V(:,1)
ans =
- 43.622372
- 44.645043
- 62.46377
- 5.9395756
- 31.74322

This tells girl 3 strongly prefers this rating (Actor , Dancer Teacher and Scientist equal
preference )as her score is 62.4 highest in the projection on V(:,1) axis.

Check actual rating table ie matrix A: Seems acceptable.

A =
Snt Ta Da Ac G3 has most similar rating
G1 12. 12. 18. 36. as given by V(:,1)
G2 11. 11. 13. 40.
G3 17. 17. 25. 52.
G4 2. 2. 4. 4.
G5 8. 8. 10. 28.
3 Verify with published example (Movie Rating by Users)

(ref:
http://rstudio-pubs-
static.s3.amazonaws.com/335300_11d40bf12d8940f78d9661b3c63150dc.html#testing-using-svd-
in-toy-dataset
)

//xxx

https://www.academia.edu/6065432/A_SINGULAR_VALUE_DECOMPOSITION_APPROA
CH_FOR_RECOMMENDATION_SYSTEMS
Now consider user2 movie rating viz

olduser=M(2,:)

This user can be expressed in movie rating eigenspace V co-ords by taking the dot
product of olduser movie ratung and eigen co-ord matrix V

oldusereig= olduser*V

= - 7.6 - 1.9 - 2.8 0.8 0.0

Now we want to go into user space. We do this by dividing each component

of oldusereig by correponding sig value in S matrix.

sd=diag(S)' // gives diagonal values of S

= 16. 6.2 4.4 2.9 1.6

olduserspace=oldusereig./sd' /divides componentwise

= - 0.5 - 0.3 - 0.6 0.3 0.0

Compare with SVD U matrix row 2:

U(2,:) =- 0.5 - 0.3 - 0.6 0.3 0.0 - 0.4

We notice U(2,6) = -.4 . We cannot get this because rank of S is 5, which means that
really only 5 distinct classifications are possible in both user space and movie rating space.
We can ignore the 6th component of U and call it noise.
How to use SVD of movie rating?

We can use SVD to predict a newuser's preference by comparing his incomplete choice
with existing users who are closest to him.

Let us assume that 0 rating means newuser has not seen the movie. Should we recommend
this to him? If so what maybe the strength of recommendation. This is explored below.

new user rating of movies 3 and 5 , say are:

nu=[ 0 0 3 0 4]

1. We can obtain his user space co-ords as before with

userspace=olduser*V*S5^-1
= - 0.5 - 0.3 - 0.6 0.3 0.0

where S5 =S(1:5,1:5) [ (S6,6) does not exist]

Next we locate nearest user in user space by taking dot product

wrt to co-ords of old users called Cosine-based similarity .
We need to add an extra 0 in user space to make it compatible
with 6dim user space

userspace =[ userspace 0]

similarity= userspace*U

= 0.4 - 0.7 0.2 - 0.2 0.0 0.3

Recall newuser rating nu = 0. 0. 3. 0. 4.

From cosine similarity

closest old user is 2 (-.7)

So base for recommending is M(2,:) = 5. 4. 4. 3. 2.

In other words new user would like movies 1 2 2 3 2 if offered all

5 movies!
4 Other Examples.

We can visualize other examples based on the above analyses . The intuition is that
givn any table of 2 dimension we can think of it as two spaces cols and rows , which
can be separated and connected using matrix M=USV' .

Some possible practical aplications are

4.1 Students / subject marks

4.2 Teams / matches

4.3 Brides/ Bridegrooms matchmaking

4.4 menu / customers

4.5 Fruits/ customers

//xxxxxxxxxxxxxxx
//xxxxxxxxxxxxxxxxxxxxxx

2 Scilab5.4.4 code

//Expt 1
x1 = int(5*rand(5,1)) ;
x2 = int(5* rand(4,1));
x1=x1(:)
x2=x2(:)
x3 = int(5*rand(5,1)) ;
x4 = int(5* rand(4,1));
x3=x3(:)
x4=x4(:)

sig1 = 3
sig2 =2

A= sig1*x1 *x2' + sig2*x3*x4'

[U ,S ,V ] =svd(A)

// Oc=A'*A
// [ oc1 , lc1] = spec(Oc)
// lcs1=sqrt(lc1)
// gg=A*A'
// [ gg1 , lc2] = spec(gg)
// lgg2=sqrt(lc2)

V(:,1)
V(:,2)
A*V(:,1) // girl rating of set V(:,1)

//xxxxxxxxxxxxxxx

//Expt 2

M= [ 1 5 0 5 4;
5 4 4 3 2;
0 4 0 0 5;
4 4 1 4 0;
0 4 3 5 0;
2 4 3 5 3]
//M is 6x 5 matrix col =users row = movies

//New user with rating: 0,0,2,0,4

//## [,1] [,2]

//## [1,] -0.1504557 0.08419174

[U,S,V] =svd(M)
//U*S*V' gives back M
u2=[ U(:,1) U(:,2)]
s2=S(1:2,1:2)
v2=[ V(:,1) V(:,2)]
m2 =u2*s2*v2'
nu=[ 0 0 3 0 4]

//some more code used

//Expt 2

M= [ 1 5 0 5 4;
5 4 4 3 2;
0 4 0 0 5;
4 4 1 4 0;
0 4 3 5 0;
2 4 3 5 3]

//M is 6x 5 matrix col =users row = movies

//New user with rating: 0,0,2,0,4

//## [,1] [,2]

//## [1,] -0.1504557 0.08419174

[U,S,V] =svd(M)
//U*S*V' gives back M
u2=[ U(:,1) U(:,2)]
s2=S(1:2,1:2)
v2=[ V(:,1) V(:,2)]
m2 =u2*s2*v2'
nu=[ 0 0 3 0 4]

format(4); // outputs only 4 char for each variable

k=2
Uk =U(1:6, 1:2)Vk
Vk =V(1:5, 1:2)
Sk =S(1:2, 1:2)
M2=Uk*Sk*Vk' // reduced rank 2 approx of M
nuvk(1,1)*Vk(:,1)' +nuvk(1,2)*Vk(:,2)' // recover nu in red dim
//nf=nu*V //for full component in V
//nf*V' // see recovered nu

//M'*U // this gives users in eigen basis

//U5 =U(1:5,1:5)
olduser=M(2,:)
S5 =S(1:5,1:5)
userspace=olduser*V*S5^-1