MLT: Week-1

Standard PCA

A Aniruddha
Steps involved in PCA
Steps involved in PCA

Step 1: Center the dataset

Steps involved in PCA

Step 1: Center the dataset

Step 2: Calculate the covariance matrix of the centered data

Steps involved in PCA

Step 1: Center the dataset

Step 2: Calculate the covariance matrix of the centered data

Step 3: Compute the eigenvectors and eigenvalues

Steps involved in PCA

Step 1: Center the dataset

Step 2: Calculate the covariance matrix of the centered data

Step 3: Compute the eigenvectors and eigenvalues

Step 4: Sort the eigenvalues in descending order and choose the top k
eigenvectors corresponding to the highest eigenvalues
Steps involved in PCA

Step 1: Center the dataset

Step 2: Calculate the covariance matrix of the centered data

Step 3: Compute the eigenvectors and eigenvalues

Step 4: Sort the eigenvalues in descending order and choose the top k
eigenvectors corresponding to the highest eigenvalues

Step 5: Transform the original data by multiplying it with the selected

eigenvectors(PC's) to obtain a lower-dimensional representation.
Observe the dataset
Let's take a datasetX of shape dn
( , ) where
d
* : no. of features
n
* : no. of datapoints
Observe the dataset
Let's take a datasetX of shape dn
( , ) where
d
* : no. of features
n
- -
* : no. of datapoints

X =
4

1
5

4
6

3
7

4 - - - -
2

1
1

2
0

3
1

4
Observe the dataset
Let's take a dataset of shape X dn
( , ) where
d
* : no. of features
n
- -
* : no. of datapoints

4
X =
4

1
5

4
6

3
7

4 - - - -
2

1
1

2
0

3
1

4
3

1
mean
-5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10

-1

-2

-3

-4

-5
Observe the dataset
Let's take a dataset of shape X dn
( , ) where
d
* : no. of features
n
- -
* : no. of datapoints

4
X =
4

1
5

4
6

3
7

4 - - - -
2

1
1

2
0

3
1

4
3

1
mean
-5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10

-1 We can see that the

-2 dataset is not centered.
-3 Let us calculate the mean
-4 and center the dataset
-5
Step 1: Center the dataset
Step 1: Center the dataset
The mean of the points of the given dataset is

X mean =
2.75

0.25
Step 1: Center the dataset
The mean of the points of the given dataset is

X mean =
2.75

0.25

The centered dataset is given by

X centered = X-X mean

Step 1: Center the dataset
The mean of the points of the given dataset is

X mean =
2.75

0.25

The centered dataset is given by

X centered = X-X mean

For the point (4, 2) , the value after updating

will be:
4

2
- 2.75

0.25
=
1.25

0.75
Step 1: Center the dataset
The mean of the points of the given dataset is

X mean =
2.75

0.25

The centered dataset is given by

X centered = X-X mean

For the point (4, 2) , the value after updating

will be:
4

2
- 2.75

0.25
=
1.25

0.75

X - - - -
- - - -
1.25 2.25 3.25 4.25 0.75 3.75 2.75 3.75
centered =
0.75 3.75 2.75 3.75 1.25 2.25 3.25 4.25
Step 1: Center the dataset
4
The mean of the points of the given dataset is

X
3

2.75
mean =
2

0.25 1

new mean
-5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 1
The centered dataset is given by

X X-X
-1

-2
centered = mean
-3

-4
For the point (4, 2) , the value after updating
-5
will be:
4

2
- 2.75

0.25
=
1.25

0.75

X - - - -
- - - -
1.25 2.25 3.25 4.25 0.75 3.75 2.75 3.75
centered =
0.75 3.75 2.75 3.75 1.25 2.25 3.25 4.25
Step 1: Center the dataset

Dataset before centering

1
mean
-5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10

-1

-2

-3

-4

-5
Step 1: Center the dataset

Dataset before centering Dataset after centering

4 4

3 3

2 2

1 1
mean new mean
-5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10

-1 -1

-2 -2

-3 -3

-4 -4

-5 -5
Step 2: Calculate the covariance matrix
Step 2: Calculate the covariance matrix
The covariance matrix is given by C =
1
∑
ni x
n
i
= 1
x Ti =
1

n XX T
Step 2: Calculate the covariance matrix
The covariance matrix is given by C =
1
∑
ni x
n
i
= 1
x Ti =
1

n XX T
1.25 0.75

2.25 3.75

- - - -
3.25 2.75

XXT =
1.25 2.25 3.25 4.25

-
0.75

-
3.75

-
2.75

-
3.75

-
4.25

-
3.75

- -
0.75 3.75 2.75 3.75 1.25 2.25 3.25 4.25 0.75 1.25

- -
3.75 2.25

- -
2.75 3.25

3.75 4.25
Step 2: Calculate the covariance matrix
The covariance matrix is given by C =
1
∑
ni x
n
i
= 1
x Ti =
1

n XX T
1.25 0.75

2.25 3.75

- - - -
3.25 2.75

XXT =
1.25 2.25 3.25 4.25

-
0.75

-
3.75

-
2.75

-
3.75

-
4.25

-
3.75

- -
0.75 3.75 2.75 3.75 1.25 2.25 3.25 4.25 0.75 1.25

- -
3.75 2.25

- -
2.75 3.25

C =
1

n XX T =
1

8
71.5

68.5
68.5

71.5
3.75 4.25

8.9375 8.5625
=
8.5625 8.9375
Step 3: Compute the eigenvectors and eigenvalues
Step 3: Compute the eigenvectors and eigenvalues
Now we calculate the eigenvalues and the corresponding eigenvectors of the covariance
matrix

(𝜆, w ) = Solver(C)
Step 3: Compute the eigenvectors and eigenvalues
Now we calculate the eigenvalues and the corresponding eigenvectors of the covariance
matrix

w
3

(𝜆, ) = Solver(C) 2

w1
1

0.707
𝜆1 = 17.5 = -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7
0.707
-1

-2

-3

-4
Step 3: Compute the eigenvectors and eigenvalues
Now we calculate the eigenvalues and the corresponding eigenvectors of the covariance
matrix

w
3

(𝜆, ) = Solver(C) 2

w1
1

0.707
𝜆1 = 17.5 = -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7
0.707
-1

𝜆2 = 0.375 w2 =
- 0.707
-2

-3
0.707
-4
Question - 1:
How many principal components do we need to explain 95% of the variance in the dataset?
Question - 1:
How many principal components do we need to explain 95% of the variance in the dataset?

k
k
∑ 𝜆 i
The choice of can be given by i = 1

d ≥ 0.95

∑ 𝜆 i
i = 1
Question - 1:
How many principal components do we need to explain 95% of the variance in the dataset?

k
k
∑ 𝜆 i
The choice of can be given by i = 1

d ≥ 0.95

∑ 𝜆 i
i = 1

k
𝜆1 17.5
For = 1, we have =
𝜆1 + 𝜆2 17.5 + 0.375

= 0.979
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

- - - - w 0.707

X
1
=

- - - -
1.25 2.25 3.25 4.25 0.75 3.75 2.75 3.75 0.707
centered =
0.75 3.75 2.75 3.75 1.25 2.25 3.25 4.25
w2 =
-0.707

0.707
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

- - - - w 0.707

X
1
=

- - - -
1.25 2.25 3.25 4.25 0.75 3.75 2.75 3.75 0.707
centered =
0.75 3.75 2.75 3.75 1.25 2.25 3.25 4.25
w2 =
-0.707

0.707

th
To represent the datapoints along the k principal component we simply need to
th
multiply it with the the eigenvector corresponding to the k -largest eigenvalue
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

- - - - w 0.707

X
1
=

- - - -
1.25 2.25 3.25 4.25 0.75 3.75 2.75 3.75 0.707
centered =
0.75 3.75 2.75 3.75 1.25 2.25 3.25 4.25
w2 =
- 0.707

0.707

th
To represent the datapoints along the k principal component we simply need to
th
multiply it with the the eigenvector corresponding to the k -largest eigenvalue

T
Xcentered
T . w 1
= 1.414 4.242 4.242 5.656 -
1.414 - 4.242 -
4.242 - 5.656
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

- - - - w 0.707

X
1
=

- - - -
1.25 2.25 3.25 4.25 0.75 3.75 2.75 3.75 0.707
centered =
0.75 3.75 2.75 3.75 1.25 2.25 3.25 4.25
w2 =
-0.707

0.707

th
To represent the datapoints along the k principal component we simply need to
th
multiply it with the the eigenvector corresponding to the k -largest eigenvalue

T
Xcentered
T . w 1
= 1.414 4.242 4.242 5.656 -1.414 - 4.242 -
4.242 - 5.656

T
Xcentered
T . w 2
= - 0.353 1.060 -
0.353 - 0.353 - 0.353 1.060 -
0.353 -
0.353
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

Projection on first PC = w XT
. . w T
1 centered 1

=
0.707

0.707
1.414 4.242 4.242 5.656 - 1.414 - 4.242 - 4.242 - 5.656

- - - -
- - - -
1 3 3 4 1 3 3 4
=
1 3 3 4 1 3 3 4
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

- - - -
- - - -
1 3 3 4 1 3 3 4
The matrix of reconstructed points is =
1 3 3 4 1 3 3 4
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

- - - -
- - - -
1 3 3 4 1 3 3 4
The matrix of reconstructed points is =
1 3 3 4 1 3 3 4

-5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8

-1

-2

-3

-4
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

- - - -
- - - -
1 3 3 4 1 3 3 4
The matrix of reconstructed points is =
1 3 3 4 1 3 3 4

Comparison of the original dataset

0
-5 -4 -3 -2 -1 1 2 3 4 5 6 7 8
and the projected dataset
-1

-2

-3

-4
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

Projection on first PC = w XT
. . w T
1 centered 1

=
0.707

0.707
1.414 4.242 4.242 5.656 - 1.414 - 4.242 - 4.242 - 5.656

- - - -
- - - -
1 3 3 4 1 3 3 4
=
1 3 3 4 1 3 3 4

Projection on second PC = w XT. . w T

2 centered 2

=
0.707

0.707
- 0.353 1.060 - 0.353 - 0.353 - 0.353 1.060 - 0.353 - 0.353

- -
- - - - - -
0.25 0.75 0.25 0.25 0.25 0.75 0.25 0.25
=
0.25 0.75 0.25 0.25 0.25 0.75 0.25 0.25
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components
Adding the two projections, we get

- - - - - -
- - - - - - - - - -
1 3 3 4 1 3 3 4 0.25 0.75 0.25 0.25 0.25 0.75 0.25 0.25
= +
1 3 3 4 1 3 3 4 0.25 0.75 0.25 0.25 0.25 0.75 0.25 0.25
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components
Adding the two projections, we get

- - - - - -
- - - - - - - - - -
1 3 3 4 1 3 3 4 0.25 0.75 0.25 0.25 0.25 0.75 0.25 0.25
= +
1 3 3 4 1 3 3 4 0.25 0.75 0.25 0.25 0.25 0.75 0.25 0.25

- - - -
- - - -
1.25 2.25 3.25 4.25 0.75 3.75 2.75 3.75
=
0.75 3.75 2.75 3.75 1.25 2.25 3.25 4.25

As the points are in R

, we can get back the original centered dataset when we use both
2

the Principal Components

Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

Projection on first PC = w XT
. . w T
1 centered 1

=
0.707

0.707
1.414 4.242 4.242 5.656 - 1.414 - 4.242 - 4.242 - 5.656

- - - -
- - - -
1 3 3 4 1 3 3 4
=
1 3 3 4 1 3 3 4
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

Projection on first PC = w XT
. . w T
1 centered 1

=
0.707

0.707
1.414 4.242 4.242 5.656 - 1.414 - 4.242 - 4.242 - 5.656

- - - -
- - - -
1 3 3 4 1 3 3 4
=
1 3 3 4 1 3 3 4

The first point of XT

centered is
1.25

0.75
and the same point when reconstructed using the
1
first principal component is given by
1
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

Projection on first PC = w XT
. . w T
1 centered 1

=
0.707

0.707
1.414 4.242 4.242 5.656 - 1.414 - 4.242 - 4.242 - 5.656

- - - -
- - - -
1 3 3 4 1 3 3 4
=
1 3 3 4 1 3 3 4

The first point of XT

centered is
1.25

0.75
and the same point when reconstructed using the
1
first principal component is given by
1

The reconstruction error for this point is given by = length error

2
( )

= || x - xTw w
1 1 1
||
2

- -
1

2 2
= (1.25 1) + (0.75 1)

= 0.125
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

The reconstruction error for all of the points in X centered when reconstructed using the
first principal component turns out to be,

e 1
= 0.125 1.125 0.125 0.125 0.125 1.125 0.125 0.125

The MSE w.r.t to first PC will be = 0.375

Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

The reconstruction error for all of the points in X centered when reconstructed using the
first principal component turns out to be,

e 1
= 0.125 1.125 0.125 0.125 0.125 1.125 0.125 0.125

The MSE w.r.t to first PC will be = 0.375

We can now take another unit-vector wr =

0

1
and project X centered on it. We will then
find the reconstruction error and compare it with that of the first principal component
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

Projection on wr wr X T
= .
centered . wr T
=
0

1
0.75 3.75 2.75 3.75 - 1.25 - 2.25 - 3.25 - 4.25

- - - -
0 0 0 0 0 0 0 0
=
0.75 3.75 2.75 3.75 1.25 2.25 3.25 4.25
Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

Projection on wr wr X T
= .
centered . wr T
=
0

1
0.75 3.75 2.75 3.75 - 1.25 - 2.25 - 3.25 - 4.25

- - - -
0 0 0 0 0 0 0 0
=
0.75 3.75 2.75 3.75 1.25 2.25 3.25 4.25

The reconstruction error for all of the points in X when reconstructed using the
w
centered
vector r turns out to be,

er = 1.5625 5.0625 10.5625 18.0625 0.5625 14.0625 7.5625 14.0625

The MSE w.r.t to wr will be = 8.9375

Question - 2:
Represent the data using its two principal components and find the reconstruction error
on each of these components

Projection on wr wr X T
= .
centered . wr T
=
0

1
0.75 3.75 2.75 3.75 - 1.25 - 2.25 - 3.25 - 4.25

- - - -
0 0 0 0 0 0 0 0
=
0.75 3.75 2.75 3.75 1.25 2.25 3.25 4.25

The reconstruction error for all of the points in X when reconstructed using the
w
centered
vector r turns out to be,

er = 1.5625 5.0625 10.5625 18.0625 0.5625 14.0625 7.5625 14.0625

The MSE w.r.t to wr will be = 8.9375

We observe that the reconstruction error when the points are projected on a vector wr
is higher as compared to the error when projected on the first principal component
Question - 3:

You are given w1 =

1 1

1
as the first principal component. Among the following, which
2

of them could be the second principal component?

1 1 1 - 1 -
-
0 1 1 1
(a) (b) (c) (d)
1 0 1 1
2 2 2 2
Question - 3:

You are given w1 =

1 1

1
as the first principal component. Among the following, which
2

of them could be the second principal component?

1 1 1 - 1 -
-
0 1 1 1
(a) (b) (c) (d)
1 0 1 1
2 2 2 2

We know that the principal components will be orthogonal to each other and based on
that option (c) can be the second principal component.

1 1
.
1 - 1
= - 1
+
1
= 0
1 1 2 2
2 2