Principal Component Analysis - A Tutorial

Alaa Tharwat

Electrical Dept. - Suez Canal University- Egypt

April 2, 2016

Alaa Tharwat April 2, 2016 1 / 37

 Agenda

Introduction.
Principal Component Analysis (PCA).
Numerical Examples.
Experiments.
Conclusions and Future Works.

Alaa Tharwat April 2, 2016 2 / 37

 Introduction

The main objective is to:

Explain the Explain how to use PCA in different fields.
Introduce numerical examples to explain how to apply PCA in
different applications.

Alaa Tharwat April 2, 2016 3 / 37

 Introduction

The goal of the PCA technique is to find a lower dimensional space

or PCA space (W ) that is used to transform the data
(X = {x1 , x2 , . . . , xN }) from a higher dimensional space (RM ) to a
lower dimensional space (Rk ), where N represents the total number
of samples or observations.

PCA
Orthogonal Transformation
Axes Rotation

x2 PC1 M k
(Direction of the
maximum variance)
ℜ ℜ
PC2
PC2

2
σ22 σ1
PC1

x1
Figure: Example of the two-dimensional data (x1 , x2 ).
Alaa Tharwat April 2, 2016 4 / 37
 Introduction Principal Components (PCs)

The PCA space consists of k principal components. The principal

components are orthonormal, uncorrelated, and it represents the
direction of the maximum variance.
The first principal component ((P C1 or v1 ) ∈ RM ×1 ) of the PCA
space represents the direction of the maximum variance of the data,
the second principal component has the second largest variance, and
so on.

Alaa Tharwat April 2, 2016 5 / 37

 Introduction Covariance Matrix Method

In this method, there are two main steps to calculate the PCs of the
PCA space. First, the covariance matrix of the data matrix (X) is
calculated. Second, the eigenvalues and eigenvectors of the
covariance matrix are calculated.
The covariance matrix is used when the number of variables more
than one.
Covariance matrix is a symmetric matrix (i.e. X = X T ) and always
positive semi-definite matrix.
 
V ar(x1 , x1 ) Cov(x1 , x2 ) . . . Cov(x1 , xM )
 Cov(x2 , x1 ) V ar(x2 , x2 ) . . . Cov(x2 , xM ) 
(1)
 
 .. .. .. .. 
 . . . . 
Cov(xM , x1 ) Cov(xM , x2 ) V ar(xM , xM )

Alaa Tharwat April 2, 2016 6 / 37

 Introduction Covariance Matrix Method

The covariance matrix is solved by calculating the eigenvalues (λ)

and eigenvectors (V ) as follows:

V Σ = λV (2)
where V and λ represent the eigenvectors and eigenvalues of the
covariance matrix, respectively.
The eigenvalues are scalar values, while the eigenvectors are non-zero
vectors, which represent the principal components, i.e. each
eigenvector represents one principal component.
The eigenvectors represent the directions of the PCA space, and the
corresponding eigenvalues represent the scaling factor, length,
magnitude, or the robustness of the eigenvectors.
The eigenvector with the highest eigenvalue represents the first
principal component and it has the maximum variance.

Alaa Tharwat April 2, 2016 7 / 37

 Introduction Covariance Matrix Method

A
DataLMatrixL MeanL Mean-CentringLDataL
(X) (μ) (D=X-μ)
x1 x2 xN d1 d2 dN

X= ‫ــ‬ =

Data (MxN) (Mx1) (MxN)

Sample
C
B
A
CalculatingLEigenvaluesLandL B
A
Eigenvectors CovarianceLMatrixL
Method
Sorted
Eigenvalues
Σ=DDT
λ1 λ2 λk λM

LargestLkL
Eigenvalues
kLSelected CovarianceL
Eigenvectors MatrixL(Σ)

(MxM)
PCA
LSpace
V1 V2 Vk VM

(Mxk)
Eigenvectors

Alaa Tharwat April 2, 2016 8 / 37

 Introduction Covariance Matrix Method

1: Given a data matrix (X = [x1 , x2 , . . . , xN ]), where N represents the

total number of samples and xi represents the ith sample.
2: Compute the mean of all samples as follows:
N
1 X
µ= xi (3)
N
i=1
3: Subtract the mean from all samples as follows:
N
X
D = {d1 , d2 , . . . , dN } = xi − µ (4)
i=1
4: Compute the covariance matrix as follows:
1
Σ= D × DT (5)
N −1
5: Compute the eigenvectors V and eigenvalues λ of the covariance
matrix (Σ).
6: Sort eigenvectors according to their corresponding eigenvalues.
7: Select the eigenvectors that have the largest eigenvalues
= {v
W Alaa 1 , . . . , vk }.
Tharwat April 2, 2016 9 / 37
 Introduction Singular Value Decomposition (SVD) Method

Singular value decomposition is one of the most important linear

algebra principles.
The aim of the SVD method is to diagonalize the data matrix
(X ∈ Rp×q ) into three matrices as in Equation (6).
 
s1 0 0 0
  0 s2 0 0  −r1T −
  

 0 0 ... 0   T 
 −r2 − 
 
T
X = LSR = l1 · · · lM
    .  (6)
  .. 
 0 0 0 sN  

 .. .. .. ..  −rT −
. . . .  N
0 0 0 0
where L(p × p) are called left singular vectors, S(p × q) is a diagonal
matrix represents the singular values that are sorted from
high-to-low, i.e. the highest singular value in the upper-left index of
S, thus, s1 ≥ s2 ≥ · · · ≥ sq ≥ 0, and R(q × q) represents the right
singular vectors. The left and right singular matrices, i.e. L and R,
are orthonormal bases.
Alaa Tharwat April 2, 2016 10 / 37
 Introduction Singular Value Decomposition (SVD) Method

The columns of the right singular vectors (R) represent the

eigenvectors of X T X or the principal components of the PCA space,
and s2i , ∀ i = 1, 2, . . . , q represent their corresponding eigenvalues

Alaa Tharwat April 2, 2016 11 / 37

 Introduction Singular Value Decomposition (SVD) Method

A
DataNMatrixN-X) MeanN-μ) Mean-CentringN DT
DataN-D=X-μ)
x1 x2 xN d1 d2 dN

d1
d2

Transpose
X= ‫ــ‬ =

dN
-NxM)
-MxN) -Mx1) -MxN)

Data
Sample
SingularNValueN
DecompositionNMethod

B
A
L S RT
li M M

ri

si M
-MxM)
N

-NxN)

C
A

λi=si2

LargestNkN
Eigenvalues
V1 V2 Vk VM

λ1 λ2 λk λM
SortedN Eigenvectors
Eigenvalues

PCA
NSpace kNSelected
-W) Eigenvectors

-Mxk)

Figure: Visualized steps to calculate the PCA space using SVD method.
Alaa Tharwat April 2, 2016 12 / 37
 Introduction Singular Value Decomposition (SVD) Method

1: Given a data matrix (X = [x1 , x2 , . . . , xN ]), where N represents the

total number of samples and xi (M × 1) represents the ith sample.
2: Compute the mean of all samples as in Equation (3).
3: Subtract the mean from all samples as in Equation (4).
4: Construct a matrix Z = √N1−1 DT , Z(N × M ).
5: Calculate SVD for Z matrix as in Equation (6).
6: The diagonal elements of S represent the square root of the sorted
eigenvalues, λ = diag(S 2 ), while the PCs are represented by the
columns of R.
7: Select the eigenvectors that have the largest eigenvalues
W = {R1 , R2 , . . . , Rk } to construct the PCA space.
8: All samples are projected on the lower dimensional space of PCA (W )
as follows, Y = W T D.

Alaa Tharwat April 2, 2016 13 / 37

 Introduction Singular Value Decomposition (SVD) Method

The covariance matrix is the product of DDT , where

D = {di }Ni=1 , di = xi − µ. Using Equation (6) that is used to
calculate SVD, the covariance matrix can be calculated as follows:

DDT = (LSRT )T (LSRT ) = RS T LT LSRT (7)

where LT L = I

DDT = RS 2 RT = (SV D(DT ))2 (8)

where S 2 represents the eigenvalues of DT D or DDT and the

columns of the right singular vector (R) represent the eigenvectors of
DDT .
To conclude, the square root of the eigenvalues that are calculated
using the covariance matrix method are equal to the singular values
of SVD method. Moreover, the eigenvectors of Σ are equal to the
columns of R. Thus, the eigenvalues and eigenvectors that are
calculated using the two methods are equal.
Alaa Tharwat April 2, 2016 14 / 37
 Introduction PCA Space (Lower Dimensional Space)

To construct the lower dimensional space of PCA (W ), a linear

combination of k selected PCs that have the most k eigenvalues are
used to preserve the maximum amount of variance, i.e. preserve the
original data, while the other eigenvectors or PCs are neglected.
The dimension of the original data is reduced by projecting it after
subtracting the mean onto the PCA space as in Equation (9).
N
X
T
Y =W D= W T (xi − µ) (9)
i=1

Alaa Tharwat April 2, 2016 15 / 37

 Introduction PCA Space (Lower Dimensional Space)

DatayMatrixy(D) DatayAftery
Projectiony(Y)
d1 d2 dN
y1 y2 yN

PCA
Projection Y=
D= ySpace
Y=WTD (W)

D∈RMxN (Mxk) Y∈RkxN

Figure: Data projection in PCA as in Equation (9).

Alaa Tharwat April 2, 2016 16 / 37

 Introduction Data Reconstruction

The original data can be reconstructed again as in Equation (10).

N
X
X̂ = W Y + µ = W yi + µ (10)
i=1

where X̂ represents the reconstructed data.

The deviation between the original data and the reconstructed data
are called the reconstruction error or residuals as denoted in Equation
(11).
XN
Error = X − X̂ = (xi − x̂i )2 (11)
i=1
The robustness of the PCA is controlled by the number of selected
eigenvectors (k) and it is measured by the sum of the selected
eigenvalues, which is called total variance as in Equation (12).
Pk
Total Variance of W λi
Robustness of the PCA space = = Pi=1M
Total Variance i=1 λi
(12)
Alaa Tharwat April 2, 2016 17 / 37
 Numerical Examples First Example: 2D-Class Example

Given a data matrix X = {x1 , x2 , . . . , x8 }, where xi represents the

ith sample as denoted in Equation (13).
 
1.00 1.00 2.00 0.00 5.00 4.00 5.00 3.00
X= (13)
3.00 2.00 3.00 3.00 4.00 5.00 5.00 4.00

Each sample of the matrix was represented by one column that

consists of two features (xi ∈ R2 ) to visualize it. The total mean
 (µ)
2.63
was then calculated as in Equation (3) and its value was µ = .
3.63
The data were then subtracted from the mean as in Equation (4)
and the values of D will be as follows:

 
−1.63 −1.63 −0.63 −2.63 2.38 1.38 2.38 0.38
D=
−0.63 −1.63 −0.63 −0.63 0.38 1.38 1.38 0.38
(14)

Alaa Tharwat April 2, 2016 18 / 37

 Numerical Examples First Example: 2D-Class Example

The covariance matrix (Σ) were then calculated as in Equation (5).

The eigenvalues (λ) and eigenvectors (V ) of the covariance matrix
were then calculated. The values of the Σ, λ, and V are shown
below.
     
3.70 1.70 0.28 0.00 0.45 −0.90
Σ= , λ= , and V =
1.70 1.13 0.00 4.54 −0.90 −0.45
(15)
Calculating the Projected Data: The mean-centring data (i.e.
data − total mean) were then projected on each eigenvector as
follows, (Yv1 = v1T D and Yv2 = v2T D), where Yv1 and Yv2 represent
the projection of the D on the first and second eigenvectors, i.e.
v1 and v2 , respectively. The values of Yv1 and Yv1 are shown below.
 
Yv1 = −0.16 0.73 0.28 −0.61 0.72 −0.62 −0.18 −0.17
 
Yv2 = 1.73 2.18 0.84 2.63 −2.29 −1.84 −2.74 −0.50
(16)
Alaa Tharwat April 2, 2016 19 / 37
 Numerical Examples First Example: 2D-Class Example

Calculating the Projected Data: The reconstruction error between

the original data and the reconstructed data using all eigenvectors or
all principal components, i.e. there is no information lost,
approximately tend to zero.
 
2.55 2.95 2.75 2.35 2.95 2.35 2.55 2.55
X̂1 = v1 Yv1 + µ =
3.77 2.97 3.37 4.17 2.98 4.18 3.78 3.78
 
1.07 0.67 1.88 0.27 4.68 4.28 5.08 3.08
X̂2 = v2 Yv2 + µ =
2.85 2.66 3.25 2.46 4.65 4.45 4.84 3.85
(17)

Alaa Tharwat April 2, 2016 20 / 37

 Numerical Examples First Example: 2D-Class Example

The error between the original data and the reconstructed data that
were projected on the first and second eigenvectors are denoted by
Ev1 and Ev2 , respectively. The values of Ev1 and Ev2 are as follows:

Ev1 = X − X̂1
 
−1.55 −1.95 −0.75 −2.35 2.05 1.65 2.45 0.45
=
−0.77 −0.97 −0.37 −1.17 1.02 0.82 1.22 0.22
Ev2 = X − X̂2
 
−0.07 0.33 0.12 −0.27 0.32 −0.28 −0.08 −0.08
=
0.15 −0.66 −0.25 0.54 −0.65 0.55 0.16 0.15
(18)

Alaa Tharwat April 2, 2016 21 / 37

 Numerical Examples First Example: 2D-Class Example

x2 )
C1
st t (P
Fir onen
6 mp
l Co
pa
nci
tion Pri
ec
5 P roj

4 PC2
µ
Pr
in c

3
ipa

µ
Se mpo

PC1
lC
co nen
o
nd

2 Data Sample
Projection on PC1
t (P

Projection on PC2
C2
)

1
1 2 3 4 5 6 x1

(a) (b)
Figure: A visualized example of the PCA technique, (a) the dotted line
represents the first eigenvector (v1 ), while the solid line represents the second
eigenvector (v2 ) and the blue and green lines represent the reconstruction error
using P C1 and P C2 , respectively; (b) projection of the data on the principal
components, the blue and green stars represent the projection onto the first and
second principal components, respectively.
Alaa Tharwat April 2, 2016 22 / 37
 Numerical Examples First Example: 2D-Class Example

The first three steps in SVD method and covariance matrix methods
are common. In the fourth step in SVD, the original data were
transposed as follows, Z = N 1−1 DT . The values of Z are as follows:

−0.61 −0.24
 
−0.61 −0.61
 
−0.24 −0.24
 
−0.99 −0.24
Z=
 0.90
 (19)
 0.14 

 0.52 0.52 
 
 0.90 0.52 
0.14 0.14

Alaa Tharwat April 2, 2016 23 / 37

 Numerical Examples First Example: 2D-Class Example

SVD was then used to calculate L, S, and R as in Equation (6) and

their values are as follows:

−0.31 0, 12 −0, 07 −0, 60 0, 58 0, 15 0, 41 0, 04


−0.39 −0, 52 −0, 24 0, 20 −0, 29 0, 53 0, 31 0, 14
−0.15 −0, 20 0, 96 −0, 01 −0, 01 0, 08 0, 07 0, 02 

−0.47 0, 43 0, 02 0, 69 0, 32 −0, 05 0.12 −0, 01
L=  0.41 −0, 51 −0, 04 0, 31 0, 68 0, 08 −0, 09 0, 02  ,
 
 0.33 0.44 0, 08 0, 02 0, 02 0, 82 −0, 15 −0, 05
0, 49 0, 12 0, 05 0, 17 −0, 15 −0, 12 0, 83 −0, 03
0, 09 0, 12 0, 02 0, 00 0, 01 −0, 05 −0, 04 0, 99
2.13 0

0 0.53
0 0 
 

0 0 
S=

0 0  ,
 
 0 0 
0 0
0 0
h i
0.90 −0.45
R= 0.45 0.90
(20)
Alaa Tharwat April 2, 2016 24 / 37
 Numerical Examples Multi-Class Example

In this example, each sample was represented by four variables. In

this experiment, the second variable was constant for all observations
as shown below.

 
1.00 1.00 2.00 0.00 7.00 6.00 7.00 8.00
2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00
X=
5.00
 (21)
6.00 5.00 9.00 1.00 2.00 1.00 4.00
3.00 2.00 3.00 3.00 4.00 5.00 5.00 4.00

The covariance matrix of the given data was calculated and its values
are shown below.
 
10.86 0 −7.57 2.86
 0 0 0 0 
Σ= −7.57 0
 (22)
7.55 −2.23
2.86 0 −2.23 1.13

Alaa Tharwat April 2, 2016 25 / 37

 Numerical Examples Multi-Class Example

The eigenvalues and eigenvectors of are

   
17.75 0.00 0.00 0.00 0.76 0.62 −0.20 0.00
 0.00 1.46 0.00 0.00  0.00 0.00 0.00 1.00
λ=
 0.00
 V =
−0.61

0.00 0.33 0.00 0.79 0.10 0.00
0.00 0.00 0.00 0.00 0.21 0.05 0.98 0.00
(23)
1 The first eigenvector represents the first principal component because
17.75
it was equal to 17.75+1.46+0.33+0 ≈ 90.84% of the total variance.
2 The first three eigenvectors represent 100% of the total variance of
the total data and the fourth eigenvector was redundant.
3 The second variable, i.e. second row, will be neglected completely
when the data were projected on any of the best three eigenvectors.
4 The projected data on the fourth eigenvector preserved only the
second variable and all the other original data were lost, and the
reconstruction error was ≈ 136.75, while the reconstruction error was
≈ 12.53 , 126.54 , 134.43 when the data were projected on the first
three eigenvectors, respectively,
Alaa Tharwat April 2, 2016 26 / 37
 Numerical Examples Multi-Class Example

The projected data are

 
Yv1 = −2.95 −3.78 −2.19 −6.16 4.28 3.12 4.49 3.20
 
Yv2 = −1.20 −0.46 −0.58 1.32 −0.58 −0.39 −0.54 2.39
 
Yv3 = 0.06 −0.82 −0.14 0.64 −0.52 0.75 0.45 −0.43
 
Yv4 = 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
(24)

Alaa Tharwat April 2, 2016 27 / 37

 Numerical Examples Multi-Class Example

140

120

100
Reconstruction Error

80

60

40

20

0
1 2 3 4
Index of the Eigenvectors

Figure: Reconstruction error of the four eigenvectors in the multi-class example.

Alaa Tharwat April 2, 2016 28 / 37

 Experiments Biometric Experiment

Table: A comparison between ORL, Ear, and Yale datasets in terms of accuracy
(%), CPU time (sec), and cumulative variance (%) using different number of
eigenvectors (biometric experiment).
ORL Dataset Ear Dataset Yale Dataset
Number of CPU Cum. CPU Cum. CPU
Acc. Acc. Acc. Cum.
Eigenvectors Time Var. Time Var. Time
(%) (%) (%) Var. (%)
(sec) (%) (sec) (%) (sec)
1 13.33 0.074 18.88 15.69 0.027 29.06 26.67 0.045 33.93
5 80.83 0.097 50.17 80.39 0.026 66.10 76.00 0.043 72.24
10 94.17 0.115 62.79 90.20 0.024 83.90 81.33 0.042 85.13
15 95.00 0.148 69.16 94.12 0.028 91.89 81.33 0.039 90.18
20 95.83 0.165 73.55 94.12 0.033 91.89 84.00 0.042 93.36
30 95.83 0.231 79.15 94.12 0.033 98.55 85.33 0.061 96.60
40 95.83 0.288 82.99 94.12 0.046 99.60 85.33 0.064 98.22
50 95.83 0.345 85.75 94.12 0.047 100.00 85.33 0.065 99.12
100 95.83 0.814 93.08 94.12 0.061 100.00 85.33 0.091 100.00
Acc. accuracy; Cum. Cumulative; Var. variance.

Alaa Tharwat April 2, 2016 29 / 37

 Experiments Image Compression Experiment

(a) One principal (b) 10% of the (c) 50% of the

component, principal components, principal components,
CR=512:1, and CR=10:1, and CR=2:1, and
M SE=30.7 M SE=5.31 M SE=0.909

(d) OneAlaa
principal
Tharwat (e) 10% of the (f) 50%
April of the
2, 2016 30 / 37
 Experiments Image Compression Experiment

Table: Compression ratio and mean square error of the compressed images using
different percentages of the eigenvectors (image compression experiment).
Lena Image Cameraman Image
Percentage of the Cumulative Cumulative
M SE CR M SE CR
used Eigenvectors Variance (%) Variance (%)
10 5.3100 512:51.2 97.35 8.1057 256:25.6 94.56
20 2.9700 512:102.4 99.25 4.9550 256:51.2 98.14
30 1.8900 512:153.6 99.72 3.3324 256:76.8 99.24
40 1.3000 512:204.8 99.87 2.0781 256:102.4 99.73
50 0.9090 512:256 99.94 1.1926 256:128 99.91
60 0.6020 512:307.2 99.97 0.5588 256:153.6 99.98
70 0.3720 512:358.4 99.99 0.1814 256:179.2 100.00
80 0.1935 512:409.6 100.00 0.0445 256:204.8 100.00
90 0.0636 512:460.8 100.00 0.0096 256:230.4 100.00
100 (All) 0.0000 512:512=1 100.00 0.0000 1 100.00

Alaa Tharwat April 2, 2016 31 / 37

 Experiments Image Compression Experiment

5
x 10
4
Lena
3.5 Cameraman
Robustness of the Eigenvectors

2.5

1.5

0.5

0
10 20 30 40 50 60 70 80 90 100
The Index of Eigenvectors

Figure: The robustness, i.e. total variance (see Equation (12)), of the first 100
eigenvectors using Lena and Cameraman images.

Alaa Tharwat April 2, 2016 32 / 37

 Experiments Data Visualization Experiment

Table: Datasets descriptions.

Number of Number of Number of

Dataset
Classes Features Samples
Iris 3 4 150
Iono 2 34 351
Ovarian 2 4000 216
ORL 5 10304 50
Ear64×64 5 4096 30
Yale 5 77760 55

Alaa Tharwat April 2, 2016 33 / 37

 Experiments Data Visualization Experiment

1.5 4 0.8
Cancer
Setosa Bad Radar
Versicolour Good Radar Normal
1 3 0.6
Virginica
Second Principal Component (PC2)

Second Principal Component (PC2)

Second Principal Component (PC2)
2 0.4
0.5
1 0.2
0
0 0
−0.5
−1 −0.2

−1 −2 −0.4

−1.5 −3 −0.6
−4 −3 −2 −1 0 1 2 3 4 −5 −4 −3 −2 −1 0 1 2 3 −1.8 −1.6 −1.4 −1.2 −1 −0.8 −0.6 −0.4
First Principal Component (PC1) First Principal Component (PC1) First Principal Component (PC1)

(a) (b) (c)

1200 800 1000 Subject 1
Subject 1 Subject 1 Subject 2
1000 Subject 2 Subject 2 Subject 3
Subject 3 600 500
Subject 3 Subject 4
Second Principal Component (PC2)

Second Principal Component (PC2)

Second Principal Component (PC2)

800 Subject 4 Subject 4 Subject 5
Subject 5 Subject 5 0
600 400

400 −500
200
200 −1000
0 0
−1500
−200
−200 −2000
−400

−600 −400 −2500

−1200 −1000 −800 −600 −400 −200 0 200 400 600 −200 −100 0 100 200 300 400 500 600 700 −2500 −2000 −1500 −1000 −500 0 500 1000 1500 2000
First Principal Component (PC1) First Principal Component (PC1) First Principal Component (PC1)

(d) (e) (f)

Figure: 2D Visualization of the datasets listed in Table 3, (a) Iris dataset, (b)
Iono dataset, (c) Ovarian dataset, (d) ORL dataset, (e) Ear dataset, (f) Yale
dataset.

Alaa Tharwat April 2, 2016 34 / 37

 Experiments Data Visualization Experiment

Setosa
Versicolour Bad Radar
Good Radar Cancer
Virginica
Normal
1 4 0.3
Third Principal Component (PC3)

Third Principal Component (PC3)

Third Principal Component (PC3)

0.2
0.5 2
0.1
0 0
0
−0.5 −2
−0.1

−1 −4 −0.2
2 4 1
1 4 2 6 0.5 −0.5
0 2 0 4
2 −1
0 0 0
−1 −2 −2 −1.5
−2
Second Principal Component (PC2) −2 −4 Second Principal Component (PC2) −4 −4 First Principal Component (PC1) Second Principal Component (PC2) −0.5 −2
First Principal Component (PC1) First Principal Component (PC1)

(a) (b) (c)

Subject 1 Subject 1
Subject 1 Subject 2
Subject 2
Third Principal Component (PC3)

Subject 2 Subject 3
Subject 3 400 2000
Third Principal Component (PC3)

Third Principal Component (PC3)

1000 Subject 3
Subject 4 Subject 4
Subject 5 Subject 4 Subject 5
200
500 Subject 5 1000
0
0 0
−200
−500 −1000
−400

−1000 −600 −2000

2000 1000 1000
1000 500 800 0 2000
1000 600 1000
0 400 −1000 0
0 0 200 −2000 −1000
−1000 0 −2000
−1000 Second Principal Component (PC2) −500 −200 Second Principal Component (PC2) −3000 −3000
Second Principal Component (PC2) −2000 First Principal Component (PC1) First Principal Component (PC1) First Principal Component (PC1)

(d) (e) (f)

Figure: 3D Visualization of the datasets that were listed in Table 3, (a) Iris
dataset, (b) Iono dataset, (c) Ovarian dataset, (d) ORL dataset, (e) Ear dataset,
(f) Yale dataset.

Alaa Tharwat April 2, 2016 35 / 37

 Experiments Data Visualization Experiment

Table: A comparison between 2D and 3D visualization in terms of M SE and

robustness using the datasets that were listed in Table 3.

2D 3D
Dataset
Robustness Robustness
M SE M SE
(in %) (in %)
Iris 97.76 0.12 99.48 0.05
Iono 43.62 0.25 51.09 0.23
Ovarian 98.75 0.04 99.11 0.03
ORL 34.05 24.03 41.64 22.16
Ear64×64 41.17 15.07 50.71 13.73
Yale 48.5 31.86 57.86 28.80

Alaa Tharwat April 2, 2016 36 / 37

 Conclusions and Future Work

How to calculate or construct PCA space.

For any other questions send to [email protected]

Alaa Tharwat April 2, 2016 37 / 37