Principal Component Analysis
Uploaded by
dsdamosPrincipal Component Analysis
Uploaded by
dsdamosOverview
Principal component analysis
Herve Abdi1 and Lynne J. Williams2
Principal component analysis (PCA) is a multivariate technique that analyzes a data
table in which observations are described by several inter-correlated quantitative
dependent variables. Its goal is to extract the important information from the table,
to represent it as a set of new orthogonal variables called principal components, and
to display the pattern of similarity of the observations and of the variables as points
in maps. The quality of the PCA model can be evaluated using cross-validation
techniques such as the bootstrap and the jackknife. PCA can be generalized
as correspondence analysis (CA) in order to handle qualitative variables and as
multiple factor analysis (MFA) in order to handle heterogeneous sets of variables.
Mathematically, PCA depends upon the eigen-decomposition of positive semidefinite matrices and upon the singular value decomposition (SVD) of rectangular
matrices. 2010 John Wiley & Sons, Inc. WIREs Comp Stat 2010 2 433459
rincipal component analysis (PCA) is probably the
most popular multivariate statistical technique
and it is used by almost all scientific disciplines. It
is also likely to be the oldest multivariate technique.
In fact, its origin can be traced back to Pearson1 or
even Cauchy2 [see Ref 3, p. 416], or Jordan4 and also
Cayley, Silverster, and Hamilton, [see Refs 5,6, for
more details] but its modern instantiation was formalized by Hotelling7 who also coined the term principal
component. PCA analyzes a data table representing
observations described by several dependent variables, which are, in general, inter-correlated. Its goal
is to extract the important information from the data
table and to express this information as a set of new
orthogonal variables called principal components.
PCA also represents the pattern of similarity of the
observations and the variables by displaying them as
points in maps [see Refs 810 for more details].
PREREQUISITE NOTIONS AND
NOTATIONS
Matrices are denoted in upper case bold, vectors are
denoted in lower case bold, and elements are denoted
in lower case italic. Matrices, vectors, and elements
Correspondence
from the same matrix all use the same letter (e.g.,
A, a, a). The transpose operation is denoted by the
superscriptT . The identity matrix is denoted I.
The data table to be analyzed by PCA comprises
I observations described by J variables and it is
represented by the I J matrix X, whose generic
element !is x"i,j . The matrix X has rank L where
L min I, J .
In general, the data table will be preprocessed
before the analysis. Almost always, the columns of X
will be centered so that the mean of each column
is equal to 0 (i.e., XT 1 = 0, where 0 is a J by
1 vector of zeros and 1 is an I by 1 vector of
ones).
addition, each element of X is divided
If in
by I (or I 1), the analysis is referred to as
a covariance PCA because, in this case, the matrix
XT X is a covariance matrix. In addition to centering,
when the variables are measured with different units,
it is customary to standardize each variable to unit
norm. This is obtained by dividing each variable by
its norm (i.e., the square root of the sum of all the
squared elements of this variable). In this case, the
analysis is referred to as a correlation PCA because,
then, the matrix XT X is a correlation matrix (most
statistical packages use correlation preprocessing as a
default).
The matrix X has the following singular value
decomposition [SVD, see Refs 1113 and Appendix B
for an introduction to the SVD]:
1 School
of Behavioral and Brain Sciences, The University of Texas
at Dallas, MS: GR4.1, Richardson, TX 75080-3021, USA
2 Department
of Psychology, University of Toronto Scarborough,
Ontario, Canada
DOI: 10.1002/wics.101
Volume 2, July /August 2010
X = P!QT
(1)
where P is the I L matrix of left singular vectors,
Q is the J L matrix of right singular vectors, and !
2010 John Wiley & Sons, Inc.
433
Overview
www.wiley.com/wires/compstats
is the diagonal matrix of singular values. Note that
!2 is equal to " which is the diagonal matrix of the
(nonzero) eigenvalues of XT X and XXT .
The inertia of a column is defined as the sum of
the squared elements of this column and is computed
as
2j =
I
#
x2i,j .
(2)
The sum of all the 2j is denoted I and it is called
the inertia of the data table or the total inertia. Note
that the total inertia is also equal to the sum of
the squared singular values of the data table (see
Appendix B).
The center of gravity of the rows [also called
centroid or barycenter, see Ref 14], denoted g, is the
vector of the means of each column of X. When X is
centered, its center of gravity is equal to the 1 J row
vector 0T .
The (Euclidean) distance of the i-th observation
to g is equal to
2
=
di,g
J
#
$
%2
xi,j gj .
(3)
When the data are centered Eq. 3 reduces to
2
=
di,g
J
#
x2i,j .
Finding the Components
In PCA, the components are obtained from the SVD
of the data table X. Specifically, with X = P!QT
(cf. Eq. 1), the I L matrix of factor scores, denoted
F, is obtained as:
F = P!.
GOALS OF PCA
The goals of PCA are to
(1) extract the most important information from the
data table;
(2) compress the size of the data set by keeping only
this important information;
(3) simplify the description of the data set; and
(4) analyze the structure of the observations and the
variables.
In order to achieve these goals, PCA computes
new variables called principal components which
are obtained as linear combinations of the original
variables. The first principal component is required
(5)
The matrix Q gives the coefficients of the linear
combinations used to compute the factors scores.
This matrix can also be interpreted as a projection
matrix because multiplying X by Q gives the values
of the projections of the observations on the principal
components. This can be shown by combining Eqs. 1
and 5 as:
F = P! = P!QT Q = XQ.
(4)
2
Note that the sum of all di,g
is equal to I which is the
inertia of the data table .
434
to have the largest possible variance (i.e., inertia and
therefore this component will explain or extract
the largest part of the inertia of the data table).
The second component is computed under the
constraint of being orthogonal to the first component
and to have the largest possible inertia. The other
components are computed likewise (see Appendix A
for proof). The values of these new variables for
the observations are called factor scores, and these
factors scores can be interpreted geometrically as the
projections of the observations onto the principal
components.
(6)
The components can also be represented
geometrically by the rotation of the original axes.
For example, if X represents two variables, the length
of a word (Y) and the number of lines of its dictionary
definition (W), such as the data shown in Table 1, then
PCA represents these data by two orthogonal factors.
The geometric representation of PCA is shown in
Figure 1. In this figure, we see that the factor scores
give the length (i.e., distance to the origin) of the
projections of the observations on the components.
This procedure is further illustrated in Figure 2. In
this context, the matrix Q is interpreted as a matrix
of direction cosines (because Q is orthonormal). The
matrix Q is also called a loading matrix. In this
context, the matrix X can be interpreted as the
product of the factors score matrix by the loading
matrix as:
X = FQT
with FT F = !2 and QT Q = I.
(7)
This decomposition is often called the bilinear
decomposition of X [see, e.g., Ref 15].
2010 Jo h n Wiley & So n s, In c.
Vo lu me 2, Ju ly /Au gu s t 2010
Volume 2, July /August 2010
Therefore
Generality
2010 John Wiley & Sons, Inc.
15
160
10
120
Blot
3.83
6.98
4.76
5.52
7.52
1.61
1.92
3.91
6.06
5.60
1.07
4.14
3.07
0.00
4.99
2.99
0.84
4.68
0.84
6.67
F1
2.30
2.07
1.84
1.23
1.23
2.53
1.15
1.30
2.07
2.38
1.69
0.92
0.77
0.00
0.38
2.84
0.54
1.76
0.54
0.69
F2
1
9.41
2.59
12
3
1
14
14.71
100
10
100
48.71
12
30.49
444
I
2
1
20
53
26
32
58
17
41
37
18
10
25
17
25
45
d2
52
5.29
4.29
3.39
1.51
1.51
6.41
1.32
1.70
4.29
5.65
2.85
0.85
0.59
0.00
0.15
8.05
0.29
3.11
0.29
0.48
F22
392
22.61
56.49
3.68
15.30
3
9
4
31.35
36.71
11
1.15
17.15
24.85
6
0
8.95
0.71
1
15
21.89
0.71
44.52
F12
6
0
ctr2 100
11
ctr1 100
74
92
87
95
97
29
74
90
90
85
29
95
94
99
53
71
88
71
99
cos21 100
26
13
71
26
10
10
15
71
47
29
12
29
cos22 100
MW = 8, MY = 6. The following abbreviations are used to label the columns: w = (W MW ); y = (Y MY ). The contributions and the squared cosines are multiplied by 100 for ease of reading. The positive
important contributions are italicized , and the negative important contributions are represented in bold.
Infectious
&
13
10
Arise
Pretentious
For
Neither
With
12
Scoundrel
11
Slope
Relief
This
Monastery
Solid
By
Insane
11
On
11
14
Across
Bag
Gravity, and Squared Cosines of the Observations for the Example Length of Words (Y) and Number of Lines (W)
TABLE 1 Raw Scores, Deviations from the Mean, Coordinates, Squared Coordinates on the Components, Contributions of the Observations to the Components, Squared Distances to the Center of
WIREs Computational Statistics
Principal component analysis
435
Overview
www.wiley.com/wires/compstats
Projecting New Observations onto the
Components
(a)
9
8
wsur
= Ysur MY = 3 6 = 3
Blot
Solid
This
On
For
Arise
Bag
By
9 10 11 12 13 14 15
Number of lines of the definition
Pretentious
Generality
4
Infectious
3
Scoundrel
2
Monastery
and
= Wsur MW = 12 8 = 4.
Then we plot the supplementary word in the graph
that we have already used for the active analysis.
Because the principal components and the original
variables are in the same space, the projections of the
supplementary observation give its coordinates (i.e.,
factor scores) on the components. This is shown in
Figure 3. Equivalently, the coordinates of the projections on the components can be directly computed
436
Insane
Slope
Relief
With
(8)
If the data table has been preprocessed (e.g., centered
or normalized), the same preprocessing should be
applied to the supplementary observations prior to
the computation of their factor scores.
As an illustration, suppose thatin addition to
the data presented in Table 1we have the French
word sur (it means on). It has Ysur = 3 letters, and
our French dictionary reports that its definition has
Wsur = 12 lines. Because sur is not an English word,
we do not want to include it in the analysis, but
we would like to know how it relates to the English
vocabulary. So, we decided to treat this word as a
supplementary observation.
The first step is to preprocess this supplementary
observation in a identical manner to the active
observations. Because the data matrix was centered,
the values of this observation are transformed into
deviations from the English center of gravity. We find
the following values:
ysur
Across
Neither
Therefore
Neither
1
7 6 5 4 3 2 1
Across
Insane
1 2 3 4
Relief Slope
With
2
2
xTsup Q.
Scoundrel
Therefore
(b)
T
fsup
Pretentious
Generality
Infectious
Monastery
11
10
Number of letters of the word
Equation 6 shows that matrix Q is a projection
matrix which transforms the original data matrix
into factor scores. This matrix can also be used to
compute factor scores for observations that were
not included in the PCA. These observations are
called supplementary or illustrative observations. By
contrast, the observations actually used to compute
the PCA are called active observations. The factor
scores for supplementary observations are obtained
by first positioning these observations into the PCA
space and then projecting them onto the principal
components. Specifically a 1 J row vector xTsup , can
be projected into the PCA space using Eq. 6. This
T ,
gives the 1 L vector of factor scores, denoted fsup
which is computed as:
Pretentious
Infectious
Therefore
Neither
On
Bag
By
Scoundrel
1
Monastery
7 6 5 4 3 2 1
Across
7
Blot
Solid
For
Generality
This
(c)
Arise
2
Solid
Insane Slope
1 2 3 4
Relief
With
2
3
For
This
Arise
5
Blot
Bag
6 7 1
On
By
FIGURE 1 | The geometric steps for finding the components of a
principal component analysis. To find the components (1) center the
variables then plot them against each other. (2) Find the main direction
(called the first component) of the cloud of points such that we have the
minimum of the sum of the squared distances from the points to the
component. Add a second component orthogonal to the first such that
the sum of the squared distances is minimum. (3) When the
components have been found, rotate the figure in order to position the
first component horizontally (and the second component vertically),
then erase the original axes. Note that the final graph could have been
obtained directly by plotting the observations from the coordinates
given in Table 1.
2010 Jo h n Wiley & So n s, In c.
Vo lu me 2, Ju ly /Au gu s t 2010
WIREs Computational Statistics
Principal component analysis
Pretentious
1
Neither
7 6 5 4 3 2 1
Across
1
2
3
Blot
.38
With
Solid
This
For
On
4
Infectious
3
Scoundrel
2
Neither
1
7 6 5 4 3 2 1
1 2 3 4 5 6
Insane
Arise
Relief Slope
Across
1
Bag
(b)
components. The projections (or coordinates) of the word neither on
the first and the second components are equal to 5.60 and 2.38.
from Eq. 8 (see also Table 3 for the values of
Q) as:
)
*
'
(
0.5369 0.8437
T
= xTsup Q = 3 4
fsup
0.8437 0.5369
'
(
= 4.9853 0.3835 .
(9)
INTERPRETING PCA
Inertia explained by a component
The importance of a component is reflected by its
inertia or by the proportion of the total inertia
explained by this factor. In our example (see
Table 2) the inertia of the first component is equal
to 392 and this corresponds to 83% of the total
inertia.
Contribution of an Observation to a
Component
Recall that the eigenvalue associated to a component
is equal to the sum of the squared factor scores
for this component. Therefore, the importance of an
observation for a component can be obtained by the
ratio of the squared factor score of this observation by
the eigenvalue associated with that component. This
ratio is called the contribution of the observation to the
component. Formally, the contribution of observation
i to component # is, denoted ctri,# , obtained as:
Volume 2, July /August 2010
With
Solid
This
For
On
Sur
By
Blot
Bag
Pretentious
(10)
0.38
4
Infectious
Monastery
3
4.99
Scoundrel
Therefore
2
Neither
1
7 6 5 4 3 2 1
1 2 3 4 5 6 7
Insane
Arise
Across
ReliefSlope
1
With
Blot
Solid
2
This
3
For
On
Bag 1
Sur
4
By
2
FIGURE 2 | Plot of the centered data, with the first and second
2
2
fi,#
fi,#
= #
=
2
#
fi,#
1 2 3 4 5 6
Insane
Arise
Relief Slope
By
Generality
ctri,#
Monastery
Therefore
4
Infectious
Monastery
3
5
.60
Scoundrel
Therefore
2
Pretentious
Generality
On second component
Generality
(a)
Projection of
neither
On first component
FIGURE 3 | How to find the coordinates (i.e., factor scores) on the
principal components of a supplementary observation: (a) the French
word sur is plotted in the space of the active observations from its
deviations to the W and Y variables; and (b) The projections of the sur
on the principal components give its coordinates.
where # is the eigenvalue of the #-th component.
The value of a contribution is between 0 and 1 and,
for a given component, the sum of the contributions
of all observations is equal to 1. The larger the
value of the contribution, the more the observation
contributes to the component. A useful heuristic
is to base the interpretation of a component on
the observations whose contribution is larger than
the average contribution (i.e., observations whose
contribution is larger than 1/I). The observations
with high contributions and different signs can then
be opposed to help interpret the component because
these observations represent the two endpoints of this
component.
The factor scores of the supplementary observations are not used to compute the eigenvalues and
therefore their contributions are generally not computed.
Squared Cosine of a Component with an
Observation
The squared cosine shows the importance of a
component for a given observation. The squared
2010 John Wiley & Sons, Inc.
437
Overview
www.wiley.com/wires/compstats
TABLE 2 Eigenvalues and Percentage of Explained Inertia by Each
Component
Component
i
(eigenvalue)
Cumulated
(eigenvalues)
Cumulated
(percentage)
392
392
83.29
83.29
52
444
11.71
100.00
cosine indicates the contribution of a component to
the squared distance of the observation to the origin.
It corresponds to the square of the cosine of the
angle from the right triangle made with the origin, the
observation, and its projection on the component and
is computed as:
2
2
fi,#
fi,#
= 2
cos2i,# = #
2
di,g
fi,#
(11)
2
is the squared distance of a given
where di,g
2
observation to the origin. The squared distance, di,g
, is
computed (thanks to the Pythagorean theorem) as the
sum of the squared values of all the factor scores of
this observation (cf. Eq. 4). Components with a large
value of cos2i,# contribute a relatively large portion to
the total distance and therefore these components are
important for that observation.
The distance to the center of gravity is defined for
supplementary observations and the squared cosine
can be computed and is meaningful. Therefore, the
value of cos2 can help find the components that are
important to interpret both active and supplementary
observations.
Loading: Correlation of a Component and a
Variable
The correlation between a component and a variable
estimates the information they share. In the PCA
framework, this correlation is called a loading. Note
that the sum of the squared coefficients of correlation
between a variable and all the components is equal
to 1. As a consequence, the squared loadings are easier
to interpret than the loadings (because the squared
loadings give the proportion of the variance of the
variables explained by the components). Table 3 gives
the loadings as well as the squared loadings for the
word length and definition example.
It is worth noting that the term loading has
several interpretations. For example, as previously
mentioned, the elements of matrix Q (cf. Eq. B.1)
are also called loadings. This polysemy is a potential
source of confusion, and therefore it is worth checking
438
Percent of
of Inertia
what specific meaning of the word loadings has been
chosen when looking at the outputs of a program or
when reading papers on PCA. In general, however,
different meanings of loadings lead to equivalent
interpretations of the components. This happens
because the different types of loadings differ mostly
by their type of normalization. For example, the
correlations of the variables with the components
are normalized such that the sum of the squared
correlations of a given variable is equal to one; by
contrast, the elements of Q are normalized such that
the sum of the squared elements of a given component
is equal to one.
Plotting the Correlations/Loadings of the
Variables with the Components
The variables can be plotted as points in the
component space using their loadings as coordinates.
This representation differs from the plot of the
observations: The observations are represented by
their projections, but the variables are represented by
their correlations. Recall that the sum of the squared
loadings for a variable is equal to one. Remember,
also, that a circle is defined as the set of points
with the property that the sum of their squared
coordinates is equal to a constant. As a consequence,
when the data are perfectly represented by only two
components, the sum of the squared loadings is equal
to one, and therefore, in this case, the loadings will
be positioned on a circle which is called the circle of
correlations. When more than two components are
needed to represent the data perfectly, the variables
will be positioned inside the circle of correlations.
The closer a variable is to the circle of correlations,
the better we can reconstruct this variable from the
first two components (and the more important it is to
interpret these components); the closer to the center
of the plot a variable is, the less important it is for the
first two components.
Figure 4 shows the plot of the loadings of the
variables on the components. Each variable is a point
whose coordinates are given by the loadings on the
principal components.
We can also use supplementary variables to
enrich the interpretation. A supplementary variable
should be measured for the same observations
2010 Jo h n Wiley & So n s, In c.
Vo lu me 2, Ju ly /Au gu s t 2010
WIREs Computational Statistics
Principal component analysis
TABLE 3 Loadings (i.e., Coefficients of Correlation between Variables and Components)
and Squared Loadings
Loadings
Component
1
0.9927
0.9810
0.9855
0.9624
0.8437
0.0145
0.0376
0.5369
1.0000
1.0000
0.1203
2
&
Squared Loadings
0.1939
0.8437
0.5369
The elements of matrix Q are also provided.
TABLE 4 Supplementary Variables for
TABLE 5 Loadings (i.e., Coefficients of Correlation) and Squared
the Example Length of Words and Number
of lines
Loadings between Supplementary Variables and Components
Frequency
Bag
# Entries
Loadings
Component
Across
230
On
700
12
2
&
Insane
By
500
Monastery
Relief
Slope
Scoundrel
700
With
Neither
Pretentious
Solid
This
500
For
900
Therefore
Generality
Arise
10
Blot
Infectious
0.3012
0.7218
# Entries
Frequency
# Entries
0.6999
0.0907
0.4899
0.4493
0.5210
0.2019
.6117
.6918
These data are shown in Table 4. A table of loadings
for the supplementary variables can be computed
from the coefficients of correlation between these
variables and the components (see Table 5). Note
that, contrary to the active variables, the squared
loadings of the supplementary variables do not add
up to 1.
STATISTICAL INFERENCE:
EVALUATING THE QUALITY
OF THE MODEL
Fixed Effect Model
Frequency is expressed as number of occurrences per 100,000 words, # Entries is
obtained by counting the number of entries
for the word in the dictionary.
used for the analysis (for all of them or part
of them, because we only need to compute a
coefficient of correlation). After the analysis has been
performed, the coefficients of correlation (i.e., the
loadings) between the supplementary variables and the
components are computed. Then the supplementary
variables are displayed in the circle of correlations
using the loadings as coordinates.
For example, we can add two supplementary
variables to the word length and definition example.
Volume 2, July /August 2010
Frequency
Squared Loadings
The results of PCA so far correspond to a fixed
effect model (i.e., the observations are considered
to be the population of interest, and conclusions
are limited to these specific observations). In this
context, PCA is descriptive and the amount of the
variance of X explained by a component indicates its
importance.
For a fixed effect model, the quality of the
PCA model using the first M components is obtained
by first computing the estimated matrix, denoted
+[M] , which is matrix X reconstituted with the first
X
M components. The formula for this estimation is
obtained by combining Eqs 1, 5, and 6 in order to
obtain
2010 John Wiley & Sons, Inc.
X = FQT = XQQT .
(12)
439
Overview
www.wiley.com/wires/compstats
(a)
+[M] . Several coefficients can be used
between X and X
for this task [see, e.g., Refs 1618]. The squared
coefficient of correlation is sometimes used, as well as
the RV coefficient.18,19 The most popular coefficient,
however, is the residual sum of squares (RESS). It is
computed as:
PC 2
Length
(number of letters)
Number of
lines of the
definition
+[M] &2
RESSM = &X X
,
= trace ET E
PC1
=I
(b)
(15)
#=1
where & & is the norm of X (i.e., the square root of the
sum of all the squared elements of X), and where the
trace of a matrix is the sum of its diagonal elements.
The smaller the value of RESS, the better the PCA
model. For a fixed effect model, a larger M gives a
+[M] . For a fixed effect model,
better estimation of X
the matrix X is always perfectly reconstituted with L
components (recall that L is the rank of X).
In addition, Eq. 12 can be adapted to compute
the estimation of the supplementary observations as
PC 2
Length
(number of letters)
M
#
Number of
lines of the
definition
PC1
# Entries
[M] [M]T
+
x[M]
Q
.
sup = xsup Q
Frequency
(16)
Random Effect Model
FIGURE 4 | Circle of correlations and plot of the loadings of (a) the
variables with principal components 1 and 2, and (b) the variables and
supplementary variables with principal components 1 and 2. Note that
the supplementary variables are not positioned on the unit circle.
+[M] is built back using Eq. 12
Then, the matrix X
keeping only the first M components:
+[M] = P[M] ![M] Q[M]T
X
= F[M] Q[M]T
= XQ[M] Q[M]T
(13)
where P[M] , ![M] , and Q[M] represent, respectively
the matrices P, !, and Q with only their first M
components. Note, incidentally, that Eq. 7 can be
rewritten in the current context as:
+[M] + E = F[M] Q[M]T + E
X=X
(14)
(where E is the error matrix, which is equal to
+[M] ).
XX
To evaluate the quality of the reconstitution of
X with M components, we evaluate the similarity
440
In most applications, the set of observations represents
a sample from a larger population. In this case, the
goal is to estimate the value of new observations from
this population. This corresponds to a random effect
model. In order to estimate the generalization capacity
of the PCA model, we cannot use standard parametric
procedures. Therefore, the performance of the PCA
model is evaluated using computer-based resampling
techniques such as the bootstrap and cross-validation
techniques where the data are separated into a learning
and a testing set. A popular cross-validation technique
is the jackknife (aka leave one out procedure). In the
jackknife,2022 each observation is dropped from the
set in turn and the remaining observations constitute
the learning set. The learning set is then used to
estimate (using Eq. 16) the left-out observation which
constitutes the testing set. Using this procedure, each
observation is estimated according to a random effect
model. The predicted observations are then stored in
.
a matrix denoted X.
The overall quality of the PCA random effect
model using M components is evaluated as the
.[M] . As with the fixed
similarity between X and X
effect model, this can also be done with a squared
coefficient of correlation or (better) with the RV
2010 Jo h n Wiley & So n s, In c.
Vo lu me 2, Ju ly /Au gu s t 2010
WIREs Computational Statistics
Principal component analysis
coefficient. Similar to RESS, one can use the predicted
residual sum of squares (PRESS). It is computed as:
.[M] &2
PRESSM = &X X
(17)
The smaller the PRESS the better the quality of the
estimation for a random model.
Contrary to what happens with the fixed effect
model, the matrix X is not always perfectly reconstituted with all L components. This is particularly the
case when the number of variables is larger than the
number of observations (a configuration known as the
small N large P problem in the literature).
How Many Components?
Often, only the important information needs to be
extracted from a data matrix. In this case, the problem
is to figure out how many components need to be
considered. This problem is still open, but there
are some guidelines [see, e.g.,Refs 9,8,23]. A first
procedure is to plot the eigenvalues according to their
size [the so called scree, see Refs 8,24 and Table 2]
and to see if there is a point in this graph (often called
an elbow) such that the slope of the graph goes from
steep to flat and to keep only the components
which are before the elbow. This procedure, somewhat
subjective, is called the scree or elbow test.
Another standard tradition is to keep only
the components whose eigenvalue is larger than the
average. Formally, this amount to keeping the #-th
component if
the optimal number of components to keep when the
goal is to generalize the conclusions of an analysis to
new data.
A simple approach stops adding components
when PRESS decreases. A more elaborated approach
[see e.g., Refs 2731] begins by computing, for each
component #, a quantity denoted Q2# is defined as:
Q2# = 1
PRESS#
RESS#1
(19)
with PRESS# (RESS# ) being the value of PRESS (RESS)
for the #-th component (where RESS0 is equal to
the total inertia). Only the components with Q2#
greater or equal to an arbitrary critical value (usually
1 0.952 = 0.0975) are kept [an alternative set of
critical values sets the threshold to 0.05 when I 100
and to 0 when I > 100; see Ref 28].
Another approachbased on cross-validation
to decide upon the number of components to keep uses
the index W# derived from Refs 32 and 33. In contrast
to Q2# , which depends on RESS and PRESS, the index
W# , depends only upon PRESS. It is computed for the
#-th component as
W# =
dfresidual, #
PRESS#1 PRESS#
,
PRESS#
df#
(20)
where PRESS0 is the inertia of the data table, df# is the
number of degrees of freedom for the #-th component
equal to
# >
1#
1
# = I
L
L
(18)
df# = I + J 2#,
(where L is the rank of X). For a correlation PCA,
this rule boils down to the standard advice to keep
only the eigenvalues larger than 1 [see, e.g., Ref
25]. However, this procedure can lead to ignoring
important information [see Ref 26 for an example of
this problem].
Random Model
As mentioned earlier, when using a random model,
the quality of the prediction does not always increase
with the number of components of the model. In fact,
when the number of variables exceeds the number
of observations, quality typically increases and then
decreases. When the quality of the prediction decreases
as the number of components increases this is an
indication that the model is overfitting the data (i.e.,
the information in the learning set is not useful to fit
the testing set). Therefore, it is important to determine
Volume 2, July /August 2010
(21)
and dfresidual, # is the residual number of degrees of
freedom which is equal to the total number of degrees
of freedom of the table [equal to J(I 1)] minus the
number of degrees of freedom used by the previous
components. The value of dfresidual, # is obtained as:
dfresidual, # = J(I 1)
#
#
k=1
(I + J 2k)
= J(I 1) #(I + J # 1).
(22)
Most of the time, Q2# and W# will agree on the number
of components to keep, but W# can give a more
conservative estimate of the number of components
to keep than Q2# . When J is smaller than I, the value
of both Q2L and WL is meaningless because they both
involve a division by zero.
2010 John Wiley & Sons, Inc.
441
Overview
www.wiley.com/wires/compstats
Bootstrapped Confidence Intervals
After the number of components to keep has been
determined, we can compute confidence intervals
. using the bootstrap.3439
for the eigenvalues of X
To use the bootstrap, we draw a large number of
samples (e.g., 1000 or 10,000) with replacement
from the learning set. Each sample produces a set
of eigenvalues. The whole set of eigenvalues can then
be used to compute confidence intervals.
variable tends to be associated with one (or a small
number) of the components, and each component
represents only a small number of variables. In
addition, the components can often be interpreted
from the opposition of few variables with positive
loadings to few variables with negative loadings.
Formally varimax searches for a linear combination
of the original factors such that the variance of the
squared loadings is maximized, which amounts to
maximizing
ROTATION
After the number of components has been determined,
and in order to facilitate the interpretation, the
analysis often involves a rotation of the components
that were retained [see, e.g., Ref 40 and 67, for
more details]. Two main types of rotation are used:
orthogonal when the new axes are also orthogonal
to each other, and oblique when the new axes are
not required to be orthogonal. Because the rotations
are always performed in a subspace, the new axes
will always explain less inertia than the original
components (which are computed to be optimal).
However, the part of the inertia explained by the
total subspace after rotation is the same as it was
before rotation (only the partition of the inertia has
changed). It is also important to note that because
rotation always takes place in a subspace (i.e., the
space of the retained components), the choice of this
subspace strongly influences the result of the rotation.
Therefore, it is strongly recommended to try several
sizes for the subspace of the retained components in
order to assess the robustness of the interpretation of
the rotation. When performing a rotation, the term
loadings almost always refer to the elements of matrix
Q. We will follow this tradition in this section.
Orthogonal Rotation
An orthogonal rotation is specified by a rotation
matrix, denoted R, where the rows stand for the
original factors and the columns for the new (rotated)
factors. At the intersection of row m and column n we
have the cosine of the angle between the original axis
and the new one: rm,n = cos m,n . A rotation matrix
has the important property of being orthonormal
because it corresponds to a matrix of direction cosines
and therefore RT R = I.
Varimax rotation, developed by Kaiser,41 is the
most popular rotation method. For varimax a simple
solution means that each component has a small
number of large loadings and a large number of zero
(or small) loadings. This simplifies the interpretation
because, after a varimax rotation, each original
442
(q2j,# q2# )2
(23)
with q2j,# being the squared loading of the j-th variable
of matrix Q on component # and q2# being the mean
of the squared loadings.
Oblique Rotations
With oblique rotations, the new axes are free to
take any position in the component space, but the
degree of correlation allowed among factors is small
because two highly correlated components are better
interpreted as only one factor. Oblique rotations,
therefore, relax the orthogonality constraint in order
to gain simplicity in the interpretation. They were
strongly recommended by Thurstone,42 but are used
more rarely than their orthogonal counterparts.
For oblique rotations, the promax rotation has
the advantage of being fast and conceptually simple.
The first step in promax rotation defines the target
matrix, almost always obtained as the result of a
varimax rotation whose entries are raised to some
power (typically between 2 and 4) in order to force
the structure of the loadings to become bipolar.
The second step is obtained by computing a least
square fit from the varimax solution to the target
matrix. Promax rotations are interpreted by looking
at the correlationsregarded as loadingsbetween
the rotated axes and the original variables. An
interesting recent development of the concept of
oblique rotation corresponds to the technique of
independent component analysis (ica) where the
axes are computed in order to replace the notion
of orthogonality by statistical independence [see Ref
43,for a tutorial].
When and Why Using Rotations
The main reason for using rotation is to facilitate the
interpretation. When the data follow a model (such
as the psychometric model) stipulating (1) that each
variable load on only one factor and (2) that there
is a clear difference in intensity between the relevant
2010 Jo h n Wiley & So n s, In c.
Vo lu me 2, Ju ly /Au gu s t 2010
WIREs Computational Statistics
Principal component analysis
TABLE 6 An (Artificial) Example of PCA using a Centered and Normalized Matrix
Hedonic
For
Meat
For
Dessert
Price
Sugar
Alcohol
Acidity
Wine 1
14
13
Wine 2
10
14
Wine 3
10
12
Wine 4
16
11
Wine 5
13
10
Five wines are described by seven variables [data from Ref 44].
factors (whose eigenvalues are clearly larger than one)
and the noise (represented by factors with eigenvalues
clearly smaller than one), then the rotation is likely
to provide a solution that is more reliable than the
original solution. However, if this model does not
accurately represent the data, then rotation will make
the solution less replicable and potentially harder to
interpret because the mathematical properties of PCA
have been lost.
Suppose that we have five wines described by the
average ratings of a set of experts on their hedonic
dimension, how much the wine goes with dessert, and
how much the wine goes with meat. Each wine is also
described by its price, its sugar and alcohol content,
and its acidity. The data [from Refs 40,44] are given
in Table 6.
A PCA of this table extracts four factors (with
eigenvalues of 4.76, 1.81, 0.35, and 0.07, respectively). Only two components have an eigenvalue
larger than 1 and, together, these two components
account for 94% of the inertia. The factor scores for
the first two components are given in Table 7 and the
corresponding map is displayed in Figure 5.
TABLE 7 PCA Wine Characteristics Factor scores, contributions
of the observations to the components, and squared cosines of the
observations on principal components 1 and 2.
F1
F2
ctr1
ctr2
cos21
cos22
1.17
0.55
29
17
77
17
0.61
23
21
69
24
0.08
0.19
34
Wine 4
0.89
17
41
50
46
Wine 5
1.23
0.86
32
20
78
19
Wine 3
1.04
Wine 5
Wine 3
PC1
Wine 1
Wine 4
observations plotted on the first two components. 1 = 4.76,
1 = 68%; 2 = 1.81, 2 = 26%.
Correlation PCA
Wine 2
PC2
FIGURE 5 | PCA wine characteristics. Factor scores of the
EXAMPLES
Wine 1
Wine 2
0.61
The positive important contributions are italicized, and the negative
important contributions are represented in bold. For convenience, squared
cosines and contributions have been multiplied by 100 and rounded.
Volume 2, July /August 2010
We can see from Figure 5 that the first component separates Wines 1 and 2 from Wines 4 and
5, while the second component separates Wines 2
and 5 from Wines 1 and 4. The examination of the
values of the contributions and cosines, shown in
Table 7, complements and refines this interpretation
because the contributions suggest that Component 1
essentially contrasts Wines 1 and 2 with Wine 5 and
that Component 2 essentially contrasts Wines 2 and
5 with Wine 4. The cosines show that Component
1 contributes highly to Wines 1 and 5, while
Component 2 contributes most to Wine 4.
To find the variables that account for these
differences, we examine the loadings of the variables
on the first two components (see Table 8) and the circle
of correlations (see Figure 6 and Table 9). From these,
we see that the first component contrasts price with
the wines hedonic qualities, its acidity, its amount
of alcohol, and how well it goes with meat (i.e., the
wine tasters preferred inexpensive wines). The second
component contrasts the wines hedonic qualities,
acidity, and alcohol content with its sugar content and
how well it goes with dessert. From this, it appears
that the first component represents characteristics that
are inversely correlated with a wines price while the
second component represents the wines sweetness.
To strengthen the interpretation, we can apply
a varimax rotation, which gives a clockwise rotation
2010 John Wiley & Sons, Inc.
443
Overview
www.wiley.com/wires/compstats
TABLE 8 PCA Wine Characteristics. Loadings (i.e., Q matrix) of the Variables on the
First Two Components
PC 1
For
For
Hedonic
Meat
Dessert
Price
Sugar
Alcohol
Acidity
0.40
0.45
0.26
0.42
0.05
0.44
0.45
0.11
PC 2
0.11
0.59
0.31
0.72
0.06
0.09
TABLE 9 PCA Wine Characteristics. Correlation of the Variables with the First Two
Components
For
PC 1
PC 2
For
Hedonic
meat
dessert
Price
Sugar
Alcohol
Acidity
0.87
0.97
0.58
0.91
0.11
0.96
0.99
0.15
0.15
0.79
PC 2
Hedonic
Acidity
Alcohol
PC1
For meat
Price
For dessert
Sugar
FIGURE 6 | PCA wine characteristics. Correlation (and circle of
correlations) of the Variables with Components 1 and 2. 1 = 4.76,
1 = 68%; 2 = 1.81, 2 = 26%.
of 15 (corresponding to a cosine of 0.97). This gives
the new set of rotated loadings shown in Table 10.
The rotation procedure is illustrated in Figure 7. The
improvement in the simplicity of the interpretation
is marginal, maybe because the component structure
of such a small data set is already very simple. The
first dimension remains linked to price and the second
dimension now appears more clearly as the dimension
of sweetness.
Covariance PCA
Here we use data from a survey performed in the
1950s in France [data from Ref 45]. The data table
gives the average number of Francs spent on several
categories of food products according to social class
and the number of children per family. Because a
Franc spent on one item has the same value as a Franc
444
0.42
0.97
0.07
0.12
spent on another item, we want to keep the same unit
of measurement for the complete space. Therefore
we will perform a covariance PCA, rather than a
correlation PCA. The data are shown in Table 11.
A PCA of the data table extracts seven components (with eigenvalues of 3,023,141.24, 290,575.84,
68,795.23, 25,298.95, 22,992.25, 3,722.32, and
723.92, respectively). The first two components
extract 96% of the inertia of the data table, and
we will keep only these two components for further
consideration (see also Table 14 for the choice of the
number of components to keep). The factor scores for
the first two components are given in Table 12 and
the corresponding map is displayed in Figure 8.
We can see from Figure 8 that the first
component separates the different social classes, while
the second component reflects the number of children
per family. This shows that buying patterns differ
both by social class and by number of children
per family. The contributions and cosines, given in
Table 12, confirm this interpretation. The values of the
contributions of the observations to the components
indicate that Component 1 contrasts blue collar
families with three children to upper class families
with three or more children whereas Component
2 contrasts blue and white collar families with
five children to upper class families with three and
four children. In addition, the cosines between the
components and the variables show that Component
1 contributes to the pattern of food spending seen
by the blue collar and white collar families with two
and three children and to the upper class families with
three or more children while Component 2 contributes
to the pattern of food spending by blue collar families
with five children.
To find the variables that account for these
differences, we refer to the squared loadings of the
variables on the two components (Table 13) and to
2010 Jo h n Wiley & So n s, In c.
Vo lu me 2, Ju ly /Au gu s t 2010
WIREs Computational Statistics
Principal component analysis
TABLE 10 PCA Wine Characteristics: Loadings (i.e., Q matrix), After Varimax
Rotation, of the Variables on the First Two Components
For
For
Hedonic
Meat
Dessert
Price
Sugar
Alcohol
Acidity
0.41
0.41
0.11
0.48
0.13
0.20
0.71
0.44
0.46
PC 1
PC 2
0.02
0.21
0.63
0.05
0.03
TABLE 11 Average Number of Francs Spent (per month) on Different Types of Food According To
Social Class and Number of Children [dataset from Ref 45]
Bread
Vegetables
332
Type of Food
Fruit Meat Poultry
Milk
Wine
354
247
427
Blue collar
2 Children
428
1437
526
White collar
2 Children
293
559
388
1527
567
239
258
Upper class
2 Children
372
767
562
1948
927
235
433
Blue collar
3 Children
406
563
341
1507
544
324
407
White collar
3 Children
386
608
396
1501
558
319
363
Upper class
3 Children
438
843
689
2345
1148
243
341
Blue collar
4 Children
534
660
367
1620
638
414
407
White collar
4 Children
460
699
484
1856
762
400
416
Upper class
4 Children
385
789
621
2366
1149
304
282
Blue collar
5 Children
655
776
423
1848
759
495
486
White collar
5 Children
584
995
548
2056
893
518
319
Upper class
5 Children
515
1097
887
2630
1167
561
284
447
732
505
1887
803
358
369
107
189
165
396
250
117
72
Mean
S
TABLE 12 PCA Example. Amount of Francs Spent (per month) by Food Type, Social Class,
and Number of Children. Factor scores, contributions of the observations to the components,
and squared cosines of the observations on principal components 1 and 2
F1
635.05
F2
ctr1
ctr2
cos21
cos22
120.89
13
95
142.33
86
139.75
26
40
12.05
100
Blue collar
2 Children
White collar
2 Children
488.56
Upper class
2 Children
Blue collar
3 Children
112.03
White collar
3 Children
485.94
1.17
98
Upper class
3 Children
12
89
4 Children
188.44
11
Blue collar
588.17
144.54
83
15
White collar
4 Children
57.51
42.86
40
22
Upper class
4 Children
206.76
15
86
11
5 Children
571.32
11
Blue collar
264.47
24
79
White collar
5 Children
235.92
19
57
36
Upper class
5 Children
296.04
97.15
33
97
520.01
333.95
39.38
992.83
The positive important contributions are italicized, and the negative important contributions are
represented in bold. For convenience, squared cosines and contributions have been multiplied by 100
and rounded.
Volume 2, July /August 2010
2010 John Wiley & Sons, Inc.
445
Overview
www.wiley.com/wires/compstats
(a)
PC 2
BC5
x2
UC5
Hedonic
Acidity
x1
Alcohol
PC1
WC5
UC3
BC3
WC3
UC4
For meat
BC4
WC4
UC2
BC2
WC2
FIGURE 8 | PCA example. Amount of Francs spent (per month) on
Price
food type by social class and number of children. Factor scores for
principal components 1 and 2. 1 = 3,023,141.24, 1 = 88%;
2 = 290,575.84, 2 = 8%. BC = blue collar; WC = white collar; UC
= upper class; 2 = 2 children; 3 = 3 children; 4 = 4 children; 5 = 5
children..
For dessert
Sugar
TABLE 13 PCA example: Amount of Francs Spent (per
x2
(b)
month) on Food Type by Social Class and Number of Children.
Squared Loadings of the Variables on Components 1 and 2
y2
Bread Vegetables Fruit Meat Poultry Milk Wine
Hedonic
Acidity
Alcohol
x1
= 15
For meat
Price
y1
For dessert
Sugar
(c)
y2
Hedonic
Acidity
y1
Alcohol
Price
For meat
For dessert
Sugar
FIGURE 7 | PCA wine characteristics. (a) Original loadings of the
seven variables. (b) The loadings of the seven variables showing the
original axes and the new (rotated) axes derived from varimax. (c) The
loadings after varimax rotation of the seven variables.
446
PC 1 0.01
0.33
0.16 0.01
0.03
0.45 0.00
PC 2 0.11
0.17
0.09 0.37
0.18
0.03 0.06
the circle of correlations (see Figure 9). From these,
we see that the first component contrasts the amount
spent on wine with all other food purchases, while
the second component contrasts the purchase of milk
and bread with meat, fruit, and poultry. This indicates
that wealthier families spend more money on meat,
poultry, and fruit when they have more children, while
white and blue collar families spend more money on
bread and milk when they have more children. In
addition, the number of children in upper class families
seems inversely correlated with the consumption of
wine (i.e., wealthy families with four or five children
consume less wine than all other types of families).
This curious effect is understandable when placed in
the context of the French culture of the 1950s, in
which wealthier families with many children tended
to be rather religious and therefore less inclined to
indulge in the consumption of wine.
Recall that the first two components account
for 96% of the total inertia [i.e., (1 + 2 )/I =
(3,023,141.24 + 290,575.84)/3,435,249.75 = 0.96].
From Table 14 we find that RESS2 is equal to 4% and
+ is estimated
this value represents the error when X
from Components 1 and 2 together. This means that
for a fixed effect model, a two-component solution
represents X well. PRESS2 , the error of estimation
using a random effect model with two components, is
. represents
equal to 8% and this value indicates that X
X adequately. Together, the values of RESS and
PRESS suggest that only the first two components
should be kept.
2010 Jo h n Wiley & So n s, In c.
Vo lu me 2, Ju ly /Au gu s t 2010
WIREs Computational Statistics
Principal component analysis
the rows and one for the columns. These factor scores
are, in general, scaled such that their inertia is equal to
their eigenvalue (some versions of CA compute row or
column factor scores normalized to unity). The grand
total of the table is denoted N.
Bread PC2
Milk
Vegetable
Wine
Computations
PC1
The first step of the analysis is to compute the
probability matrix Z = N1 X. We denote r the vector
of the row totals of Z, (i.e., r = Z1, with 1 being a
conformable vector of 1s), c the vector of the columns
totals, and Dc = diag {c}, Dr = diag {r}. The factor
scores are obtained from the following generalized
SVD (see Appendix ):
Meat
Fruit
Poultry
FIGURE 9 | PCA example: Amount of Francs spent (per month) on
food type by social class and number of children. Correlations (and
circle of correlations) of the variables with Components 1 and 2.
1 = 3,023,141.24, 1 = 88%; 2 = 290,575.84, 2 = 8%.
To confirm the number of components to keep,
we look at Q2 and W. The Q2 values of 0.82 and 0.37
for Components 1 and 2 both exceed the critical value
of 0.095, indicating that both components should be
kept. Note that a negative Q2 value suggests that a
component should not be kept. In contrast, the W
values of 1.31 and 0.45 for the first two components
suggest that only the first component should be kept
because only W1 is greater than 1.
SOME EXTENSIONS OF PCA
Correspondence Analysis
Correspondence analysis (CA; see Refs 4651) is
an adaptation of PCA tailored to handle nominal
variables. It can be interpreted as a particular case
of generalized PCA for which we take into account
masses (for the rows) and weights (for the columns).
CA analyzes a contingency table and provides factor
scores for both the rows and the columns of
the contingency table. In CA, the inertia of the
contingency table is proportional to the 2 which
can be computed to test the independence of the
rows and the columns of this table. Therefore, the
factor scores in CA decompose this independence 2
into orthogonal components (in the CA tradition,
these components are often called factors rather than
components, here, for coherence, we keep the name
component for both PCA and CA).
Notations
The I J contingency table to be analyzed is denoted
X. CA will provide two sets of factor scores: one for
Volume 2, July /August 2010
0
. T with .
. . T 1 .
.Q
P!
Z rcT = .
PT D1
r P = Q Dc Q = I
(24)
The row and (respectively) column factor scores are
obtained as:
..
F = D1
r P!
and
..
G = D1
c Q!
(25)
In CA, the rows and the columns of the
table have a similar role, and therefore we have
contributions and cosines for both sets. These are
obtained in a similar way as for standard PCA,
but the computations of the contributions need to
integrate the values of the masses (i.e., the elements
of r) and weights (i.e., the elements of c). Specifically,
the contribution of row i to component # and of
column j to component # are obtained respectively
as:
ctri,# =
2
ri fi,#
and
ctrj,# =
2
cj gj,#
(26)
(with ri begin the ith element of r and cj being the
j-th element of c). As for standard PCA, contributions
help locating the observations or variables important
for a given component.
The vector of the squared ( 2 ) distance from the
rows and columns to their respective barycenter are
obtained as:
,
,
and
dc = diag GGT
dr = diag FFT
(27)
As for PCA, the total inertia in CA is equal to the
sum of the eigenvalues. By contrast with PCA, the
total inertia can also be computed equivalently as the
weighted sum of the squared distances of the rows or
2010 John Wiley & Sons, Inc.
447
Overview
www.wiley.com/wires/compstats
TABLE 14 PCA Example: Amount of Francs Spent (per month) on Food Type by Social Class and Number of children. Eigenvalues,
cumulative eigenvalues, RESS, PRESS, Q2 , and W values for all seven components
&
&
PRESS/I
Q2
610,231.19
0.18
0.82
1.31
259,515.13
0.08
0.37
0.45
0.02
155,978.58
0.05
0.27
27,438.49
0.01
152,472.37
0.04
0.28
4,446.25
0.00
54,444.52
0.02
1.35
1.00
723.92
0.00
7,919.49
0.00
0.98
1.00
0.00
0.00
0.00
0.00
1.00
/I
RESS
0.88
412,108.51
0.12
0.96
121,532.68
0.04
3,382,512.31
0.98
52,737.44
0.01
3,407,811.26
0.99
0.01
3,430,803.50
1.00
3,722.32
0.00
3,434,525.83
723.92
0.00
3,435,249.75
3,435,249.75
1.00
/I
PC 1
3,023,141.24
0.88
3,023,141.24
PC 2
290,575.84
0.08
3,313,717.07
PC 3
68,795.23
0.02
PC 4
25,298.95
PC 5
22,992.25
PC 6
PC 7
&
PRESS
1.89
0.01
0.78
8.22
the columns to their respective barycenter. Formally,
the inertia can be computed as:
I=
L
#
l
# = rT dr = cT dc
(28)
The squared cosine between row i and component # and column j and component # are obtained
respectively as:
cos2i,# =
2
fi,#
2
dr,i
and
cos2j,# =
2
gj,#
2
dc,j
(29)
2
2
(with dr,i
and dc,j
, being respectively the i-th element
of dr and the j-th element of dc ). Just like for
PCA, squared cosines help locating the components
important for a given observation or variable.
And just like for PCA, supplementary or
illustrative elements can be projected onto the
components, but the CA formula needs to take into
account masses and weights. The projection formula,
is called the transition formula and it is specific to CA.
Specifically, let iTsup being an illustrative row and jsup
being an illustrative column to be projected (note that
in CA, prior to projection, a illustrative row or column
is re-scaled such that its sum is equal to one). Their
coordinates of the illustrative rows (denoted fsup ) and
column (denoted gsup ) are obtained as:
/
01
. 1 and gsup
fsup = iTsup 1
iTsup G!
/
01
. 1
= jTsup 1
jTsup F!
(30)
/
01
/
01
[note that the scalar terms iTsup 1
and jTsup 1
are used to ensure that the sum of the elements of isup
448
RESS/I
or jsup is equal to one, if this is already the case, these
terms are superfluous].
Example
For this example, we use a contingency table that
gives the number of punctuation marks used by the
French writers Rousseau, Chateaubriand, Hugo, Zola,
Proust, and Giraudoux [data from Ref 52]. This table
indicates how often each writer used the period, the
comma, and all other punctuation marks combined
(i.e., interrogation mark, exclamation mark, colon,
and semicolon). The data are shown in Table 15.
A CA of the punctuation table extracts two
components which together account for 100% of
the inertia (with eigenvalues of .0178 and .0056,
respectively). The factor scores of the observations
(rows) and variables (columns) are shown in Tables 16
and the corresponding map is displayed in Figure 10.
We can see from Figure 10 that the first
component separates Proust and Zolas pattern of
punctuation from the pattern of punctuation of
the other four authors, with Chateaubriand, Proust,
and Zola contributing most to the component.
The squared cosines show that the first component
accounts for all of Zolas pattern of punctuation (see
Table 16).
The second component separates Giraudouxs
pattern of punctuation from that of the other authors.
Giraudoux also has the highest contribution indicating
that Giraudouxs pattern of punctuation is important
for the second component. In addition, for Giraudoux
the highest squared cosine (94%), is obtained for
Component 2. This shows that the second component
is essential to understand Giraudouxs pattern of
punctuation (see Table 16).
In contrast with PCA, the variables (columns)
in CA are interpreted identically to the rows. The
factor scores for the variables (columns) are shown in
2010 Jo h n Wiley & So n s, In c.
Vo lu me 2, Ju ly /Au gu s t 2010
WIREs Computational Statistics
Principal component analysis
TABLE 15 The Punctuation Marks of Six French Writers (from Ref 52).
Authors Name
Period
Comma
Other
xi+
Rousseau
7,836
13,112
6,026
26,974
Chateaubriand
r
0.0189
53,655
102,383
42,413
198,451
0.1393
Hugo
115,615
184,541
59,226
359,382
0.2522
Zola
161,926
340,479
62,754
565,159
0.3966
Proust
38,177
105,101
12,670
155,948
0.1094
Giraudoux
46,371
58,367
14,299
119,037
0.0835
x+j
423,580
803,983
197,388
N = 142,4951
1.0000
cT
0.2973
0.5642
0.1385
The column labeled xi+ gives the total number of punctuation marks used by each author. N is the
grand total of the data table. The vector of mass for the rows, r, is the proportion of punctuation marks
used by each author (ri = xi+ /N). The row labeled x+j gives the total number of times each punctuation
mark was used. The centroid row, cT , gives the proportion of each punctuation mark in the sample
(cj = x+j /N).
TABLE 16 CA punctuation. Factor scores, contributions, mass, mass squared factor scores, inertia to barycenter, and
squared cosines for the rows.
Rousseau
Chateaubriand
Hugo
F1
F2
0.24
0.07
0.10
0.03
0.00
0.19
0.09
Zola
0.22
Proust
Giraudoux
&
0.05
ri
ri
F12
F22
2
dr,i
0.0189
0.0011
0.0001
0.0012
28
29
0.1393
0.0050
0.0016
15
.2522
0.0027
0.0002
ctr2
0.11
ri
ri
ctr1
cos21
cos22
91
0.0066
76
24
0.0029
92
19
.3966
0.0033
0.0000
0.0033
100
0.06
31
.1094
0.0055
0.0004
.0059
93
0.20
58
0.0835
0.0002
0.0032
0.0034
94
100
100
.0178
.0056
.0234
76%
24%
The positive important contributions are italicized, and the negative important contributions are represented in bold. For convenience,
squared cosines and contributions have been multiplied by 100 and rounded.
PC2
Chateaubriand
Proust
Comma
PC1
Zola
Rousseau
Hugo
Other
marks
Period
Giraudoux
FIGURE 10 | CA punctuation. The projections of the rows and the
columns are displayed in the same map. 1 = 0.0178, 1 = 76.16;
2 = 0.0056, 2 = 23.84.
Volume 2, July /August 2010
Table 17 and the corresponding map is displayed in
the same map as the observations shown in Figure 10.
From Figure 10 we can see that the first
component also separates the comma from the
others punctuation marks. This is supported by the
high contributions of others and comma to the component. The cosines also support this interpretation
because the first component accounts for 88% of the
use of the comma and 91% of the use of the others
punctuation marks (see Table 17).
The second component separates the period
from both the comma and the other punctuation
marks. This is supported by the periods high contribution to the second component and the components
contribution to the use of the period (see Table 17).
2010 John Wiley & Sons, Inc.
449
Overview
www.wiley.com/wires/compstats
TABLE 17 CA punctuation. Factor scores, contributions, mass, mass squared factor scores, inertia to
barycenter, and squared cosines for the columns
F1
Period
Comma
Other
&
F2
0.05
0.10
0.29
ctr2
cj
cj
F12
F22
cj
2
dc,j
cos21
cos22
0.11
66
.2973
0.0007
0.0037
0.0044
16
84
0.04
30
14
.5642
0.0053
0.0008
0.0061
88
12
66
20
.1385
0.0118
0.0011
0.0129
91
100
100
.0178
0.0056
0.0234
I
0.09
ctr1
cj
76%
24%
The positive important contributions are italicized, and the negative important contributions are represented in bold. For
convenience, squared cosines, and contributions have been multiplied by 100 and rounded.
Together, the pattern of distribution of the
points representing the authors and the punctuation
marks suggest that some of the differences in the
authors respective styles can be attributed to differences in their use of punctuation. Specifically, Zolas
uvre is characterized by his larger than average use
of the comma, while Chateaubriands is characterized
by his larger than average use of other types of
punctuation marks than the period and the comma.
In addition, Giraudouxs uvre is characterized by a
larger than average use of the period.
Multiple Factor Analysis
Multiple factor analysis [MFA; see Refs 5355] is
used to analyze a set of observations described by
several groups of variables. The number of variables
in each group may differ and the nature of the
variables (nominal or quantitative) can vary from
one group to the other but the variables should be of
the same nature in a given group. The analysis derives
an integrated picture of the observations and of the
relationships between the groups of variables.
Notations
The data consists of T data sets. Each data set is called
a subtable. Each subtable is an I [t] J rectangular
data matrix, denoted [t]Y, where I is the number of
observations and [t] J the number of variables of the
t-th subtable. The total number of variables is equal
to J, with:
J=
[t] J
(31)
Each subtable is preprocessed (e.g., centered and
normalized) and the preprocessed data matrices
actually used in the analysis are denoted [t]X.
450
The #-th eigenvalue of the t-th subtable is
denoted [t] (# . The #-th singular value of the t-th
subtable is denoted [t] # .
Computations
The goal of MFA is to integrate different groups
of variables (i.e., different subtables) describing the
same observations. In order to do so, the first step is
to make these subtables comparable. Such a step is
needed because the straightforward analysis obtained
by concatenating all variables would be dominated
by the subtable with the strongest structure (which
would be the subtable with the largest first singular
value). In order to make the subtables comparable,
we need to normalize them. To normalize a subtable,
we first compute a PCA for this subtable. The first
singular value (i.e., the square root of the first
eigenvalue) is the normalizing factor which is used
to divide the elements of this subtable. So, formally,
The normalized subtables are computed as:
[t]Z
[t] ( 1
[t]X
[t] 1
[t]X
(32)
The normalized subtables are concatenated
into an I J matrix called the global data matrix
and denoted Z. A PCA is then run on Z to get a
global solution. Note that because the subtables have
previously been centered and normalized with their
first singular value, Z is centered but it is not normalized (i.e., columns from different subtables have, in
general, different norms).
To find out how each subtable performs relative
to the global solution, each subtable (i.e., each [t]X)
is projected into the global space as a supplementary
element.
As in standard PCA, variable loadings are
correlations between original variables and global
2010 Jo h n Wiley & So n s, In c.
Vo lu me 2, Ju ly /Au gu s t 2010
WIREs Computational Statistics
Principal component analysis
factor scores. To find the relationship between the
variables from each of the subtables and the global
solution we compute loadings (i.e., correlations)
between the components of each subtable and the
components of the global analysis.
(a)
PC 2
4
6
Example
Suppose that three experts were asked to rate six
wines aged in two different kinds of oak barrel from
the same harvest of Pinot Noir [example from Ref 55].
Wines 1, 5, and 6 were aged with a first type of oak,
and Wines 2, 3, and 4 with a second type of oak. Each
expert was asked to choose from 2 to 5 variables to
describe the wines. For each wine, the expert rated the
intensity of his/her variables on a 9-point scale. The
data consist of T = 3 subtables, which are presented
in Table 18.
The PCAs on each of the three subtables
extracted eigenvalues of 1 (1 = 2.86, 2 (1 = 3.65,
and 3 (1 = 2.50 with singular values of 1 1 = 1.69,
2 1 = 1.91, and 3 1 = 1.58, respectively.
Following normalization and concatenation of
the subtables, the global PCA extracted five components (with eigenvalues of 2.83, 0.36, 0.11, 0.03, and
0.01). The first two components explain 95% of the
inertia. The factor scores for the first two components
of the global analysis are given in Table 19 and the
corresponding map is displayed in Figure 11a.
We can see from Figure 11 that the first
component separates the first type of oak (Wines 1, 5,
and 6) from the second oak type (Wines 2, 3, and 4).
In addition to examining the placement of the
wines, we wanted to see how each experts ratings
fit into the global PCA space. We achieved this by
projecting the data set of each expert as a supplementary element [see Ref 18 for details of the procedure].
The factor scores are shown in Table 19. The experts
placement in the global map is shown in Figure 11b.
Note that the position of each wine in the global analysis is the center of gravity of its position for the experts.
The projection of the experts shows that Expert 3s
ratings differ from those of the other two experts.
The variable loadings show the correlations
between the original variables and the global factor
scores (Table 20). These loadings are plotted in
Figure 12. This figure also represents the loadings (Table 21) between the components of each
subtable and the components of the global analysis
as the circle of correlations specific to each expert.
From this we see that Expert 3 differs from the other
experts, and is mostly responsible for the second
component of the global PCA.
Volume 2, July /August 2010
5
PC1
(b)
PC 2
4
6
5
PC1
FIGURE 11 | MFA wine ratings and oak type. (a) Plot of the global
analysis of the wines on the first two principal components. (b)
Projection of the experts onto the global analysis. Experts are
represented by their faces. A line segment links the position of the wine
for a given expert to its global position. 1 = 2.83, 1 = 84%;
2 = 2.83, 2 = 11%.
CONCLUSION
PCA is very versatile, it is the oldest and remains the
most popular technique in multivariate analysis. In
addition to the basics presented here, PCA can also
be interpreted as a neural network model [see, e.g.,
Refs 56,57]. In addition to CA, covered in this paper,
generalized PCA can also be shown to incorporate
a very large set of multivariate techniques such as
2010 John Wiley & Sons, Inc.
451
Overview
www.wiley.com/wires/compstats
TABLE 18 Raw Data for the Wine Example [from Ref 55
Expert 1
Wines
Oak Type
Fruity
Woody
Expert 2
Coffee
Red Fruit
Roasted
Expert 3
Vanillin
Woody
Fruity
Butter
Woody
Wine 1
Wine 2
Wine 3
Wine 4
Wine 5
Wine 6
canonical variate analysis, linear discriminant analysis
[see, e.g., Ref 47], and barycentric discriminant
analysis techniques such as discriminant CA [see e.g.,
Refs 5861].
the maximum (or minimum) of functions involving
these matrices. Specifically PCA is obtained from the
eigen-decomposition of a covariance or a correlation
matrix.
APPENDIX A: EIGENVECTORS AND
EIGENVALUES
Notations and Definition
There are several ways to define eigenvectors and
eigenvalues, the most common approach defines an
eigenvector of the matrix A as a vector u that satisfies
the following equation:
Eigenvectors and eigenvalues are numbers and vectors
associated to square matrices. Together they provide
the eigen-decomposition of a matrix, which analyzes
the structure of this matrix. Even though the eigendecomposition does not exist for all square matrices,
it has a particularly simple expression for matrices
such as correlation, covariance, or cross-product
matrices. The eigen-decomposition of this type of
matrices is important because it is used to find
Au = u .
(A.1)
When rewritten, the equation becomes:
(A I)u = 0,
(A.2)
TABLE 19 MFA Wine Ratings and Oak Type. Factor Scores for the Global Analysis, Expert 1,
Expert 2, and Expert 3 for the First Two Components
Expert 1sup
[1] F2
Global
F1
F2
Wine 1
2.18
Wine 2
0.56
Wine 3
Wine 4
Wine 5
0.51
0.20
2.32
0.83
1.40
0.05
1.13
0.58
1.83
Wine 6
Expert 2sup
[1] F1
0.90
2.76
0.77
1.10
0.30
0.81
1.99
1.98
1.29
0.69
sup, supplementary element.
[2] F1
0.93
0.62
0.30
Expert 3sup
[3] F2
[2] F2
2.21
[3] F1
0.86
0.28
0.13
2.39
1.23
0.50
2.11
1.49
0.44
0.76
2.85
3.80
1.43
1.27
1.62
2.28
1.12
0.49
1.08
1.54
0.61
0.24
0.56
TABLE 20 MFA Wine Ratings and Oak Type. Loadings (i.e., correlations) on the Principal Components of the Global Analysis of the
Original Variables. Only the First Three Dimensions are Kept
Loadings with Original Variables
Expert 2
Expert 1
452
PC
(%)
Fruity
Woody
2.83
85
0.98
0.36
11
0.97
0.12
0.23
0.02
0.15
0.02
Coffee
0.92
0.06
0.37
Fruit
0.89
0.38
0.21
Roasted
Vanillin
0.96
0.95
0.00
0.20
0.28
0.00
2010 Jo h n Wiley & So n s, In c.
Expert 3
Woody
Fruity
Butter
Woody
0.97
0.59
0.95
0.99
0.19
0.00
0.08
0.24
0.11
0.10
0.14
0.80
Vo lu me 2, Ju ly /Au gu s t 2010
WIREs Computational Statistics
Principal component analysis
TABLE 21 MFA Wine Ratings and Oak Type. Loadings (i.e., correlations) on the
Principal Components of the Global Analysis of the Principal Components of the Subtable
PCAs
Loadings with First Two Components from Subtable PCAs
Expert 1
Expert 2
Expert 3
PC
(%)
[1] PC1
2.83
85
.98
0.36
11
0.12
0.15
0.14
[1] PC2
[2] PC1
[2] PC2
[3] PC1
[3] PC2
0.08
0.99
0.13
0.16
0.94
0.28
0.35
0.58
0.05
0.84
0.09
0.76
Only the first three dimensions are kept.
where is a scalar called the eigenvalue associated to
the eigenvector.
In a similar manner, we can also say that a vector
u is an eigenvector of a matrix A if the length of the
vector (but not its direction) is changed when it is
multiplied by A.
For example, the matrix:
)
*
2 3
A=
2 1
(A.3)
has for eigenvectors:
) *
3
u1 =
2
with eigenvalue
1 = 4
(A.4)
and
0.35
0.94
.01
For the previous example we obtain:
A = U"U1
)
*)
*)
*
3 1
4
0
2 2
=
2
1
0 1
4 6
)
*
2 3
=
(A.9)
2 1
Together, the eigenvectors and the eigenvalues
of a matrix constitute the eigen-decomposition of this
matrix. It is important to note that not all matrices
have an eigen-decomposition.
This is the case, e.g.,
)
*
0 1
of the matrix
. Also some matrices can have
0 0
imaginary eigenvalues and eigenvectors.
Positive (semi-)Definite Matrices
)
*
1
u2 =
1
with eigenvalue
2 = 1
(A.5)
For most applications we normalize the eigenvectors (i.e., transform them such that their length is
equal to one), therefore
uT u = 1
(A.6)
Traditionally, we put the set of eigenvectors
of A in a matrix denoted U. Each column of U is
an eigenvector of A. The eigenvalues are stored in
a diagonal matrix (denoted "), where the diagonal
elements gives the eigenvalues (and all the other values
are zeros). We can rewrite the Eq. A.1 as:
AU = "U;
(A.7)
A = U"U1 .
(A.8)
or also as:
Volume 2, July /August 2010
A type of matrices used very often in statistics are
called positive semi-definite. The eigen-decomposition
of these matrices always exists, and has a particularly
convenient form. A matrix is said to be positive semidefinite when it can be obtained as the product of a
matrix by its transpose. This implies that a positive
semi-definite matrix is always symmetric. So, formally,
the matrix A is positive semi-definite if it can be
obtained as:
A = XXT
(A.10)
for a certain matrix X (containing real numbers).
In particular, correlation matrices, covariance, and
cross-product matrices are all positive semi-definite
matrices.
The important properties of a positive semidefinite matrix is that its eigenvalues are always
positive or null, and that its eigenvectors are pairwise
orthogonal when their eigenvalues are different.
The eigenvectors are also composed of real values
(these last two properties are a consequence of
the symmetry of the matrix, for proofs see, e.g.,
2010 John Wiley & Sons, Inc.
453
Overview
www.wiley.com/wires/compstats
(a)
Refs 62,63). Because eigenvectors corresponding to
different eigenvalues are orthogonal, it is possible to
store all the eigenvectors in an orthogonal matrix
(recall that a matrix is orthogonal when the product
of this matrix by its transpose is a diagonal matrix).
This implies the following equality:
PC 2
Fruity
PC1
1PC2
Coffee
1PC1
Woody
U1 = UT .
We can, therefore, express the positive semi-definite
matrix A as:
A = U"UT
PC2
2PC2
Red Fruit
Woody
A = U"UT
2PC1
Vanillin
with
3
Expert 2
3PC2
PC2
3PC1
Butter
Woody
PC1
Fruity
Expert 3
FIGURE 12 | MFA wine ratings and oak type. Circles of correlations
for the original variables. Each experts variables have been separated
for ease of interpretation. .
454
1
32
1
2
3
*
1
4 0 32
0 2
1
1
32
1
2
1
32
12
)
*
3 1
=
1 3
(c)
(A.12)
can be decomposed as:
Roasted
PC1
UT U = I
with
where U is the matrix storing the normalized
eigenvectors; if these are not normalized then UT U
is a diagonal matrix.
For example, the matrix:
)
*
3 1
A=
(A.13)
1 3
Expert 1
(b)
(A.11)
1
32
12
1
32
1
2
1
32
12
1
32
12
(A.14)
)
*
= 1 0 (A.15)
0 1
Statistical Properties of the
Eigen-decomposition
The eigen-decomposition is important because it is
involved in problems of optimization. Specifically, in
PCA, we want to find row factor scores, obtained as
linear combinations of the columns of X such that
these factor scores explain as much of the variance
of X as possible and such that the sets of factor scores
are pairwise orthogonal. We impose as a constraint
that the coefficients of the linear combinations are
finite and this constraint is, in general, expressed as
imposing to the sum of squares of the coefficients of
each linear combination to be equal to unity. This
amounts to defining the factor score matrix as:
2010 Jo h n Wiley & So n s, In c.
F = XQ,
(A.16)
Vo lu me 2, Ju ly /Au gu s t 2010
WIREs Computational Statistics
Principal component analysis
(with the matrix Q being the matrix of coefficients
of the to-be-found linear combinations) under the
constraints that
T
F F = Q X XQ
(A.17)
is a diagonal matrix (i.e., F is an orthogonal matrix)
and that
QT Q = I
(A.18)
(i.e., Q is an orthonormal matrix). The solution of
this problem can be obtained with the technique of
the Lagrangian multipliers where the constraint from
Eq. A.18 is expressed as the multiplication with a
diagonal matrix of Lagrangian multipliers, denoted
", in order to give the following expression
0
/
" QT Q I
(A.19)
(see Refs 62,64; for details). This amount for defining
the following equation
,
/
0L = trace FT F " QT Q I
(A.20)
/
0,
= trace QT XT XQ " QT Q I
(where the trace {} operator gives the sum of the
diagonal elements of a square matrix). In order to find
the values of Q which give the maximum values of L,
we first compute the derivative of L relative to Q:
L
= 2XT XQ 2Q",
Q
(A.21)
and then set this derivative to zero:
T
X XQ Q" = 0 X XQ = Q".
(A.22)
This implies also that
XT X = Q"QT .
(A.23)
Because " is diagonal, this is clearly an eigendecomposition problem, and this indicates that " is
the matrix of eigenvalues of the positive semi-definite
matrix XT X ordered from the largest to the smallest
and that Q is the matrix of eigenvectors of XT X
associated to ". Finally, we find that the factor matrix
has the form
F = XQ.
Volume 2, July /August 2010
(A.24)
The variance of the factors scores is equal to the
eigenvalues because:
FT F = QT XT XQ = QT Q"QT Q = ".
(A.25)
Taking into account that the sum of the eigenvalues
is equal to the trace of XT X, this shows that the first
factor scores extract as much of the variance of the
original data as possible, and that the second factor
scores extract as much of the variance left unexplained
by the first factor, and so on for the remaining factors.
1
Incidently, the diagonal elements of the matrix " 2
which are the standard deviations of the factor scores
are called the singular values of matrix X (see Section
on Singular Value Decomposition).
APPENDIX B: SINGULAR VALUE
DECOMPOSITION (SVD)
The SVD is a generalization of the eigendecomposition. The SVD decomposes a rectangular
matrix into three simple matrices: two orthogonal
matrices and one diagonal matrix. If A is a rectangular
matrix, its SVD gives
A = P!QT ,
(B.1)
with
P: The (normalized) eigenvectors of the matrix
AAT (i.e., PT P = I). The columns of P are called
the left singular vectors of A.
Q: The (normalized) eigenvectors of the matrix
AT A (i.e., QT Q = I). The columns of Q are called
the right singular vectors of A.
!: The diagonal matrix of the singular values,
1
! = " 2 with " being the diagonal matrix of the
eigenvalues of matrix AAT and of the matrix AT A
(as they are the same).
The SVD is a straightforward consequence of the
eigen-decomposition of positive semi-definite matrices
[see, e.g., Refs 5,11,47,65].
Note that Eq. B.1 can also be rewritten as:
A = P!QT =
L
#
# p# qT# ,
(B.2)
#=1
with L being the rank of X and # , p# , and q# being
(respectively) the #-th singular value, left and right
singular vectors of X. This shows that X can be
2010 John Wiley & Sons, Inc.
455
Overview
www.wiley.com/wires/compstats
reconstituted as a sum of L rank one matrices (i.e.,
the # p# qT# terms). The first of these matrices gives the
best reconstitution of X by a rank one matrix, the sum
of the first two matrices gives the best reconstitution
of X with a rank two matrix, and so on, and, in
general, the sum of the first M matrices gives the best
reconstitution of X with a matrix of rank M.
Generalized Singular Value Decomposition
The generalized singular value decomposition (GSVD)
decomposes a rectangular matrix and takes into
account constraints imposed on the rows and the
columns of the matrix. The GSVD gives a weighted
generalized least square estimate of a given matrix
by a lower rank matrix For a given I J matrix
A, generalizing the SVD, involves using two positive
definite square matrices with size I I and J J. These
two matrices express constraints imposed on the rows
and the columns of A, respectively. Formally, if M
is the I I matrix expressing the constraints for the
rows of A and W the J J matrix of the constraints for
the columns of A. The matrix A is now decomposed
into:
.T
.Q
A =.
P!
. T WQ
. = I.
with: .
PT M.
P=Q
(B.3)
In other words, the generalized singular vectors are
orthogonal under the constraints imposed by M
and W.
This decomposition is obtained as a result of the
standard SVD. We begin by defining the matrix .
A as:
1
1
1
1
.
AW 2 .
A = M 2 AW 2 A = M 2 .
(B.4)
We then compute the standard singular value
decomposition as .
A as:
.
A = P!QT
with: PT P = QT Q = I.
(B.5)
The matrices of the generalized eigenvectors are
obtained as:
1
.
P = M 2 P and
. = W 12 Q.
Q
(B.6)
The diagonal matrix of singular values is simply equal
to the matrix of singular values of .
A:
We verify that:
. = !.
!
.T
.Q
A =.
P!
456
(B.7)
by substitution:
1
AW 2
A = M 2 .
1
= M 2 P!QT W 2
.T
=.
P!Q
(from Eq. B.6).
(B.8)
To show that Condition B.3 holds, suffice it to
show that:
and
1
1
.
PT M.
P = PT M 2 MM 2 P = PT P = I
(B.9)
. T WQ
. = QT W 12 WW 12 Q = QT Q = I.
Q
(B.10)
Mathematical Properties of the Singular
Value Decomposition
It can be shown that the SVD has the important
property of giving an optimal approximation of
a matrix by another matrix of smaller rank [see,
e.g., Ref 62,63,65]. In particular, the SVD gives the
best approximation, in a least square sense, of any
rectangular matrix by another rectangular matrix of
same dimensions, but smaller rank.
Precisely, if A is an I J matrix of rank L (i.e.,
A contains L singular values that are not zero), we
denote it by P[M] (respectively Q[M] , ![M] ) the matrix
made of the first M columns of P (respectively Q, !):
'
(
P[M] = p1 , . . . , pm , . . . , pM
(B.11)
'
(
Q[M] = q1 , . . . , qm , . . . , qM
(B.12)
![M] = diag { 1 , . . . , m , . . . , M } .
(B.13)
The matrix A reconstructed from the first M
eigenvectors is denoted A[M] . It is obtained as:
T
A[M] = P[M] ![M] Q[M] =
M
#
m pm qTm ,
(B.14)
(with m being the m-th singular value).
The reconstructed matrix A[M] is said to be
optimal (in a least squares sense) for matrices of rank
M because it satisfies the following condition:
62
6
6
6
6A A[M] 6
2010 Jo h n Wiley & So n s, In c.
= trace
7/
[M]
AA
6
62
= min 6A X6
X
0/
[M]
AA
0T 8
(B.15)
Vo lu me 2, Ju ly /Au gu s t 2010
WIREs Computational Statistics
Principal component analysis
for the set of matrices X of rank smaller or equal to M
[see, e.g., Ref 65,66]. The quality of the reconstruction
is given by the ratio of the first M eigenvalues (i.e., the
squared singular values) to the sum of all the eigenvalues. This quantity is interpreted as the reconstructed
proportion or the explained variance, it corresponds
to the inverse of the quantity minimized by Eq. B.14.
The quality of reconstruction can also be interpreted
as the squared coefficient of correlation [precisely as
the Rv coefficient,18 between the original matrix and
its approximation.
The GSVD minimizes an expression similar to
Eq. B.14, namely
9
,
-:
A[M] = min trace M (A X) W (A X)T ,
X
(B.16)
for the set of matrices X of rank smaller or equal to
M.
ACKNOWLEDGEMENT
We would like to thank Yoshio Takane for his very helpful comments on a draft of this article.
REFERENCES
1. Pearson K. On lines and planes of closest fit to systems
of points in space. Philos Mag A 1901, 6:559572.
2. Cauchy AL. Sur lequation a` laide de laquelle on
determine les inegalites seculaires des mouvements des
plan`etes, vol. 9. Oeuvres Compl`etes (II`eme Serie); Paris:
Blanchard; 1829.
12. Abdi H. Eigen-decomposition: eigenvalues and eigenvectors. In: Salkind NJ, ed. Encyclopedia of Measurement and Statistics. Thousand Oaks: Sage Publications;
2007, 304308.
3. Grattan-Guinness I. The Rainbow of Mathematics.
New York: Norton; 1997.
13. Takane Y. Relationships among various kinds of eigenvalue and singular value decompositions. In: Yanai H,
Okada A, Shigemasu K, Kano Y, Meulman J, eds.
New Developments in Psychometrics. Tokyo: Springer
Verlag; 2002, 4556.
4. Jordan C. Memoire sur les formes bilineaires. J Math
Pure Appl 1874, 19:3554.
14. Abdi H. Centroid. Wiley Interdisciplinary Reviews:
Computational Statistics, 2009, 1:259260.
5. Stewart GW. On the early history of the singular value
decomposition. SIAM Rev 1993, 35:551566.
15. Kruskal JB. Factor analysis and principal component
analysis: Bilinear methods. In: Kruskal WH, Tannur
JM, eds. International Encyclopedia of Statistics. New
York: The Free Press; 1978, 307330.
6. Boyer C, Merzbach U. A History of Mathematics. 2nd
ed. New York: John Wiley & Sons; 1989.
7. Hotelling H. Analysis of a complex of statistical variables into principal components. J Educ Psychol 1933,
25:417441.
8. Jolliffe IT. Principal Component Analysis. New York:
Springer; 2002.
9. Jackson JE. A Users Guide to Principal Components.
New York: John Wiley & Sons; 1991.
10. Saporta G, Niang N. Principal component analysis:
application to statistical process control. In: Govaert
G, ed. Data Analysis. London: John Wiley & Sons;
2009, 123.
11. Abdi H. Singular Value Decomposition (SVD) and
Generalized Singular Value Decomposition(GSVD). In:
Salkind NJ, ed. Encyclopedia of Measurement and
Statistics. Thousand Oaks: Sage Publications; 2007,
907912.
Volume 2, July /August 2010
16. Gower J. Statistical methods of comparing different
multivariate analyses of the same data. In: Hodson F, Kendall D, Tautu P, eds. Mathemematics in
the Archological and Historical Sciences. Edinburgh:
Edingburh University Press; 1971, 138149.
17. Lingoes J, Schonemann
P. Alternative measures of fit
for the Schonemann-Carrol
matrix fitting algorithm.
Psychometrika 1974, 39:423427.
18. Abdi H. RV cofficient and congruence coefficient. In:
Salkind NJ, ed. Encyclopedia of Measurement and
Statistics. Thousand Oaks: Sage Publications; 2007,
849853.
19. Dray S. On the number of principal components: a test
of dimensionality based on measurements of similarity between matrices. Comput Stat Data Anal 2008,
52:22282237.
20. Quenouille M. Notes on bias and estimation.
Biometrika 1956, 43:353360.
2010 John Wiley & Sons, Inc.
457
Overview
www.wiley.com/wires/compstats
21. Efron B. The Jackknife, the Bootstrap and other Resampling Plans, vol. 83, CMBF-NSF Regional Conference
Series in Applied Mathematics: New York SIAM; 1982.
39. Mehlman DW, Sheperd UL, Kelt DA. Bootstrapping
principal components analysis: a comment. Ecology
1995, 76:640643.
22. Abdi H, Williams LJ. Jackknife. In: Salkind NJ, ed.
Encyclopedia of Research Design. Thousand Oaks:
Sage Publications; 2010, (In press).
40. Abdi H. Multivariate analysis. In: Lewis-Beck M, Bryman A, Futing T, eds. Encyclopedia for Research
Methods for the Social Sciences. Thousand Oaks, CA:
Sage Publications 2003, 669702.
23. Peres-Neto PR, Jackson DA, Somers KM. How many
principal components? stopping rules for determining
the number of non-trivial axes revisited. Comput Stat
Data Anal 2005, 49:974997.
24. Cattell RB. The scree test for the number of factors.
Multivariate Behav Res 1966, 1:245276.
25. Kaiser HF. A note on Guttmans lower bound for the
number of common factors. Br J Math Stat Psychol
1961, 14:12.
26. OToole AJ, Abdi H, Deffenbacher KA, Valentin D. A
low dimensional representation of faces in the higher
dimensions of the space. J Opt Soc Am [Ser A] 1993,
10:405411.
41. Kaiser HF. The varimax criterion for analytic rotation
in factor analysis. Psychometrika 1958, 23:187200.
42. Thurstone LL. Multiple Factor Analysis. Chicago, IL:
University of Chicago Press; 1947.
43. Stone JV. Independent Component Analysis: A Tutorial
Introduction. Cambridge: MIT Press; 2004.
44. Abdi H. Partial least square regression, Projection
on latent structures Regression, PLS-Regression. Wiley
Interdisciplinary Reviews: Computational Statistics, 2
2010, 97106.
45. Lebart L, Fenelon JP. Statistique et informatique
appliquees. Paris: Dunod; 1975.
27. Geisser S. A predictive approach to the random effect
model. Biometrika 1974, 61:101107.
46. Benzecri J-P. Lanalyse des donnees, Vols. 1 and 2.
Paris: Dunod; 1973.
28. Tennenhaus M. La regression pls. Paris: Technip;
1998.
47. Greenacre MJ. Theory and Applications of Correspondence Analysis. London: Academic Press; 1984.
29. Stone M. Cross-validatory choice and assessment of
statistical prediction. J R Stat Soc [Ser A] 1974,
36:111133.
48. Greenacre MJ. Correspondence Analysis in Practice.
2nd ed. Boca Raton, FL: Chapman & Hall/CRC; 2007.
30. Wold S. PLS for multivariate linear modeling. In:
van de Waterbeemd H, ed. Chemometric Methods
in Molecular Design. Weinheim: Wiley-VCH Verlag;
1995, 195217.
31. Malinowski ER. Factor Analysis in Chemistry. 3rd ed.
New York: John Wiley & Sons; 2002.
32. Eastment HT, Krzanowski WJ. Cross-validatory choice
of the number of components from a principal component analysis. Technometrics 1982, 24:7377.
33. Wold S. Cross-validatory estimation of the number of
components in factor and principal component analysis. Technometrics 1978, 20:397405.
34. Diaconis P, Efron B. Computer intensive methods in
statistics. Sci Am 1983, 248:116130.
35. Holmes S. Using the bootstrap and the Rv coefficient in
the multivariate context. In: Diday E, ed. Data Analysis, Learning, Symbolic and Numeric Knowledge. New
York: Nova Science; 1989, 119132.
36. Efron B, Tibshirani RJ. An Introduction to the Bootstrap. New York: Chapman and Hall; 1993.
37. Jackson DA. Stopping rules in principal aomponents
analysis: a comparison of heuristical and statistical
approaches. Ecology 1993, 74:22042214.
38. Jackson DA. Bootstrapped principal components analysis: a reply to Mehlman et al. Ecology 1995,
76:644645.
458
49. Abdi H, Valentin D. Multiple correspondence analysis. In: Salkind NJ, ed. Encyclopedia of Measurement
and Statistics. Thousand Oaks, CA: Sage Publications;
2007, 651657.
50. Hwang H, Tomiuk MA, Takane Y. Correspondence
analysis, multiple correspondence analysis and recent
developments. In: Millsap R, Maydeu-Olivares A, eds.
Handbook of Quantitative Methods in Psychology.
London: Sage Publications 2009, 243263.
51. Abdi H, Williams LJ. Correspondence analysis. In:
Salkind NJ, ed. Encyclopedia of Research Design.
Thousand Oaks: Sage Publications; 2010, (In press).
52. Brunet E. Faut-il ponderer les donnees linguistiques.
CUMFID 1989, 16:3950.
53. Escofier B, Pag`es J. Analyses factorielles simples et
multiples: objectifs, methodes, interpretation. Paris:
Dunod; 1990.
54. Escofier B, Pag`es J. Multiple factor analysis. Comput
Stat Data Anal 1994, 18:121140.
55. Abdi H, Valentin D. Multiple factor analysis (mfa).
In: Salkind NJ, ed. Encyclopedia of Measurement
and Statistics. Thousand Oaks, CA: Sage Publications;
2007, 657663.
56. Diamantaras KI, Kung SY. Principal Component Neural Networks: Theory and Applications. New York:
John Wiley & Sons; 1996.
57. Abdi H, Valentin D, Edelman B. Neural Networks.
Thousand Oaks, CA: Sage; 1999.
2010 Jo h n Wiley & So n s, In c.
Vo lu me 2, Ju ly /Au gu s t 2010
WIREs Computational Statistics
Principal component analysis
58. Nakache JP, Lorente P, Benzecri JP, Chastang JF.
Aspect pronostics et therapeutiques de linfarctus
myocardique aigu. Les Cahiers de, Analyse des
Donnees, 1977, 2:415534.
62. Abdi H, Valentin D. Mathematiques pour les sciences
cognitives. Grenoble: PUG; 2006.
59. Saporta G, Niang N. Correspondence analysis and
classification. In: Greenacre M, Blasius J, eds. Multiple
Correspondence Analysis and Related Methods. Boca
Raton, FL: Chapman & Hall; 2006, 371392.
64. Harris RJ. A Primer of Multivariate Statistics. Mahwah,
NJ: Lawrence Erlbaum Associates; 2001.
60. Abdi H. Discriminant correspondence analysis. In:
Salkind NJ, ed. Encyclopedia of Measurement and
Statistics. Thousand Oaks, CA: Sage Publications;
2007, 270275.
61. Abdi H, Williams LJ. Barycentric discriminant analyis
(BADIA). In: Salkind NJ, ed. Encyclopedia of Research
Design. Thousand Oaks, CA: Sage; 2010, (In press).
63. Strang G. Introduction to Linear Algebra. Cambridge,
MA: Wellesley-Cambridge Press; 2003.
65. Good I. Some applications of the singular value
decomposition of a matrix. Technometrics 1969,
11:823831.
66. Eckart C, Young G. The approximation of a matrix
by another of a lower rank. Psychometrika 1936,
1:211218.
67. Abdi H. Factor Rotations. In Lewis-Beck M, Bryman
A., Futimg T., eds. Encyclopedia for Research Methods for the Social Sciences. Thousand Oaks, CA: Sage
Publications; 2003, 978982.
FURTHER READING
Baskilevsky A. Applied Matrix Algebra in the Statistical Sciences. New York: Elsevier; 1983.
Volume 2, July /August 2010
2010 John Wiley & Sons, Inc.
459
You might also like
- WIREs Computational Stats - 2010 - Abdi - Principal component analysisNo ratings yetWIREs Computational Stats - 2010 - Abdi - Principal component analysis27 pages
- Principal Component Analysis: Herv e Abdi and Lynne J. WilliamsNo ratings yetPrincipal Component Analysis: Herv e Abdi and Lynne J. Williams27 pages
- Journal of Statistical Software: The Pls Package: Principal Component and Partial Least Squares Regression in RNo ratings yetJournal of Statistical Software: The Pls Package: Principal Component and Partial Least Squares Regression in R23 pages
- PCA For Non Precise Data-Carlo N. LauroNo ratings yetPCA For Non Precise Data-Carlo N. Lauro12 pages
- Tharrault - DPS - 2007 Fault Detection and Isolation With Robust Principal PCANo ratings yetTharrault - DPS - 2007 Fault Detection and Isolation With Robust Principal PCA9 pages
- Principal Components in Regression AnalysisNo ratings yetPrincipal Components in Regression Analysis27 pages
- A Predator-Prey System Involving Group Defense: A Connection Matrix ApproachNo ratings yetA Predator-Prey System Involving Group Defense: A Connection Matrix Approach15 pages
- Classical and Robust Regression Analysis With Compositional DataNo ratings yetClassical and Robust Regression Analysis With Compositional Data36 pages
- Chapter-4 Principal Component Analysis-Based FusionNo ratings yetChapter-4 Principal Component Analysis-Based Fusion27 pages
- American Statistical Association American Society For QualityNo ratings yetAmerican Statistical Association American Society For Quality8 pages
- Partial Least Squares Regression and Projection On Latent Structure Regression (PLS Regression)No ratings yetPartial Least Squares Regression and Projection On Latent Structure Regression (PLS Regression)10 pages
- 10-601 Machine Learning (Fall 2010) Principal Component AnalysisNo ratings yet10-601 Machine Learning (Fall 2010) Principal Component Analysis8 pages
- Functional Logistic Regression: A Comparison of Three MethodsNo ratings yetFunctional Logistic Regression: A Comparison of Three Methods20 pages
- 2012-408 Understanding Correlation MatricesNo ratings yet2012-408 Understanding Correlation Matrices6 pages
- Jonathan Tennyson - On The Calculation of Matrix Elements Between Polynomial Basis FunctionsNo ratings yetJonathan Tennyson - On The Calculation of Matrix Elements Between Polynomial Basis Functions3 pages
- Abstract Interpretation as Automated DeductionNo ratings yetAbstract Interpretation as Automated Deduction15 pages
- An Overview of Functional Data Analysis: Woncheol JangNo ratings yetAn Overview of Functional Data Analysis: Woncheol Jang5 pages
- Vector Autoregressive Models: T T 1 T 1 P T P T TNo ratings yetVector Autoregressive Models: T T 1 T 1 P T P T T4 pages
- Estimation For Multivariate Linear Mixed ModelsNo ratings yetEstimation For Multivariate Linear Mixed Models7 pages
- Deng - Tian - 2011 - A New Fault Isolation Method Based On Unified Contribution PlotsNo ratings yetDeng - Tian - 2011 - A New Fault Isolation Method Based On Unified Contribution Plots6 pages
- Multiple Taxicab Correspondence Analysis of A Survey Related To Health ServicesNo ratings yetMultiple Taxicab Correspondence Analysis of A Survey Related To Health Services25 pages
- Incomplete Data With Multi-Valued Information Systems: KeywordsNo ratings yetIncomplete Data With Multi-Valued Information Systems: Keywords10 pages
- Scale-Based Clustering The Radial Basis Function: NetworkNo ratings yetScale-Based Clustering The Radial Basis Function: Network6 pages
- Sciencedirect: Categorical Principal Component Logistic Regression: A Case Study For Housing Loan ApprovalNo ratings yetSciencedirect: Categorical Principal Component Logistic Regression: A Case Study For Housing Loan Approval7 pages
- Estimation of Multivariate Probit Models: A Mixed Generalized Estimating/pseudo-Score Equations Approach and Some Nite Sample ResultsNo ratings yetEstimation of Multivariate Probit Models: A Mixed Generalized Estimating/pseudo-Score Equations Approach and Some Nite Sample Results19 pages
- Estimation For Multivariate Linear Mixed ModelsNo ratings yetEstimation For Multivariate Linear Mixed Models6 pages
- Bootstrap Methods: Another Look at The JackknifeNo ratings yetBootstrap Methods: Another Look at The Jackknife27 pages
- # Loop Over Classes: 6.2 Principal Components Analysis (Pca)No ratings yet# Loop Over Classes: 6.2 Principal Components Analysis (Pca)10 pages
- A Generalized Framework For Conflict AnalysisNo ratings yetA Generalized Framework For Conflict Analysis16 pages
- Estimation of Simultaneous Systems of Spatially Interrelated Cross Sectional EquationsNo ratings yetEstimation of Simultaneous Systems of Spatially Interrelated Cross Sectional Equations24 pages
- Journal of Statistical Software: Factominer: An R Package For Multivariate AnalysisNo ratings yetJournal of Statistical Software: Factominer: An R Package For Multivariate Analysis18 pages
- WIREs Computational Stats - 2010 - Abdi - Principal component analysisWIREs Computational Stats - 2010 - Abdi - Principal component analysis
- Principal Component Analysis: Herv e Abdi and Lynne J. WilliamsPrincipal Component Analysis: Herv e Abdi and Lynne J. Williams
- Journal of Statistical Software: The Pls Package: Principal Component and Partial Least Squares Regression in RJournal of Statistical Software: The Pls Package: Principal Component and Partial Least Squares Regression in R
- Tharrault - DPS - 2007 Fault Detection and Isolation With Robust Principal PCATharrault - DPS - 2007 Fault Detection and Isolation With Robust Principal PCA
- A Predator-Prey System Involving Group Defense: A Connection Matrix ApproachA Predator-Prey System Involving Group Defense: A Connection Matrix Approach
- Classical and Robust Regression Analysis With Compositional DataClassical and Robust Regression Analysis With Compositional Data
- Chapter-4 Principal Component Analysis-Based FusionChapter-4 Principal Component Analysis-Based Fusion
- American Statistical Association American Society For QualityAmerican Statistical Association American Society For Quality
- Partial Least Squares Regression and Projection On Latent Structure Regression (PLS Regression)Partial Least Squares Regression and Projection On Latent Structure Regression (PLS Regression)
- 10-601 Machine Learning (Fall 2010) Principal Component Analysis10-601 Machine Learning (Fall 2010) Principal Component Analysis
- Functional Logistic Regression: A Comparison of Three MethodsFunctional Logistic Regression: A Comparison of Three Methods
- Jonathan Tennyson - On The Calculation of Matrix Elements Between Polynomial Basis FunctionsJonathan Tennyson - On The Calculation of Matrix Elements Between Polynomial Basis Functions
- An Overview of Functional Data Analysis: Woncheol JangAn Overview of Functional Data Analysis: Woncheol Jang
- Deng - Tian - 2011 - A New Fault Isolation Method Based On Unified Contribution PlotsDeng - Tian - 2011 - A New Fault Isolation Method Based On Unified Contribution Plots
- Multiple Taxicab Correspondence Analysis of A Survey Related To Health ServicesMultiple Taxicab Correspondence Analysis of A Survey Related To Health Services
- Incomplete Data With Multi-Valued Information Systems: KeywordsIncomplete Data With Multi-Valued Information Systems: Keywords
- Scale-Based Clustering The Radial Basis Function: NetworkScale-Based Clustering The Radial Basis Function: Network
- Sciencedirect: Categorical Principal Component Logistic Regression: A Case Study For Housing Loan ApprovalSciencedirect: Categorical Principal Component Logistic Regression: A Case Study For Housing Loan Approval
- Estimation of Multivariate Probit Models: A Mixed Generalized Estimating/pseudo-Score Equations Approach and Some Nite Sample ResultsEstimation of Multivariate Probit Models: A Mixed Generalized Estimating/pseudo-Score Equations Approach and Some Nite Sample Results
- Co-Clustering: Models, Algorithms and ApplicationsFrom EverandCo-Clustering: Models, Algorithms and Applications
- # Loop Over Classes: 6.2 Principal Components Analysis (Pca)# Loop Over Classes: 6.2 Principal Components Analysis (Pca)
- Estimation of Simultaneous Systems of Spatially Interrelated Cross Sectional EquationsEstimation of Simultaneous Systems of Spatially Interrelated Cross Sectional Equations
- Journal of Statistical Software: Factominer: An R Package For Multivariate AnalysisJournal of Statistical Software: Factominer: An R Package For Multivariate Analysis
- Vector and Tensor Analysis with ApplicationsFrom EverandVector and Tensor Analysis with Applications