Multivariate Data Analysis

(MGT513, BAT531, TIM711)

Lecture 3
 Chapter 5

Exploratory Factor Analysis (EFA)

 What is Exploratory Factor Analysis?
• Exploratory Factor Analysis (EFA)
– Examines the interrelationships among a large number of
variables and then attempts to explain them in terms of
their common underlying dimensions.
– Identifies the common factors and explain their
relationship
– The observed variance in each variable = a few of
(unobservable) common factors + a specific factor
(unrelated to any factor)

For use with Hair, Black, Babin and Anderson, Multivariate Data Analysis 8e © 2018 Cengage Learning EMEA
 What is Exploratory Factor Analysis?
• Latent Traits or Unobservable Characteristics
– For length or weight, such a distinction may be
unnecessary because the property is almost perfectly
Observable
– However, for attitudes, belief, perceptions, and other
psychological notions, our measurement instruments are
imperfect

For use with Hair, Black, Babin and Anderson, Multivariate Data Analysis 8e © 2018 Cengage Learning EMEA
 Types of Factor Analysis
Exploratory Factor Analysis (EFA)
• used to discover the factor structure of a construct and
examine its reliability. It is data driven.

Confirmatory Factor Analysis (CFA)

• used to confirm the fit of the hypothesized factor structure to
the observed (sample) data. It is theory driven.
 Objectives
Data Summarization versus Data Reduction
• Data summarization
– Defining structure through underlying dimensions that, when
interpreted and understood, describe the data in a much smaller
number of concepts than the original individual variables. The
basis for scale development.
• Data reduction
– Extends the process of data summarization by deriving an
empirical value (factor score or summated scale) for each
dimension (factor) and then substituting this value for the
original variable values in subsequent analysis.
 Example 1: Brand Personality

114 personal traits

Five-factor solution:
1. Sincerity
2. Excitement
3. Competence
4. Sophistication
5. Ruggedness
For use with Hair, Black, Babin and Anderson, Multivariate Data Analysis 8e © 2018 Cengage Learning EMEA
 Example 2: New Luxury Car Concept

For use with Hair, Black, Babin and Anderson, Multivariate Data Analysis 8e © 2018 Cengage Learning EMEA
 Simple illustration
• Holzinger and Swineford (1939)
– Psychological Testing of Children
– Five tests with 7th & 8th-graded children(n=145)
– X1 = Paragraph Comprehension (PARA)
– X2 = Sentence Completion (SENT)
– X3 = Word Meaning (WORD)
– X4 = Addition (ADD)
– X5 = Counting Dots (DOTS)

For use with Hair, Black, Babin and Anderson, Multivariate Data Analysis 8e © 2018 Cengage Learning EMEA
 Simple illustration

• Correlation Matrix

PARA SENT WORD ADD DOTS

PARA 1.000
SENT 0.722 1.000
WORD 0.714 0.685 1.000
ADD 0.203 0.246 0.170 1.000
DOTS 0.095 0.181 0.113 0.585 1.000

For use with Hair, Black, Babin and Anderson, Multivariate Data Analysis 8e © 2018 Cengage Learning EMEA
 Simple illustration: One-Factor Model

• One-factor model with five variables

𝑋1 𝑋2 𝑋3 𝑋4 𝑋5

𝛿 1
For use with Hair, Black,
𝛿
Babin and Anderson, 2
𝛿 𝛿
3 © 2018 Cengage Learning4EMEA
Multivariate Data Analysis 8e
𝛿5
 Simple illustration : One-Factor Model
• Let 𝜉 = common factor and 𝛿𝑖 = a specific factor
• One-factor model with five variables
𝑋𝑖 = 𝜆𝑖 𝜉 + 𝛿𝑖 , 𝑖 = 1,2,3,4,5
where cor 𝛿𝑖 , 𝛿𝑗 = 0, 𝑖 ≠ 𝑗 and cor 𝛿𝑖 , 𝜉 = 0
• Assume 𝑋 and 𝜉 are standardized variables. Then
Var 𝑋𝑖 = Var 𝜆𝑖 𝜉 + 𝛿𝑖 = 𝜆𝑖 2 + 𝑉𝑎𝑟 𝛿𝑖 = 1
𝜆𝑖 2 = communality of 𝑋𝑖
= the proportion of the variation in 𝑋𝑖 explained by 𝜉
= 1 − Var 𝛿𝑖 = 1 − 𝜃𝑖𝑖2
where 𝜃𝑖𝑖2 = Var 𝛿𝑖 = variance of specific factor 𝑋𝑖
• As 𝜆𝑖 2 → 1 𝜃𝑖𝑖2 → 0 , 𝑋𝑖 is a nearly perfect measure of 𝜉
• As 𝜆𝑖 2 → 0 𝜃𝑖𝑖2 → 1 , 𝑋𝑖 is not explained by 𝜉
 What is Exploratory Factor Analysis?

• Two-factor model with five variables

𝜉1 𝜉2

𝑋1 𝑋2 𝑋3 𝑋4 𝑋5

𝛿 1
For use with Hair, Black,
𝛿
Babin and Anderson, 2
𝛿 𝛿
3 © 2018 Cengage Learning4EMEA
Multivariate Data Analysis 8e
𝛿5
 Simple illustration : Two-Factor Model
• Two-factor model with five variables
𝑋𝑖 = 𝜆𝑖1 𝜉1 + 𝜆𝑖2 𝜉2 + 𝛿𝑖 , 𝑖 = 1,2,3,4,5
where cor 𝛿𝑖 , 𝛿𝑗 = 0, 𝑖 ≠ 𝑗 and cor 𝛿𝑖 , 𝜉𝑘 = 0, 𝑘 = 1,2
where 𝜉1 = Verbal Aptitude Factor
𝜉2 = Quantitative Aptitude Factor
• Assume 𝑋 and 𝜉 are standardized variables. Then
Var 𝑋𝑖 = Var 𝜆𝑖1 𝜉1 + 𝜆𝑖2 𝜉2 + 𝛿𝑖 = 𝜆𝑖1 2 + 𝜆𝑖2 2 + 𝑉𝑎𝑟 𝛿𝑖
= 𝜆𝑖1 2 + 𝜆𝑖2 2 + 𝜃𝑖𝑖2 = 1
𝜆𝑖1 2 + 𝜆𝑖2 2 = communality of 𝑋𝑖 = 1 − 𝜃𝑖𝑖2
• Consider a student with high 𝜉1 and low 𝜉2 . We expect the student to
perform well on those tests requiring more verbal than quantitative
ability. That is, If the student’s performance in a task of sentence
completion (measured by 𝑋1 ), we should expect a value of 𝜆11 near 1 and
a value for 𝜆12 closer to 0.
 Exploratory Factor Analysis with c Common Factors

𝑋1 = 𝜆11 𝜉1 + 𝜆12 𝜉2 + ⋯ + 𝜆1𝑐 𝜉𝑐 + 𝛿1

𝑋2 = 𝜆21 𝜉1 + 𝜆22 𝜉2 + ⋯ + 𝜆2𝑐 𝜉𝑐 + 𝛿2
⋮
𝑋𝑝 = 𝜆𝑝1 𝜉1 + 𝜆𝑝2 𝜉2 + ⋯ + 𝜆𝑝𝑐 𝜉𝑐 + 𝛿𝑝

In matrix notation,

𝑿 = 𝜩𝜦𝑇𝑐 + 𝜟

where 𝜩 = [𝜉1 , 𝜉2 , ⋯, 𝜉𝑐 ]
𝜟 = [𝛿1 ,𝛿2 , ⋯,𝛿𝑝 ]
𝜦𝑐 = 𝑝 × 𝑐 matrix of coefficients
 Exploratory Factor Analysis with c Common Factors

• Assumptions For Common Factor Models

1. Common factors 𝜉𝑖 ’s are mutually uncorrelated and have a unit variance:
1
𝜩𝑇 𝜩 = 𝐼
𝑛−1

2. Specific factors 𝛿𝑖 ’s are mutually uncorrelated and have a diagonal

covariance matrix variance :
1 2 2
𝜣= 𝜟𝑇 𝜟 = 𝑑𝑖𝑎𝑔(𝜃11 , 𝜃22 2
, ⋯ ,𝜃𝑝𝑝 )
𝑛−1

3. Common factors 𝜉𝑘 ’s and specific factors 𝛿𝑖 ’s are mutually uncorrelated:

𝜩𝑇 𝜟 = 𝟎
 Exploratory Factor Analysis with c Common Factors

• Assumptions for common factor models produce the correlation of 𝑿

𝑹 = Var 𝑿 = E 𝑿𝑻 𝑿 = E (𝜩𝜦𝑇𝑐 + 𝜟)𝑻 (𝜩𝜦𝑇𝑐 + 𝜟)

= E 𝜦𝑐 𝜩𝑇 𝜩𝜦𝑇𝑐 + E 𝜦𝑐 𝜩𝑇 𝜟 + E 𝜟𝑇 𝜩𝜦𝑇𝑐 + E 𝜟𝑇 𝜟
= 𝜦𝑐 E 𝜩𝑇 𝜩 𝜦𝑇𝑐 + 𝜦𝑐 E 𝜩𝑇 𝜟 + E 𝜟𝑇 𝜩 𝜦𝑇𝑐 + E 𝜟𝑇 𝜟
= 𝜦𝑐 𝜦𝑇𝑐 + 𝜣

• The correlation between 𝑿 and 𝜩 (factor loading) is

Cor 𝑿, 𝜩 = E 𝑿𝑻 𝜩 = E (𝜩𝜦𝑇𝑐 + 𝜟)𝑻 𝜩 = 𝜦𝑇𝑐

 Exploratory Factor Analysis with c Common Factors
• Common Factor Model:
𝑹 = 𝜦𝑐 𝜦𝑇𝑐 + 𝜣

𝑹 − 𝜣 = 𝜦𝑐 𝜦𝑇𝑐
• PCA revisited:
– Singular Value Decomposition of 𝐗
𝑿 = 𝒁𝒔 𝑫𝟏/𝟐 𝑼𝑻
𝟏 𝟏
𝑹=E 𝑿𝑻 𝑿 =E 𝑼𝑫𝟏/𝟐 (𝒁𝒔 𝑻 𝒁𝒔 )𝑫𝟏/𝟐 𝑼𝑻 = (𝑼𝑫 )(𝑼𝑫 )𝑻 = 𝑭𝑭𝑻
𝟐 𝟐

𝑹 ≈ 𝑭𝒄 𝑭𝑻𝒄
For dimension reduction, we try to extract some subset of c components that closely
approximates 𝑹: 𝑭𝑐 is the first c columns of factor loading matrix F whose elements
are interpretable as the correlations between original variables X and c extracted
common factors
 Solution Procedure

• Principal factor method (Principal axis factoring)

– Main idea: Replace the diagonal elements of 𝑹 with
communality values and conduct PCA
– No prior information about measurement error 𝛿
– Initial estimate of communalities: use Squared Multiple
Correlation (SMC)
• As a initial value, use 𝑅𝑖2 from the regression of 𝑋𝑖 on the
remaining 𝑋𝑗 ’s
– Iterate until the result converges
 Solution Procedure

One iteration Final result

 Rotation Indeterminancy
• Infinitely number of solutions: Infinitely number of bases (column vectors)
for 𝑹 − 𝜣

𝑻 = 𝑐 × 𝑐 orthogonal transformation matrix (𝑻𝑇 𝑻 = 𝑻𝑻𝑇 = 𝑰)

e.g. Two-dimensional orthogonal rotation:

cos(𝛼) −sin(𝛼)
𝑻=
sin(𝛼) cos(𝛼)

−30 degree rotation:

cos(−30) −sin(−30) 0.866 0.500
𝑻= =
sin(−30) cos(−30) −0.500 0.866

• The correlations between the rotated factors (𝜩𝑇 𝑻) and the original
variables 𝑿 = 𝜩𝜦𝑇𝑐 + 𝜟 is
1
𝜦∗𝑐 = 𝚵𝜦𝑇𝑐 + 𝚫 𝑇 𝜩𝑇 𝑻 = 𝜦𝑐 𝑻
𝑛−1
 Rotation Indeterminancy

• Communalities are unchanged by rotation

𝑹 − 𝜣 = 𝜦∗𝑐 𝜦∗𝑇
𝑐 = 𝜦𝑐 𝑻 𝜦𝑐 𝑻
𝑇 = 𝜦 𝑻𝑻𝑇 𝜦𝑇 = 𝜦 𝜦𝑇
𝑐 𝑐 𝑐 𝑐

Total variance explained by Unrotated factors

= Total variance explained by Rotated factors

• Type of rotations
1. Orthogonal rotation: varimax or quartimax
2. Non-orthogonal (Oblique) rotation: promax or direct oblimin
 Factor Rotation

• Desirable factor loading pattern (Comrey, 1973)

1. Most of the loadings on any specific factor (column)
should be small (as close to zero as possible), and only a
few loadings should be large in absolute value.
2. A specific row of the loadings matrix, containing the
loadings of a given variable with each factor, should
display nonzero loadings on only one or no more than a
few factors.
3. Any pair of factors (columns) should exhibit different
patterns of loadings. Otherwise, one could not
distinguish the two factors represented by these
columns.
Factor Rotation: Orthogonal Rotation Example
Factor Rotation: Orthogonal Rotation Example
 Factor Rotation
• The factor solution is reoriented through a process called rotation. Two
types of rotation:
– Orthogonal rotation preserves the perpendicularity of the axes (i.e., the
rotated factors remain uncorrelated). Two widely used methods for
orthogonal rotation are varimax rotation (which tries to achieve simple
structure by focusing on the columns of the factor loadings matrix) and
quartimax rotation (which focuses on the rows).
– Oblique rotation allows for correlation between the rotated factors. One
approach involves rotating the solution to match a target matrix exhibiting
simple structure. Some cautions about the properties of oblique factor
solutions:
• Because the common factors are now correlated, the communalities are no longer
given by the sum of the squared factor loadings.
• In an oblique solution, there is a distinction between a structure loading (i.e., the
correlation between a variable and a factor) and a pattern loading (i.e., the partial
correlation between a variable and a common factor controlling for other common
factors).
 Factor Rotation

Orthogonal Rotation Oblique rotation

For use with Hair, Black, Babin and Anderson, Multivariate Data Analysis 8e © 2018 Cengage Learning EMEA
 Factor Rotation

Orthogonal rotation methods

• are the most widely used rotational methods.
• are the preferred method when the research goal is data reduction to
either a smaller number of variables or a set of uncorrelated measures
for subsequent use in other multivariate techniques.

Oblique rotation methods

• best suited to the goal of obtaining several theoretically meaningful
factors or constructs because, realistically, very few constructs in the
“real world” are uncorrelated.
 Factor Rotation: Oblique Rotation
• Pattern Loading vs. Structure Loading
Factor Rotation: Oblique Rotation
 Factor Scores
• Principal Component Score
𝒛𝒊 = 𝑿𝒔 𝒖𝒊
• Consider approximating 𝜩 as
𝜩 = 𝑿𝒔 𝑩
where 𝑩 is a matrix of factor score coefficients
• 𝜩 is unobservable and so we cannot choose 𝑩 using OLS
1
• Let’s premultiply each side of the above by 𝑿𝑻𝒔
𝑛−1
1 𝑻
1
𝑿𝒔 𝜩 = 𝑿𝑻𝒔 𝑿𝒔 𝑩 ⇔ 𝜦𝑐 = 𝑹𝑩
𝑛−1 𝑛−1
which reduces to 𝑩 = 𝑹−𝟏 𝜦𝑐
• Hence, estimated factor scores are given by
𝜩 = 𝑿𝒔 𝑹−𝟏 𝜦𝑐
which is not unique because of rotational indeterminacy.
 Sample Problem: Perception of Ready-To-Eat Cereals

• To characterize the consideration behavior of cereal consumers (i.e., to

explain which brands the consumer is willing to purchase) as a function of
the underlying characteristics of 12 different types of brands
 Sample Problem: Perception of Ready-To-Eat Cereals

• Number of factors: 4 factors?

 Sample Problem: Perception of Ready-To-Eat Cereals

• SMCs for each of 25 RTE cereal attributes; explaining at least

52%=13/25
Sample Problem: Perception of Ready-To-Eat Cereals
Sample Problem: Perception of Ready-To-Eat Cereals

Factor 1: Healthful
Factor 2: Artificial
Factor 3: Non-Adult
Factor 4: Interesting
RTE Cereal: Factor Scores
RTE Cereal: Factor Scores
RTE Cereal: Factor Scores