Robust Procedures in Multivariate Analysis I: Robust Covariance Estimation

Author(s): N. A. Campbell
Source: Journal of the Royal Statistical Society. Series C (Applied Statistics), Vol. 29, No. 3 (1980)
, pp. 231-237
Published by: Wiley for the Royal Statistical Society
Stable URL: http://www.jstor.org/stable/2346896
Accessed: 07-01-2016 13:20 UTC

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at http://www.jstor.org/page/
info/about/policies/terms.jsp

JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range of content
in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new forms of scholarship.
For more information about JSTOR, please contact [email protected].

Royal Statistical Society and Wiley are collaborating with JSTOR to digitize, preserve and extend access to Journal of the
Royal Statistical Society. Series C (Applied Statistics).

http://www.jstor.org

This content downloaded from 140.127.23.2 on Thu, 07 Jan 2016 13:20:46 UTC
All use subject to JSTOR Terms and Conditions
Appl.Statist.(1980),
29, No. 3, pp. 231-237

RobustProceduresin Multivariate
Analysis
I: RobustCovarianceEstimation
By N. A. CAMPBELLt
ImperialCollege,London,UK
[ReceivedNovember1978. Final revisionMarch 19803

SUMMARY
Thedetectionofatypicalobservationsfrommultivariate data setscan be enhancedby
examiningprobabilityplotsofMahalanobis
squareddistancesusingrobust M-estimatesof
meansand ofcovariances, ratherthantheusualmaximum likelihood The
estimates.
weightsassociatedwiththerobustestimation can also be used to indicateatypical
observations.
For uncontaminated data,therobustestimates are similarto theusual
estimates.
A procedureforrobust
principal
component analysisisgiven;italsoindicates
atypical
observations
and providesan analysisrelatively
littleinfluenced bysuchobservations.
Keywords: ROBUST ESTIMATION; M-ESTIMATORS; OUTLIER DETECTION; PRINCIPAL COMPONENT
ANALYSIS; MULTIVARIATE NORMALITY

1. INTRODUCTION
OBSERVATIONS whichare grosslyatypicalin a singlecomponentcan oftenbe detectedby
applyingunivariatetechniquestoeachvariable.For multivariate
data,observations areoftenonly
foundto be atypicalwhenthe value foreach variableis consideredin relationto the other
variables;somevaluesfailto maintainthepatternofrelationshipsbetweenthevariablesevident
in themajorityoftheobservations.Since theperformance ofclassicalproceduresis seriously
influencedby atypicalvalues,robustmethodswhichare littleinfluencedby atypicalvalues
provide an attractivecomplementary approach. The surveypapers by Huber (1972) and
Hampel (1973) summarizethe importantmethods and resultsfromthe earlier years of
univariaterobuststudies,whileHampel (1977) and Huber (1977b) includea reviewof more
recentresults.An introductory paper by Hogg (1977) givesthe basic ideas.
The emphasisthroughoutthis paper is on the provisionof estimatesof means and of
covariancesfora singlegroupwhichare littleinfluenced by atypicalobservations, and on the
detectionofobservationshavingundueinfluence on theestimates.The underlying distribution
is assumed,aftertransformationifnecessary,
to be symmetric; a multivariate Gaussian formis
examinedin the probabilityplotting.Gnanadesikan(1977, Chapter 5) discussestransform-
ations to achieveapproximatesymmetry.
Robust M-estimationof means and covariancesis reviewedin Section 2, and its use in
conjunctionwithprobabilityplotsofassociatedMahalanobissquareddistancesis considered.
A procedureforrobustprincipalcomponentanalysisis proposedin Section3. Typicaldata sets
are examinedin Section4, whilesome generalrecommendations are givenin Section5.
2. ROBUST ESTIMATION OF MULTIVARIATE LOCATION AND SCATTER
Healy (1968) and Cox (1968) have suggestedan extensionofprobabilityplotsofunivariate
data to the multivariatesituation,by plottingthe Mahalanobis squared distance of each
observationagainsttheorderstatisticfora chi-squareddistributionwithv d.f.,wherev is the
t Now at Division of Mathematicsand Statistics,CSIRO, WesternAustralia.

This content downloaded from 140.127.23.2 on Thu, 07 Jan 2016 13:20:46 UTC
All use subject to JSTOR Terms and Conditions
232 APPLIED STATISTICS

number ofvariables(seealso Gnanadesikan, 1977,p. 172).Ifx represents thevx 1 vectorof

samplemeans,andV thesamplecovariance thentheMahalanobis
matrix, squareddistance of
themthobservation fromthemeanoftheobservations isdefinedbydA = (xm-x)TV o1(x,-
m= 1,...,n. The approachcombinesexamination of the distributional assumptionof a
multivariate Gaussianform withdetection ofatypicalobservations andis especiallysuitedto
informal graphical
description.
Atypical vectors
multivariate ofobservationswilltendtodeflate correlations andpossibly
inflatevariances.This willin generaldecreasethe Mahalanobisdistancefortheatypical
observation and distort
therestoftheplot.
The Mahalanobisdistances playa basicrolein multivariateM-estimation (seeMaronna,
1976;Hampel,1977;Huber,1977a,b).Theestimators canbederived byassuming anelliptically
symmetric and thenassociating
density, thiswitha contaminated Gaussiandensity.
Fromtheappliedviewpoint, M-estimators can be considered as a simplemodification of
theygivefullweight
classicalestimators; toobservations assumedtocomefrom themainbody
ofthedata,butreduced weightorinfluencetoobservations fromthetailsofthecontaminating
distribution.Inpractice,
theinfluenceofobservationswithunduly largeMahalanobis distances
is downweighted.
Theequations usedheretodefinerobust estimatorsofmeansandcovariances areas follows:
n n
X E Wm xm E Wm (1)
m=1 /m=1
and
V= E wm(-PX)(xm-)TI(m E W21), (2)
m=1 m1
where
Wm = W(dm) = (o(dm)ldm
and
dm = {(Xm-i)TV1(Xm-i)}4.
The solutionforx and V is iterative.
Hereow isa boundedinfluencefunction(Hampel,1974),often
linearovertherangeofvalues
ofdmcorresponding to reasonable data,butboundedoutsidethisrange.Hampel(1973)has
suggested andhencetheweight
thattheinfluence ofan extremeatypicalobservation
shouldbe
zero,so thatcoshouldredescend forsufficiently
largevaluesofdm.
The two-parameterformofcl)usedhereis
co(d)=d ifd do
= doexp {-i(d-do)2/lb} ifd> do, (3)
where
do= v+bi/12.
Themotivation behindthisform ofdois that,understandardassumptions, d2 ,- Xv,andso
Fisher'ssquarerootapproximation givesd N(Vv,1/12);b1 is equatedwitha percentage
pointofthestandard Gaussiandistribution.Theparameter therateofdecreaseof
b2controls
theinfluence to zero.
function
Followingextensiveempiricalexperience,it seemssufficient
forpracticalapplications to
considerthreemembers of(3). Theyare
(a) bI = oo all observations
are givenunitweight,so theusualestimates result;
(b) b, = 2,b2 = oo thenon-descending formsuggested byHuber(1964);
(c) b1= 2,b2 = 1*25 a redescending form,givingthe qualitativebehavioursuggested by
Hampel(1973),so thattheweight function ata faster
decreases ratethan
in (b).

This content downloaded from 140.127.23.2 on Thu, 07 Jan 2016 13:20:46 UTC
All use subject to JSTOR Terms and Conditions
 ROBUST PROCEDURES IN MULTIVARIATE ANALYSIS I 233

Theserecommended valuesofb1andb2arefurther supported byasymptotic andresults

results
fromtheanalysisofsomegenerated data.
Iftherobustestimates areto be usedinsubsequent itis important
analysis,
statistical that
theydiffer from
little theusualestimates whenappliedtouncontaminated data.Table1 givesa
plotoftheratioofrobusttousualestimates
stem-and-leaf ofvariance fortensetsofgenerated
Gaussiandatawith7 variables;
multivariate eachsetconsistsofsixgroupswithsamplesizeand
underlying meansand covariances corresponding to thefirstsix groupsofthe Thaisdata
consideredinSection4. Itseemsreasonable toconcludefrom Table1 thattherobustestimates
TABLE 1

generated
ratiosofrobusttousual variancesfor
Stem-and-leafplotfor multivariateGaussiandata
withsame underlyingstructureas Thais data (7 variables,60 groups).
1 01 1, 2, 2, 3, 3
1 00 0(243)b,1(19), 2(9), 3, 3, 4, 4, 5, 6, 6, 6, 7
099 0(9), 1(6), 2, 3(10), 4(7), 5(5), 6(7), 7(7), 8(13), 9(19)
0-98 1, 1,2,2,4,5,5,6,6,6,8,8,8,9,9,9
097 3, 4, 4, 7, 8, 8
096 1,2,2,2,7,9
0952, 0954, 0957, 0947, 0934, 0934, 0938, 0-927,0928, 0929, 0910, 0912,
0 912, 0-89,0-88,0-88,0-87,0 87, 0-86,0 82

a Value is 1 011
b
Value 1000 is repeated243 times.

ofthevariances
willgenerally be within2-3 percentoftheusualestimates forwell-behaved
correction
Gaussiandata,multiplicative
data.Formultivariate factors togive
canbecalculated
largesampleagreement of the expectedvalue of d2 underrobustand usual estimation.
E(d2) = v ifb, = oo. For thenon-descending
Asymptotically, influence function,
E(d2) = vCh(do; v+ 2)+ do {1-Ch (do; v)},
whereCh(.; v) denotesthecumulative distribution
chi-squared on v d.f.For the
function
influence
redescending function,

E(d2) = vCh(do; v+ 2)+ do exp{-id - do)2/b} f,(d2)d(d2)

Jdo2

wheref,(d2)is thechi-squareddensityon v d.f.Withb1 = 2-0and b2 = 125, the asymptotic

correction
factor influence
forthenon-descending
is lessthan1P020 andlessthan1P025
function
fortheredescendingfunction.
3. ROBUST PRINCIPAL COMPONENT ANALYSIS
A principal
component analysisoftheusualsamplecovariancematrixV (or associated
matrix
correlation R)involves
finding variables
Ym= UT Xm oftheoriginal
thelinearcombination
xmsuchthattheusualsamplevarianceoftheYmis a maximum. The solutionis givenbyan
eigenanalysisofV, viz.V = UEUT. The eigenvectors
ui,givenby thecolumnsofU, definethe
linearcombinations,whilethecorresponding diagonal elementsei of the diagonal matrixof
E are thesamplevariances
eigenvalues ofthederivedvariables.
Anobviouswayofmodifying istoreplaceV bytherobust
theanalysis in(2);thisis
estimator
the M-estimator solutionto robustprincipalcomponents. is weighted
An observation
accordingto its total distancedmfromthe robustestimateof location.This distancecan be
decomposedinto componentsalong each eigenvector; and an observationmayhave a large
componentalong one directionand small componentsalong the remainingdirectionsand

This content downloaded from 140.127.23.2 on Thu, 07 Jan 2016 13:20:46 UTC
All use subject to JSTOR Terms and Conditions
234 APPLIED STATISTICS

hencenotbe adequately downweighted. RobustM-estimation ofmeanand variancecan be

appliedto eachprincipalcomponent, to determine whichare littleinfluenced
directions by
atypicalobservations.
Theproposedprocedure is as follows:
1. Take as an initialestimate ofu1 thefirst froman eigenanalysis
eigenvector ofV.
2. Formtheprincipal component scoresYm= uT xm.
3. Determine theM-estimators ofmeanandvarianceofYm, andtheassociatedweights wm.
Themedianand{074 (interquartile range)}2
oftheYmcanbeusedtoprovide initial
robust
estimates.Here 0 74 = (2 x 0 675)- 1 and 0-675is the 75 per centquantilefortheN(O,1)
distribution.
Thisinitial
choiceensuresthattheproportion ofobservationsdownweighted
is keptreasonablysmall.
3(a). Afterthefirst taketheweights
iteration, wm as theminimum oftheweightsforthecurrent
and previousiterations;
thisprevents oscillation ofthesolution.
4. Calculatex and V as in (1) and (2) usingtheweights wmfromstage3.
5. Determine thefirst
eigenvalueand eigenvector u1 ofV.
6. Repeatsteps(2) to (5) untilsuccessive estimates oftheeigenvalueare sufficiently
close.
To determine successive ui,2 K i < v,project
directions thedataontothespaceorthogonalto
u1,...,ui- 1,and repeatsteps(2) to (5); take as the
thatspannedby the previouseigenvectors
initialestimate
thesecondeigenvectorfromthelastiteration
forthepreviouseigenvector.
The proposedprocedure forsuccessive
directions
can be setoutas follows.
7. Form xim= (I-Ui 1UT[ 1)xm,whereU, -1 = (u ui -J.
8. Repeatsteps(2) to (5) withximreplacing
xm,and determinethefirst u.
eigenvector
9. The principal
component scoresaregivenbyUT Xim= uT(I -Ui 1 UT 1) xmand hence
ui = (I-Ui- 1Ui)_JU.
Steps(7),(8) and(9) arerepeateduntilall veigenvalues
ei andeigenvectors with
ui,together
theassociatedweights, aredetermined.Alternativelytheprocedure maybe terminated after
somespecified proportion ofvariation
is explained.
Finally,a robustestimate ofthecovariance orcorrelationmatrix
canbe foundfrom UEUT,
toprovide analternative robustestimate.
Boththisapproachandthatdescribed intheprevious
Sectiongivea positivedefinite matrix.
correlation/covariance Robustestimationofeachentry
does notalwaysachievethis.
separately

4. SOME PRACTICAL EXAMPLES

datasettobe discussed
Thefirst is takenfroma studyofgeographic inthewhelk
variation
Thaislamellosa
(Campbell,1978).Data are availablefortwelvegroupson thewestcoastof
NorthAmerica.The numbersin each groupare: 50, 72,99, 76, 37, 36,46, 46, 51, 34,28 and 43.
Measurements weremadeon twenty variables;canonicalvariateanalysesshowthatsevenof
theseprovidemuchofthebetween-groups discrimination.
Robustcovariance estimation was
appliedtothetwelve
groupsbasedonthesevenvariables, andGamma(J;v/2)quantile-quantile
plotsoftheassociated
(Q-Q)probability d2 andGaussianQ-Q probability plotsofthed2/3were
made.The latteruses the Wilson-Hilferty (1931) transformationof a gammavariateto
Gaussianform(seealso Healy,1968,p. 159).
plotsofthedistances
Probability derivedwithb1= 2,b2 = 125showthatnineofthegroups
haveoneortwoatypical observations,whileonegrouphasseveralatypical observations.The
associated areall lessthan0-35;18ofthe21 atypical
weights observationshavezeroweight (to
twodecimalplaces).Probability plotsoftheusualdistances indicateonlytenoftheatypical
witha further
observations, fouror fivedoubtful.
Fourofthe21atypical observations
havetwoatypical valuesinthevector,giving
25outofa
totalof4326(= 618x 7 variables) variablevalueswhichareatypical. The variablesforeach
groupshowhighcorrelations; iftheobservations arelistedin increasingorderofthelargest
variable(hereoveralllength),atypicalvaluesare readilyapparentonce theobservation is

This content downloaded from 140.127.23.2 on Thu, 07 Jan 2016 13:20:46 UTC
All use subject to JSTOR Terms and Conditions
 ROBUST PROCEDURES IN MULTIVARIATE ANALYSIS I 235

indicated.Correctionsofeither100 or 50 unitswouldprovidegood agreementwiththevalues

foradjacentobservationsforall butfourvalues;interchanging theorderoftwonumberswould
explaina further two.The correctionof100or 50 unitsis an acceptableone sincethedial on the
calipersused to measurethewhelksrecordsto 50 units.In morethan4000 measurements the
linearscale will occasionallybe misread.
plotofusualdistancesforgroup3 (n = 99) (i.e.a Q-Q
Fig. l(a) showsa Gaussian probability
probabilityplot of cube root of squared Mahalanobis distances against Gaussian order
4- a) ' b)

SME24F
E
-o "97
*0*9
NT
Eq 9

1311*
24*
* 25541 0
o *22 0 244423
4OWW4l12 44555652

0.4L | 1Eb1=c
b,= | | o.a|r * .ob
=2.1.25
-3.2 -1i6 00 1-6 32 -3.2 -1i6 00 16 3.2
Gaussianquantiles Gaussianquantiles

3.12-c

*
X ~~~~~~~~~*1
1**H

31
454*
* ~~~~~46*
25* FIG. 1
* *42
143
2*
0*W* *"n

0.72 * b=20,b2=1-25
-3*2 -16 0o0 1-6 3.2
Gaussianquantiles

statisticswiththe distancescalculatedusingthe usual means and covariances);one atypical

observation(79) is indicated.RobustM-estimation withb1 = 2-0,b2 = oo givesa weightof0 03
forthis observation;a second observation(97) has a weightof 0411.With b2 = 1P25,both
observationshavea zeroweight.Fig. 1(b)showsa Gaussian probability plotofrobustdistances
whenb1 = 2-0,b2 = 1 25. Two observations, 79 and 97,are clearlyatypical.Bothare atypicalin
the second variable,the firstbeingout by 100 unitsand thesecond by 50 units.A Gaussian
probability plotofrobustdistancesforthecorrecteddata (Fig. 1(c)),withb1 = 2-0,b2 = 1P25, is
nowlinear.None oftheweightsis nowlessthan0 35.The usualestimateofstandarddeviationis
48-9;therobustestimateis 44 9 whiletherobustestimateaftercorrecting thedata is 45 8. The
correlationsofthesecond variablewiththeremainingvariablesare increasedby 0 02 to 0 06.
The seconddata setis takenfroman unpublishedstudyofmorphometric divergenceinmale
and femalescorpionsoccurringin Australia.Nine variablesweremeasuredon each specimen.
The male data forone species (n = 181) are discussedhere. Robust covarianceestimation
(b1 = 2 0, b2 = 1P25)indicatesfiveobservationswithweightslessthan0 35. A robustprincipal

This content downloaded from 140.127.23.2 on Thu, 07 Jan 2016 13:20:46 UTC
All use subject to JSTOR Terms and Conditions
236 APPLIED STATISTICS

componentanalysisindicatesa further threeobservationswithweightslessthan0 10forat least

onecomponentand a further sixteenwithweightslessthan0 35.The advantageofthecombined
approach is thatdifferent combinationsof variablesare examined.For example,one of the
observationswitha zero weightforrobustcovarianceestimationhas zero weightforprincipal
components5 and 6. Examinationoftheeigenvectors showsvariable8 to have a highloading
forthesecomponents.The measurement of 3 5 was checkedby myco-workerand foundto be
6 0 (therobustestimateofstandarddeviationis 077). Anotherobservation(notyetrechecked)
has a weightof0 60 forrobustcovarianceestimation(b1 = 2 0, b2 = 1P25)but weightsof0-06,
0 21,0 27 and 0 23 forprincipalcomponents3-6. Ofthoseobservationscheckedand corrected,
the common errorwas a measurementout by 1 or 15 mm; the metal dial calipers were
graduatedto 0 05 mm.
5. DISCUSSION
A recommendedpracticalapproachis to determinethemeans,covariances,distancesand
associated weightsforb, = oo; forb, = 20, b2 = oo; and forbI = 2 0, b2 = 1 25. Gaussian
probabilityplots of cube root of squared distances(withb1 = 20, b2 = 125) will indicate
atypicalobservations.For more than six or seven variables,a robustprincipalcomponent
atypicalobservations.The weightsw2 associatedwiththe
analysisis also usefulforidentifying
d2 also indicateatypical observations.Examinationof a numberof data sets and of the
generatedGaussian data in Section 2 shows that a weightof less than 0 30 withb1 = 2-0,
b2= 1 25 (correspondingapproximatelyto a weightof less than 0 60 withb2 = oo) always
indicatesan atypicalobservation.A weightofmorethan0 70 withb2 = oo is associatedwitha
reasonableobservation.
The Mahalanobis squareddistancesare usuallyplottedagainstthe quantilesofa gamma
distributionwith shape parameterv/2,though the quantiles of a beta {v/2,(n- v- 1)/2}
distribution may be moreappropriate(Small, 1978).Thereis good agreementbetweenQ-Q
plotsofd2 versusGamma(1, v/2)and ofd2/3versusN(0, 1),thoughthecuberoottransformation
tends to lessen the visual impact of the large distanceson the gamma plot. The general
conclusionherereinforces theremarksofHealy(1968,p. 159)thattheGaussianplotseemsmore
than adequate fordetectingatypicalobservationsand examiningthe multivariate Gaussian
assumption.
ACKNOWLEDGEMENTS
I am gratefulto F. R. Hampel forstimulating discussionsofsomeoftheproblemsin robust
estimation, fortheirsuggestions
and to M. J.R. Healy,D. R. Cox and a referee and commentson
an earlierdraftof the paper.
The authorwas supportedby a CSIRO Divisional PostgraduateResearchStudentship.

REFERENCES
variablesnail,Thais lamellosa.
dogwinkle:ecologicalgeneticsofa morphologically
CAMPBELL,C. A. (1978).The frilled
UnpublishedPh.D. thesis,Universityof California,Davis.
Cox, D. R. (1968). Notes on some aspectsof regressionanalysis.J. R. Statist.Soc. A, 131, 265-279.
GNANADESIKAN,R. (1977). MethodsforStatisticalData AnalysisofMultivariateObservations. New York: Wiley.
HAMPEL, F. R. (1973). Robust estimation:a condensedpartialsurvey.Z. Wahr,verw.Geb.,27, 87-104.
(1974). The influencecurveand its role in robustestimation.J. Amer.Statist.Ass.,69, 383-393.
(1977).Moderntrendsinthetheoryofrobustness. ResearchReportNo. 13,SwissFederalInstitute ofTechnology,
Zurich.
HEALY, M. J. R. (1968). Multivariatenormalplotting.Appl.Statist.,17, 157-161.
HOGG, R. V. (1977). An introduction to robustprocedures.Comm.Statist.Theor.Meth.,A6, 789-794.
HUBER, P. J.(1964). Robust estimationfora locationparameter.Ann.Math. Statist.,35, 73-101.
(1972). Robust statistics:a review.Ann.Math. Statist.,43, 1041-1067.
(1977a). Robustcovariances.In StatisticalDecisionTheoryand RelatedTopicsII (S. S. Gupta and D. S. Moore,
eds),pp. 165-191.New York: AcademicPress.
(1977b).RobustStatisticalProcedures.Philadelphia:SIAM.

This content downloaded from 140.127.23.2 on Thu, 07 Jan 2016 13:20:46 UTC
All use subject to JSTOR Terms and Conditions
 ROBUST PROCEDURES IN MULTIVARIATE ANALYSIS I 237

MARONNA,R. A. (1976). Robust M-estimatorsof multivariate locationand scatter.Ann.Statist.,1, 51-67.

65, 657-658.
SMALL,N. J. H. (1978). Plottingsquared radii.Biometrika,
WILK, M. B., GNANADESIKAN,R. and HUYETT, M. J. (1962). Estimationof the parameters of the gamma distribution
usingorderstatistics.Biometrika,
49, 525-545.

This content downloaded from 140.127.23.2 on Thu, 07 Jan 2016 13:20:46 UTC
All use subject to JSTOR Terms and Conditions