12/28/2016

WhatWeKnowAboutSpreadsheetErrors

PublishedintheJournalofEndUserComputing's
SpecialissueonScalingUpEndUserDevelopment
Volume10,No2.Spring1998,pp.1521
RevisedMay2008.

WhatWeKnowAboutSpreadsheet
Errors

RaymondR.Panko
UniversityofHawaii
CollegeofBusinessAdministration
2404MaileWay
Honolulu,HI96822
USA

Email:[email protected]
HomePage:http://panko.shidler.hawaii.edu
SpreadsheetResearch(SSR)Website:http://panko.shidler.hawaii.edu/ssr/
Thispaper:http://panko.shidler.hawaii.edu/ssr/Mypapers/whatknow.htm

Abstract
Althoughspreadsheetprogramsareusedforsmall"scratchpad"applications,theyare
alsousedtodevelopmanylargeapplications.Inrecentyears,wehavelearnedagood
dealabouttheerrorsthatpeoplemakewhentheydevelopspreadsheets.Ingeneral,
errorsseemtooccurinafewpercentofallcells,meaningthatforlargespreadsheets,
theissueishowmanyerrorsthereare,notwhetheranerrorexists.Theseerrorrates,
althoughtroubling,areinlinewiththoseinprogrammingandotherhumancognitive
domains.Inprogramming,wehavelearnedtofollowstrictdevelopmentdisciplinesto
eliminatemosterrors.Surveysofspreadsheetdevelopersindicatethatspreadsheet
creation,incontrast,isinformal,andfeworganizationshavecomprehensivepolicies
forspreadsheetdevelopment.Althoughprescriptivearticleshavefocusedonsuch
disciplinesasmodularizationandhavingassumptionssections,thesemaybefarless
importantthanotherinnovations,especiallycellbycellcodeinspectionafterthe
developmentphase.
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

1/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Introduction
Inaspecialissueonlargesystemdevelopmentbyendusers,itmayseemoddtosee
apaperonspreadsheetdevelopment.Spreadsheetsareoftenseenassmalland
simple"scratchpad"applications,andspreadsheetdevelopmentisoftenviewedasa
solitaryactivity.Wewillseethatneitherviewiscorrect.Manyspreadsheetsarelarge
andcomplex,anddevelopmentofteninvolvesinteractionsamongmultiplepeople.In
fact,wewouldguessthatthelargestportionoflargescaleenduserapplications
todayinvolvespreadsheetdevelopment.Inaddition,theextensiveinformationthat
hasemergedaboutspreadsheetdevelopmentmayhighlightdevelopmentissuesfor
othertechnologies.Finally,whilespreadsheetdevelopmentmayseem"old,"our
understandingofspreadsheetdevelopmenthasonlymaturedinthelastfewyears.
Thetitleofthispaperisratherpresumptuous.Itassumesthatwehavealarge
enoughbaseofresearchfindingstobeabletosummarizekeyresultsfromthe
relativelyyoungfieldofspreadsheeterrors.Yetwearguethatspreadsheetresearchis
indeedsufficientlymaturetogiveusefuladvicetocorporatemanagementand
individualdevelopers.Thereisfarmoretobedone,butwealreadyknowenoughto
speakoutsideourresearchfield,toworkingspreadsheetdevelopersandtocorporate
managersofendusercomputing.Thisisgood,becausethelessonswehavelearned
mayrequireorganizationstobegintreatingspreadsheetdevelopmentverydifferently
thantheyhaveinthepast.

ASenseofUrgency
Aswewillseebelow,therehaslongbeenampleevidencethaterrorsinspreadsheets
arepandemic.Spreadsheets,evenaftercarefuldevelopment,containerrorsinone
percentormoreofallformulacells.Inlargespreadsheetswiththousandsofformulas,
therewillbedozensofundetectederrors.Evensignificanterrorsmaygoundetected
becauseformaltestinginspreadsheetdevelopmentisrareandbecauseevenserious
errorsmaynotbeapparent.
Despitethislongstandingevidence,mostcorporationshavepaidlittleattentiontothe
prospectofseriousspreadsheeterrors.However,in2002,theU.S.Congresspassed
theSarbanesOxleyAct,whichrequirescorporationstohavewellcontrolledfinancial
reportingsystems.Evenasingleinstanceofafirmhavingamorethanremote
possibilityofamaterialerrororfraudinitsfinancialreportingsystemwillrequire
managementandindependentauditorstogivethefirmanadverseopiniononits
financialreportingsystem.Thisislikelytoresultinlowerstockpricesand,insome
cases,infinesandprisontermsforseniorexecutives.Severalothercountrieshave
developedorarecurrentlydevelopingsimilarcontrolrequirementsforfinancial
reportingsystems.
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

2/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

AsfirmsbegantoprepareforSarbanesOxleyandagrowingnumberofotherlaws
requiringthestrongcontroloffinancialreportingsystem,theycametorealizethat
manyormostfinancialreportingsystemstodayusespreadsheetsandthatthese
spreadsheetsarenotwellcontrolled.
FinancialintelligencefirmCODAreportsthat95%ofU.S.firmsusespreadsheets
forfinancialreportingaccordingtoitsexperience(www.coda.com).
In2004,RevenueRecognition.com(nowSofttrax.com),hadtheInternational
DataCorporationinterview118businessleaders.IDCfoundthat85%wereusing
spreadsheetsinreportingandforecastingandthat85%wereusingspreadsheets
forbudgetingandforecasting.
In2003,theHacketGroup(www.thehacketgroup.com)surveyedmidsize
companies.Itfoundthat47%ofcompaniesusestandalonespreadsheetsfor
planningandbudgeting.
CFO.cominterviewed168financeexecutivesin2004.Theinterviewsaskedabout
informationtechnologyusewithinthefinancedepartment.Outof14
technologiesdiscussed,onlytwowerewidelyusedspreadsheetsandbasic
budgetingandplanningsystem.Everysubjectsaidthattheirdepartmentuses
spreadsheets.
InEurope,A.R.C.Morganinterviewed376individualsresponsibleforoverseeing
SOXcomplianceinmultinationalsthatdobusinessintheUnitedStates.More
than80%oftherespondentssaidthattheirfirmsusespreadsheetsforboth
managingthecontrolenvironmentandfinancialreporting.EuropeanCompanies
aretakingafalteringapproachtoSarbanesOxley.Sept20,2004,TMCnet.com,
(http://www.tmcnet.com/usubmit/2004/Sep/1074507.htm)
Thequestion,"Whatdowedoaboutallthespreadsheets?"isincreasinglyonthe
mindsofcorporateexecutives.
Althoughthisgrowingfocusonspreadsheeterrorswithincorporationsissomewhat
gratifyingforspreadsheeterrorresearchers,italsorequiresnewresearchdirectionsin
spreadsheetresearch.Inthepast,spreadsheetdevelopershavefocusedmostheavily
on"innocenterrors"causedbyhumanfallibilitybutnotbyevilintent.However,to
copewithcorporateneedsraisedbySarbanesOxleyandotherlaws,spreadsheet
researcherswillhavetofocusonhowtodevelopcontrolsnotonlyforinnocenterrors
butforoutrightfraud.Existingtestingtechniquesaimedatfindinginnocenterrorsare
notlikelytofindfraudbasederrors.Newformsofparanoidsoftwaretestingwillbe
needed,andspreadsheetcontrolswillhavetoworkonpreventingandidentifying
fraud.Ifspreadsheetcontrolscanreducefinancialfraud,thiswouldbeamajorbenefit
tocorporations,becausefinancialfraudisalargecurrentproblem.In2004,fraud
throughfinancialstatementsrepresentedonly7%ofallfraudcasesstudiedinan
ACFEsurvey[ACFE,2004],butthemedianfraudlossforfinancialfraudwasamillion
dollars,comparedtoamedianlossofonly$100,000forfraudsingeneral.

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

3/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

ErrorRates
IntroductiontoErrorRates
Sincetheearliestdaysofspreadsheeting,therehavebeenconcernsabouterrorsin
userdevelopedspreadsheets.Davis(Davis,January1982)cautionedthat"user
developedsystemsmaybehazardoustoyourorganization"ingeneral,becauseend
usersseldomimplementthedisciplineslongknowntobenecessaryinthe
developmentofsystemsbuiltwithprogramminglanguages.Withspreadsheet
programs,endusersbecamecapableofdevelopinganalysescontainingthousandsof
cells.Undertheseconditions,unlessthepercentageofincorrectcellsisalmostzero,
therewillbeaveryhighprobabilityofbottomlineerrors.
Overtheyears,anumberofembarrassingspreadsheetdevelopmentincidentshave
beenreported(Panko,2005b).Inmostcases,eitherthefirmthatmadetheerrorwas
forcedtodiscloseitsmistakeorconsultantsfamiliarwiththeerrorrevealedit.Given
thenormalreluctanceoforganizationstotalkabouttheirmistakes,weshouldwonder
iftheseincidentswererarecasesorwhetherspreadsheeterrorsareratherfrequentin
practice.
Morebroadly,anumberofconsultants,basedonpracticalexperience,havesaidthat
20%to40%ofallspreadsheetscontainerrors(Panko,2005b).Inapersonal
communicationwiththeauthor,Dent(1995)describedanauditinaminingcompany
thatfounderrorsinabout30%oftheauditedspreadsheets.Freeman(Freeman,
1996)citesdatafromtheexperienceofaconsultingfirm,CoopersandLybrandin
England,whichfoundthat90%ofallspreadsheetswithmorethan150rowsthatit
auditedcontainederrors.OnePriceWaterhouseconsultantauditedfourlarge
spreadsheetsandfound128errors(Ditlea,1987).
Today,wehavemovedbeyondsuchanecdotalevidence,intotherealmofsystematic
fieldauditsandlaboratoryexperiments.Tables1through3summarizekeydatafrom
thesestudies.
Table1containsdatafrom13fieldauditsinvolvingrealworldspreadsheet
spreadsheets.Since1995,whenfieldauditsbegantousegood(althoughusuallynot
excellent)methodologies,88%ofthe113spreadsheetsauditedin7studiesthat
reportedthenumberofspreadsheetsauditedhavecontainederrors,reflecting
improvementinauditingmethodologies.
Furthermore,severalofthesestudiesonlyreportedmajorerrors.
Table1:StudiesofSpreadsheetErrors
Authors

Year

Number
ofSSs

Average
Size

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

Percent
ofSSs

Cell Comment
Error
4/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Audited

(Cells)

with
Errors
21%

Davies&
Ikin
Cragg&
King

1987

19

1992

20

50to
10,000
cells

25%

Butler

1992

273

11%

Dent

1994 Unknown

30%

Hicks

1995

3,856

100%

Coopers&
Lybrand

1997

23

More
than
150
rows

91%

KPMG

1998

22

91%

Lukasic

1998

2,270&
7,027

100%

Butler

2000

86%

Clermont,
Hanin,&
Mittermeier
Interview
I*

2002

100%

2003

~36/yr

100%

Interview
II*

2003

~36/yr

100%

Lawrence
andLee

2004

30

2,182
unique
formulas

100%

Powell,
Lawson,and
Baker

2007a

25

64%

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

Rate
Onlyseriouserrors

Onlyerrorslarge
enoughtorequire
additionaltax
payments
Errorscausedby
usershardwiring
numbersinformula
cells.Henceforth,all
futurecomputations
wouldbewrong.
1.2% Oneomissionerror
wouldhavecaused
anerrorofmore
thanabillion
dollars.
Offbyatleast5%

Onlysignificant
errors
2.2%, InModel2,the
2.5% investment'svalue
wasoverstatedby
16%.Quiteserious.
0.4%** Onlyerrorslarge
enoughtorequire
additionaltax
payments**
1.3%, Computedonthe
6.7%, basisofnonempty
0.1% cells
Approximately5%
hadextremely
seriouserrors
Approximately5%
hadextremely
seriouserrors
6.9%*** 30mostfinancially
significantSSs
auditedbyMercer
Finance&Risk
Consultingin
previousyear.

11of25spreadsheets
containederrorswith
nonzeroimpacts.
5/31

12/28/2016

Powell,Baker
&Lawson

Total
since
1995

WhatWeKnowAboutSpreadsheetErrors

2007b

50

86%

113

88%

0.9%

Amongtheten
spreadsheetswithnon
zeroimpactsforwhich
errorsizewasreported,
all10hadanerrorthat
exceeded$100,000,6
haderrorsexceeding
$10million,and1had
anerrorexceeding$100
million.
Percentofformulacells
containingerrorsthatgivean
incorrectresult.Including
poorpractices,1.8%ofall
formulashadissues.Including
poorpractices,94%ofthe
spreadsheetshadissues.

*In2003,theauthorspokeindependentlywithexperiencedspreadsheetauditorsintwodifferentcompaniesintheUnitedKingdom,
wherecertainspreadsheetsmustbeauditedbylaw.Eachauditedaboutthreedozenspreadsheetsperyear.Bothsaidthattheyhadnever
seenamajorspreadsheetthatwasfreeoferrors.Bothalsoindicatedthataboutfivepercentofthespreadsheetstheyauditedhavevery
seriouserrorsthatwouldhavehadmajorramificationshadtheynotbeencaught.Auditsweredonebysingleauditors,sofromthe
researchonspreadsheetandsoftwareauditing,itislikelythathalforfewoftheerrorshadbeencaught.Inaddition,virtuallyallofthe
spreadsheetshadstandardformatsrequiredfortheirspecificlegalpurposes,soerrorratesmayhavebeenlowerthantheywouldbefor
purposebuiltspreadsheetdesigns.
**Thelowcellerrorrateprobablyreflectsthefactthatthemethodologydidnotinspectallformulasinthespreadsheetbutfocusedon
higherriskformulas.However,errorhasastrongrandomcomponent,sonotcheckingallformulasislikelytomissmanyerrors.
***Unlikeotherauthors,LawrenceandLee(2004)measured"issues"ratherthanonlyquantitativeerrors.Thisexplainswhytheaverage
"errorrate"ishigherthanthoseseeninotherstudies.

Inturn,Table2containsdatafromexperimentsinvolvingoverathousandsubjects
rangingfromnovicestohighlyexperiencedspreadsheetdevelopers.Althoughthereis
greatdiversityinmethodologyanddetailedresults,onekeypatternstandsout
clearly:everystudythathasattemptedtomeasureerrorshasfoundthemandhas
foundtheminabundance.
Table2:StudiesofSpreadsheetErrors
Study

SSs

%with
Errors

CellError Remarks
Rate
(CER)

Brown&Gould(1987)

27

63%

9experienced
spreadsheetdevelopers
eachbuilt3SSs.Each
developermadeatleast
oneerror.

Olson&Nilsen(1987
1988)

14

21% Measurederrorsasthey
occurred,evenifthe
subjectcorrectedthem.
CERonlyreflects
formulacells

Lerch(1988)

21

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

Measurederrorsasthey
occurred,evenifthe
subjectcorrectedthem
6/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

CERonlyreflects
formulacells
Allformulacells

9.3% CERonlyreflects
formulacells

Referencesto
cellsinthesame
row

6.4% CERonlyreflects
formulacells

Referencesto
cellsinthesame
column

4.7% CERonlyreflects
formulacells

Referencesto
cellsindifferent
rowsand
columns

14.1% CERonlyreflects
formulacells
Noviceprogrammers
eachdevelopedseveral
spreadsheets

Hassinen(1988)

92

55%

Janvrin&Morrison1st
(2000a)

61

4.3% CERonlyreflects
formulacells
Taskwith50links
betweenspreadsheets.
CERonlyreflectssome
formulacellsofhigh
riskmostlylinks
betweenspreadsheets
Differenceswere
nonsignificant

Workingalone,
nodesign
training

14%

Workingalone,
designtraining

7%

Workinginpairs,
nodesign
training

7%

Workinginpairs,
designtraining

7%

Janvrin&Morrison2nd
(2000b)

Taskwith66links
CERonlyreflectssome
formulacellsofhigh
riskmostlylinks
betweenspreadsheets
Differenceswere
nonsignificant

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

7/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Notrainingin
design

30

16.8%

Trainingin
design

58

8.4%

73

42%

2.5%

168

42%

2.1% Walltask.

Workingalone

42

79%

5.6%

Dyads(Groups
of2)

46

78%

3.8%

Tetrads(Groups
of4)

44

64%

1.9%

Panko&Halverson(2001)

35

86%

Panko&Halverson(2001)

Panko&Sprague(1998)

26

35%

2.1% WallTask.MBAstudents
withlittleornoSS
developmentexperience

Panko&Sprague(1998)

17

24%

1.1% WallTask.MBAstudents
with250hoursormore
SSdevelopment
experience

Kreie,etal.(posttest)
(2000)
Teo&Tan(1997)
Panko&Halverson(1997)

Galumpketask.
Undergraduates
Basedonalltextand
numbercells

4.6% Undergraduates

Finally,Table3looksatcodeinspectionexperiments.Intheseexperiments,subjects
examinedspreadsheetsseededwitherrors.Theresearchshowsthatpeoplearenot
onlypronetomakeerrors.Theyareonlymoderatelygoodatcorrectingtheseerrors.
Table3:CodeInspectionExperiments
Study

Subjects

Gallettaetal.(1993)

MBAstudents&
CPAsTakinga
Continuing
EducationCourse

Totalsample

60

56%

CPAnovices

<100hoursof
workexperience
withSSs

15

57%

CPAexperts

<100hoursof
workexperience
withSSs

15

66%

MBAstudents,novices

>250hoursof

15

52%

%of Remarks
Errors
Detected
Budgetingtask

containing
seedederrors

Sample
Size

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

8/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

workexperience
withSSs
MBAstudents,
experienced

>250hoursof
workexperience
withSSs

15

48%

Domain(logic)errors
corrected
Device(mechanical)

errorscorrected
Gallettaetal.(1997)

46%

65%

MBAstudents

Overall

113

51% Sametask
used1993
study
51%

OnScreen

45

45%

OnPaper

68

55%

Panko(1999)

Undergradsworking
alone
Undergradsworking
ingroupsofthree
Panko&Sprague(1998)
Undergrads

60

Modified
versionof
Gallettawall
task.
63%

60

83%

23

16% Studentswho
madeerrorsin
theWalltask
whothenwent
ontoinspect
theirown
spreadsheets.

ConsistentwithOtherHumanErrorData
WhenmostpeoplelookatTables12,and3,theirfirstreactionisthatsuchhigherror
ratesareimpossible.Infact,theyarenotonlypossible.Theyareentirelyconsistent
withdataonhumanerrorratesfromotherworkdomains.TheHumanErrorWebsite
(Panko,2005a)presentsdatafromanumberofempiricalstudies.Broadlyspeaking,
whenhumansdosimplemechanicaltasks,suchastyping,theymakeundetected
errorsinabout0.5%ofallactions.Whentheydomorecomplexlogicalactivities,such
aswritingprograms,theerrorraterisestoabout5%.Thesearenothardandfast
numbers,becausehowfinelyonedefinesreported"action"willaffecttheerrorrate.
However,thelogicaltasksusedinthesestudiesgenerallyhadaboutthesamescope
asthecreationofaformulainaspreadsheet.
Themostcompletesetofdataonerrorratescomesfromprogramming,whichisat
leastacousinofspreadsheetdevelopment.Inprogramming,manyfirmspracticecode
inspectiononprogrammodules.Incodeinspection,teamsofinspectorsfirstinspect
themoduleindividuallyandthenmeetasagrouptogooverthemoduleagain
(Fagan,1976).Significantly,thereisarequirementtoreportthenumberoferrors
foundduringcodeinspection.Thishasresultedinthepublicationofdatafromliterally
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

9/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

thousandsofcodeinspections(Panko,2005a).Thedatafromthesestudiesshows
strongconvergence.Codeinspectionusuallyfindserrorsinabout5%ofallprogram
statementsafterthedeveloperhasfinishedbuildingandcheckingthemodule(Panko,
2005a).Whilethereissomevariationfromstudytostudy,muchofthisvariation
appearstobedotodifferencesinprogramminglanguage,moduledifficulty,and,
sadly,inhastinessindevelopment.
Notethatwehavebeentalkingaboutuncorrectederrors.Humansmakemanyerrors
whentheywork,buttheyalsocorrectmany.Correctionratesarehighformechanical
errors,buttheyareconsiderablylowerforlogicerrors,andifwehaveanomission
causedbyamisdiagnosisofthesituation,thecorrectionrateisverylow.
Interestingly,datafromastudybyRabbitandVyas(Rabbit&Vyas,1970)suggest
thatagreatdealoferrorcorrectiontakesplacebelowthelevelofawareness,so
peopleareunawareofhowmanyerrorstheymakeandhowmanytheycorrect.
Thereisevenanemergingtheoryforwhywemakesomanyerrors.Reason(Reason,
1990)haspresentedthemostcompleteframeworkforunderstandingwhyhuman
beingserr,butmanyotherwritershaveaddedtheirowncontributions(Baars,1992).
Ingeneral,theemergingtheoryarguesthathumanbeingsareamazinglyfastand
flexible(Reason,1990)andcanjugglemultipletasksandconstraints(Flower&
Hayes,1980).However,thesamecognitiveprocessesthatallowthemtoworkthis
wayinevitablyresultinoccasionalerrors.Overall,AlexanderPopewrotethat"Toerris
human."Today,wecanbothexplainandquantifythatstatement.

MeasuringErrors
ThefieldauditstudiesshowninTable1reportasimplemeasureofspreadsheeterrors
thepercentageofspreadsheetsthatcontainatleastoneseriouserror.(Minorerrors
arediscounted).Whilethisisavividstatistic,itdoesnottellushowmanyerrorsan
averagespreadsheetcontains.
Inaddition,wehavelongknownthatintasksthatcontainmanysubtasks,errorrates
willmultiplyalongcascadesofsubtasks.Inspreadsheeting,forinstance,many
bottomlinevaluesarecomputedthroughlongcascadesofnumericalandformula
cells.LorgeandSolomon(1955)gaveseveralformulasforcalculatingbottomline
errorrates.Equation1isadaptedfromoneoftheirsimplerformulas.Here,Eisthe
bottomlineerrorrate,eisthetaskerrorrate(assumedtobethesameforalltasks),
andnisthenumberoftasksteps.
Equation1:E=1(1e)^n
Inprogramming,thislinkbetweenindividualerrorratesandcascadeerrorratesis
addressedbycomputingtheerrorrateperlineofcode.Morespecifically,
programmersmeasurethenumberoffaultsperthousandlinesofnoncomment
sourcecode(faults/KLOC).Thisstatementerrorrateisroughlyindependentof
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

10/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

programsize(Putnam&Myers,1992).Althoughitisimportanttomakeadjustments
forcomplexity,size,andothermattersforfineanalysis(Ferdinand,1993),andwhile
itwouldbebettertomeasurefaultsperfunctionpoint,faults/KLOCisagoodrough
measureofprogrammingerrors.
Inspreadsheets,asimilarmetricisthecellerrorrate(CER).Thisisthenumberof
errorsdividedbythecombinednumberofnumericalandformulacells.(Labelcells
arelikecommentcells,whichareignoredinfaults/KLOC).Equivalently,thisisthe
percentageofnonlabelcellscontainingerrors.Table1showsthatjustasfaults/KLOC
usuallyfallsintoanarrowrange(Panko,2005a),theCERsseeninspreadsheet
studieshavebeenverysimilar.Therelativelyfewdivergentnumbersthatappear
withinalifecyclestage,furthermore,comewhenonlycertaincellsareconsidered
generallycellswithhighpotentialsforerror.

ErrorsbyLifeCycleStage
Inprogramming,errorratesvarybylifecyclestage.Programmersmakemanyerrors
whentheyarebuildingamodulebutcatchmanyoftheseerrorsbeforetheyfinishthe
moduleandsubmitittocodeinspection,datatesting,orboth.Modulecodeinspection
andtestingcatchmoreerrors,andasecondwaveofcodeinspectionandtestingat
finalassemblycatchesstillothererrors.Relativelyfewerrorsshouldsurviveintothe
finaloperationalsystems.

ErrorsDuringCellEntry
Onlytwostudieshavelookedaterrorsduringcellentry(Lerch,1988Olson&Nilsen,
19871988),thatis,whenthedeveloperisenteringformulas.Onemorestudylooked
aterrorswhenthesubjectsweretypinglabelcells(Floyd&Pyun,1987).Thesedata,
whilelimited,confirmthatpeoplemakemanyerrorswhiletheyworkandcatchquite
afewofthem(butnotall).Inthemethodologiesused,subjectswereinterruptedby
theexperimentertocorrecterrorsaserrorsweremade,sowedonotknowfinalerror
correctionratesfromthesestudies.
OneinterestingpatternfromtheLerch(1988)studyisthatmechanicalerrorrates
rosedramaticallywhenequationscontainedreferencestocellsthatwereinboth
differentcolumnsanddifferentrowsthanthecellcontainingtheformula.

ErrorsattheEndoftheDevelopmentStage
MoststudiesinTable2lookedaterrorsattheendofthedevelopmentstage,when
subjectssaidthattheywerefinisheddevelopingtheirspreadsheets.Apartfrom
researchthatlookedonlyatselectedhighriskformulas(Janvrin&Morrison,1996
2000),cellerrorratesacrossthesestudiesweresimilar,despitethefactthatsubjects
rangedfromnovicestohighlyexperiencedspreadsheetdevelopers.Onestudy(Panko
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

11/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

&Sprague,1998)evencomparedundergraduatebusinessstudents,MBAstudents
withlittlespreadsheetdevelopingexperience,andMBAstudentswithmorethan250
hoursofspreadsheetdevelopmentexperience.TheirCERswereverysimilar.Even
whenatask(theWalltask)wasselectedtobeverysimpleandalmostcompletelyfree
ofdomainknowledgerequirements(Panko&Sprague,1998Teo&Tan,1997),about
40%ofallspreadsheetscontainederrors,andtheCERwasabout2%.

ErrorsFoundandMissedinCodeInspection
Codeinspectionorsomeotherformofintensivetestingmaybeneededtodetect
errorsthatremainattheendofthedevelopmentstage.Fagan(Fagan,1976)
developedacomprehensivedisciplineforcodeinspection.Asnotedearlier,this
methodhasagroupofindividualsinspectaprogrammoduleindividuallybygoing
throughthemodulelinebyline.Then,theteammeetsasagroupandagaingoes
throughthemodulelinebyline.Therearestrongguidelinesforthingssuchas
maximumlengthforthemodulebeingexamined(usually100to400linesofcode)
andmaximuminspectionrates(usually100to200linesperhour).Iftheseratesare
exceeded,errordetectionratesfallconsiderably(Panko,2005a).
Experiencehasshownthatcodeinspectionisverydifficult.Inexperiments,ithas
beenfoundthatindividualinspectorscatchonlyhalforfewerofallerrorsinseeded
programmodules(Basili&Selby,1986Johnson&Tjahjono,1997Myers,1978).
Evenrealworldteaminspectionusuallyonlycatches80%orfewerofallerrorsina
programmodule(Panko,2005a).
Incodeinspection,asindevelopment,spreadsheetinghasshownstrongparallelsto
programming.Inthreeexperimentsthatinvolvedindividualsubjectscodeinspecting
spreadsheetsseededwitherrors,subjectsalsocaughtabouthalfofallerrorsorfewer
(Gallettaetal.,1993Galletta,Hartzel,Johnson,&Joseph,1997Panko&Sprague,
1998).Thesestudies,however,didnotimposetraditionaltimedisciplinesorateam
codeinspectionphase.Inonestudy(Panko,1999),traditionaltimedisciplinesanda
groupinspectionphasewereusedinacodeinspectionofaspreadsheetby
undergraduateMISmajors.Theindividualsfound67%ofallseedederrorsinthis
study,andgroupcodeinspectionbroughtthediscoveryrateto92%.Howeverthe
groupsdiscoverednoerrorsthathadnotbeendiscoveredduringtheindividualcode
inspectionphase.Infact,onegroupfailedtorecordanerrorduringthegroupphase
thatoneofitsmembershadcaughtduringtheindividualphase.

ErrorsinOperationalSpreadsheets
Laboratoryexperimentsalwaysraisequestionsinthemindsofmanypeople.
Fortunately,wehavedatafromcodeinspectionsofoperationalspreadsheets,beyond
theHicksstudyjustdescribed.Theseinspectionslackedtherigorousapproachusedin

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

12/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

formalprogrammingcodeinspection,however,sotheyprobablyfoundonlyafraction
oftheerrorsinthespreadsheetstheyexamined.
Thefirststudy,byDaviesandIkin(Davies&Ikin,1987),inspected19operational
spreadsheetsfrom10developersin10differentfirms.Subjectsexpressedconfidence
inthecorrectnessoftheirspreadsheets.Howeverfourofthespreadsheets(21%)
werefoundtohaveseriousquantitativeerrors,and76%hadquantitativeor
qualitativeerrors.Oneerrorinvolveda$7millionfundstransferbetweendivisions.In
anothercase,therewereinconsistentcurrencyconversionnumbersindifferentparts
ofthespreadsheet.Athirdproblemwasanegativebalanceforstockonhand.There
wasinadequatedocumentationin68%ofthespreadsheets.Tenofthe19failedto
usecellprotection.Onlyonehadusedauditsoftwaretoauditthespreadsheets,and
manualauditswere"rare."Unfortunately,theauthorsdidnotdescribetheirinspection
methodology.
Later,CraggandKing(Cragg&King,1993)inspected20operationalspreadsheets
from10firms.Thisauditfoundseriouserrorsin5ofthespreadsheets(25%).This
time,theauthorsdescribedtheirmethodology.Asinglepersonconductedtwohour
codeinspections.Becausethespreadsheetsrangedfrom150to10,000cellsinsize,
thecodeinspectiontimewasmuchshorterthancodeinspectiondisciplineswould
allow.Theauthorsthemselvesnotedthattheyprobablycaughtonlysomeofthe
errorsinthesespreadsheets.Theauthorsnotedthatthespreadsheetswere
characterizedbypoorlayout,theuseofcellprotectioninonly30%,theuseofrange
namesinonly45%,andotherdesignflaws.Onlyhalfhadclearlyseparatedsections
forinputandoutput,andallmixedcalculationcellswithoutputcells.Onlytwohad
checkedformulasbeforeenteringtheminthespreadsheet.Eightypercenthadbeen
checkedwithinputdata,althoughthedepthofsuchcheckingvaried.Onlyonehad
previouslybeenexaminedbyanotherperson.Halfusedmacros,indicating
considerablecomplexityandsophistication.
OneuniversalaspectoftheCraggandKing(Cragg&King,1993)spreadsheetswas
informaliterativedevelopment.Noneofthespreadsheetsusedformalspecification,
design,orcoding.Inaddition,therewasextensiverevisionbecauseofpoorinitial
design.Therehadbeenamedianofsevenrevisions,despitethefactthatthe
spreadsheetshadonlybeeninuseamedianofsixmonths.In17ofthespreadsheets,
thestructureofthespreadsheethadtobechangedduringrevisions.Insixcases,
therehadbeenproblemswiththespreadsheetinpreviousreleases.
Later,inapersonalcommunicationwiththeauthor,Butler(1996)describedtheaudit
of273spreadsheetssubmittedtoHMCustomsandExcisetaxagencyintheUnited
Kingdom.Theauditmethodologyinvolvedtheuseofaprogramdesignedtolookfor
inconsistenciesinthespreadsheet.Iftheprogramtaggedaportionofthe
spreadsheet,anindividualauditorexaminedthatportionofthespreadsheetfor
errors.Thisprogramprobablycaughtonlyafractionofallerrors,becauseitcouldnot
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

13/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

catchsuchthingsasomissions,numbersbeingmistyped,mostincorrectformulas,
andmanyothertypesoferrors.Nevertheless,theteamfounderrorsin10.7%ofthe
spreadsheets.
Since1995,fivestudieshaveexaminedspreadsheetsusingbettermethodology.Hicks
(1995)auditedanenormouscapitalbudgetingspreadsheetwith3,856cells.Thiswas
athreeperson,cellbycellcodeinspection.Errorswerefoundinonly1.2%ofalllines
ofcode,buttheerrorswouldhavebeenoverabilliondollarsiftheyhadnotbeen
caught.
InEngland,certainspreadsheetshavetobeauditedbylaw.BothCoopersand
LybrandandKPMGhaveactivespreadsheetauditingconsultancies.Together,they
foundsignificanterrorsin45ofthespreadsheetstheyaudited.Coopers&Lybrand
onlycountederrorsiftheymadesomebottomlinevalueoffbyatleast5%.KPMG
onlylisted"majorerrors."In2003,theauthorinterviewedauditorsfromthesetwo
organizationsseparately.Bothsaidthattheyhadneverseenanerrorfree
spreadsheetduringaudits.Inaddition,bothindependentlygavedataindicatingthat
about5%ofallspreadsheetstheyauditedhavehad"veryseriouserrors."Theynoted
thatthispercentagemaybelowforothertypesofspreadsheetsbecausethe
spreadsheetsbeingauditedhaveawelldefinedformat.Inaddition,both
aconsultanciesonlyusesinglepersoncodeinspection.
In2000,RayButler,whoauditedspreadsheetsforexcisetaxcollectionintheUK,
foundactionableerrorsinsixofsevenspreadsheetsauditedusingsinglepersoncode
inspection.Actionableerrorsincludeddemandsforthepaymentofadditionaltaxesor
providingrefunds.(Inhisearlierstudy,Butlerusedalessrigorousmethoddesigned
tominimizethecostofauditingandonlyfounderrorsin10.7%oftheaudited
spreadsheets).
Mostrecently,LawrenceandLee(2004)inAustraliaaudited30spreadsheetscreated
tojustifyprojectfinancing.Theseauditsusedsinglepersoncodeinspection.The
spreadsheetsaveraged2,182formulas,andonaverage6.9%oftheformulacellshad
auditingissues.Ittookanaverageofsixiterationsbeforethespreadsheetscouldbe
signedoff.
Overall,whatweseeinoperationalspreadsheetsisconsistentwithdatafrom
laboratorystudies.Althoughlaboratorystudiesgenerallyhavehighererrorrates,field
studiesdidnotusemethodologiesthatprobablycouldfindmosterrors.Inthefour
fieldauditsforwhichwehavecellerrorrates,furthermore,theCERs,whichrange
from0.4%to6.9%,aresimilarinmagnitudetoCERsseeninexperiments.In
addition,moststudiesusedsinglepersoncodeinspection,whichmissessomeerrors.

DoesGroupworkHelp?
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

14/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Intheirethnographicstudyof11spreadsheetdevelopers,NardiandMiller(Nardi&
Miller,1991)notedthatalldevelopersusedatleastoneotherpersonduring
development.Insomecases,theotherpersonhelpedthedevelopercodedifficult
partsofthespreadsheet.Inothercases,twoormorepeopleusedthespreadsheet
developmentprocessasawaytosharedomainknowledge.Inothercases,theother
personexaminedinputnumbersandresultsforreasonableness,therebyhelpingwith
errordetection.NardiandMillercalledthislastpattern"cooperativedebugging."
ThisledPankoandHalverson(Panko&Halverson,1997)toexaminegroup
spreadsheetdevelopment.Theirsubjectsdevelopedaproformaincomestatement
fromawordproblem,workingalone,ingroupsoftwo(dyads),oringroupsoffour
(tetrads).Dyadsreducederrorsbyaboutathird,whiletetradsreducederrorsby
abouttwothirds.Onlythelatterwasstatisticallysignificant.Furthermore,group
developmentwasonlyeffectiveforcatchingcertaintypesoferrors.
TheoriginalPankoandHalverson(Panko&Halverson,1997)studyallowedsome
subjectstoworkoutsidethelaboratory,inordertoavoidtheartifactsofhavingto
workundercontrivedconditions.Ofcoursethisalsolostexperimentalcontrol.In
a1999,PankoandHalversonredidthestudyinthelaboratory,usingindividualand
triadicdevelopment.Theyalsorewordedtheirtasktoremoveambiguitiesandone
portionthatsubjectsinthefirststudyfoundextremelydifficult.Takingthesechanges
intoaccount,thenewstudygavealmostidenticalresults.
Finally,asnotedearlier,astudybytheauthor(1999)foundthatteamcodeinspection
allowedundergraduateMISmajorstofind83%ofallseedederrorsinaspreadsheet,
althoughthegroupdidnotfinderrorsnotpreviouslyfoundbythemembersofthe
team,whohadinspecteditalonebeforethegroupcodeinspection.
Overall,groupdevelopmentandtestingappeartobepromisingareastopursue.

TypesofErrors
Whenmanypeoplethinkofspreadsheeterrors,theythinkofsomeonemistypinga
number,typingaplussignforaminussigninaformula,orpointingtothewrongcell
whenenteringaformula.Whiletheseerrorsdoexist,therearemanyothertypesof
errorsinspreadsheetdevelopment.PankoandHalverson(Panko&Halverson,1996)
giveataxonomyoferrortypes.
First,therearequantitativeerrors,inwhichthespreadsheetgivesanincorrectresult.
ThestudiesinTable1lookonlyatquantitativeerrors.Howevertherearealso
qualitativeerrorsthatmayleadtoquantitativeerrorslater,duringmaintenance,
whatifanalysis,orotheractivities.Reason(Reason,1990)usestheterm"latent
errors"forsuchproblems.TeoandTan(Teo&Tan,1997)demonstratedinan

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

15/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

experimenthowonetypeofqualitativeerrorledtoquantitativeerrorsduringwhatif
analysis.
PankoandHalverson(Panko&Halverson,1996),followingAllwood(Allwood,1984)
alsofounditusefultodistinguishbetweenthreetypesofquantitativeerrors.
Mechanicalerrorsaresimplemistakes,suchasmistypinganumberorpointingtothe
wrongcell.Logicerrorsinvolveenteringthewrongformulabecauseofamistakein
reasoning.Asnotedearlier,logicerrorratesarehigherthanmechanicalerrorrates.
Logicerrorsalsomoredifficulttodetectandcorrect(Allwood,1984).Themost
dangeroustypeoferroristheomissionerror,inwhichsomethingisleftout.Omission
errorsappeartobeextremelydifficulttodetect(Allwood,1984Bagnara,Stablum,
Rizzo,Fontana,&Ruo,1987Woods,1984).
WhenPankoandHalverson(Panko&Halverson,1997)analyzedthetypesoferrors
thatsubjectsmadewhiledevelopingaspreadsheet,theyfoundthatallthreeformsof
errorswerecommon.Later,PankoandSprague(Panko&Sprague,1998)foundthe
samebroadpatternoferrors.PankoandHalversoncompareddifferenttypesoferrors
tomultiplelethalpoisons.Evenifallerrorsoftheothertwotypeswereeliminated,
eachtypeoferroralonewouldhaveproducedanunacceptablenumberofincorrect
spreadsheets.
Techniquestoreduceerrorsmaybebetteratfindingsometypesoferrorsthan
others,soitisimportanttobegindevelopingcorpuses(collections)oferrorstolearn
howfrequentdifferenttypesoferrorsreallyare.

DevelopmentPracticesandPolicies
DeveloperSurveys
Afewstudieshaveexaminedtheprocessesthatdevelopersusetocreate
spreadsheets.Thesestudiesaskedthedevelopertoselectasinglespreadsheetand
describeitindetail.Thismethodpresumablyresultedinbiastowardlargerthan
averagespreadsheets.Evenwiththiscaveat,however,theresultshavebeen
extremelyinteresting.
TheDaviesandIkin(Davies&Ikin,1987)andCraggandKing(Cragg&King,1993)
fieldauditsmentionedearlierquestioneddevelopersabouttheirspreadsheet
developmentpractices.Bothfoundveryinformaldevelopmentprocesses,little
systematiccodeinspection,andlittleuseofmanyotherimportantdevelopment
disciplines.CraggandKingnotedthathalftheirspreadsheetshadbeenbuiltwithout
priordesignorplanning,almosthalfhadpoordesign,halfhadpoorlayout,andonly
halfhaddocumentationofanytype.Oneoftheirsubjectscommentedthatheorshe
hadtroubleunderstandingoldspreadsheets.
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

16/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Earlier,wementionedtheethnographicinterviewsofNardiandMiller(Nardi&Miller,
1991).Ingeneral,theyfoundthatdeveloperstookconsiderableprecautions
developingtheirspreadsheets,althoughtheinterviewsdidnotgatherquantitative
data.Therespondentswerekeenlyawareofthedangersoferrorandspent
considerabletimeworkingtoreduceerrors,includingthecreationofcrossfootings,
doingspotchecksofformulaswithcalculators,andhavingotherslookatoutput.
Howeveronlyoneofthe11intervieweessaidthattheyhaddoneacompletecode
inspectionofaspreadsheetthiswasaspreadsheetgiventoherbysomeoneelse
(Nardi,1993).Asnotedabove,somehadotherslookattheirresults,becauseerrors
couldbecomeinvisibletotheauthorthroughoverfamiliarity.Howeverthisfallsfar
shortofcodeinspection.
Later,HendryandGreen(Hendry&Green,1994)didasimilarethnographicstudy,
thistimewith10spreadsheetdevelopers.Theyaddedaphaseinwhichthesubject
walkedthroughaspreadsheetwithanauthor.Theyfoundthatsubjectsoftenhada
difficulttimeexplainingpartsofthespreadsheetthattheyhadthemselvesbuilt.They
alsofoundthatsomeofthedevelopershadrunintoseriousproblemstryingtomake
partsofthespreadsheetanalysisfittherowandcolumnlayoutofspreadsheet
program.Thesubjectsfounddebuggingtobeparticularlydifficult.Asaconsequence
ofsuchdifficulties,thesubjectsdidanumberofthingstoanticipateerrors.
Retrospectivelyincommunicationwiththeauthor,Hendry(1994)saidthatonlythree
ofthedeveloperswere"highlynumerate"anddidmanythingstoavoiderrors.In
anotherpersonalcommunicationwiththeauthor,Green(1994)saidthatdetailedcode
inspectiondidnotappeartobepartofthespreadsheetdevelopment"culture."
Overall,bothNardiandMiller(Nardi&Miller,1991)andHendryandGreen(Hendry&
Green,1994)foundthatsubjectstookconsiderablepainswhentheybuilt
spreadsheets.Yetgiventheproblemsthatusersencountered,whichwereespecially
evidentwiththeHendryandGreenmethodology,itisunclearhowsuccessfuluser
errorreductionerrorsare.Giventheinfrequencyofcomprehensivetesting,however,
completesuccessseemsunlikely.
Otherstudieshaveusedquestionnairesurveystocollectdatafromlargernumbersof
respondents.Forinstance,SchultheisandSumner(Schultheis&Sumner,1994)had
32MBAstudentseachdiscusstheirmostrecentspreadsheet.Sizesforthe
spreadsheetswerenotgiven,andaquarteroftherespondentshaddevelopedfiveor
fewerspreadsheets.Thestudyhadrespondentsrateeachspreadsheeton11clusters
ofrisksandtheuseof9clustersofcontrolpracticesduringdevelopment.Eachcluster
hadseverallistedrisksorcontrolpractices.Themostfrequentriskscitedbysubjects
includedthelifeexpectancyoftheapplication(whichmayexceedthetenureofthe
developeratthefirm),usebeyondthedeveloperwithinfirm,usebeyondthefirm,the
numberofusers,andthelevelofcomplexityofthespreadsheet.

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

17/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Higherriskspreadsheetsusedslightlymorecontrols(6.7)thanlowerrisk
spreadsheets(4.6),buttheuseofcontrolswaslowinbothcases.Themost
widespreadcontrolwasverificationofthelogic,butthisinvolvedagrabbagof
approachesofdifferentsophistication.Documentationrankedthird,afterparticipation
informaltraining,but"therewasverylittledocumentation,"withaquarterofthe
spreadsheetshavingnoneatall.Formalauditsandformalreviewswereamongthe
leastusedcontrols.Ofthe32spreadsheets,onlyfivewerereviewedbyanauditoror
consultant,andonlytenhadtheirdesignsreviewedbyanyoneotherthanthe
developer.
Onalargerscale,Floyd,Walls,andMarr(Floyd,Walls,&Marr,1995)sent
questionnairesto72endusersinfourcorporations.Eachwasaskedtodescribea
singlespreadsheet.Theaveragesizeoftheselectedspreadsheetswas6,000cells.
Thesubjectssaidthattheirspreadsheetswereimportantinthesensethatimportant
decisionswerebasedonthem.Theyalsosaid,althoughtoalesserextent,thattheir
spreadsheetshadamaterialfinancialimpactonthefirm.Materiality,bytheway,was
notcorrelatedwithsize,and"smallandlargespreadsheetswereequallylikelyto
supportmaterialdecisions."(41).Eightypercentsaidthattheywouldbeabletowork
withoutthespreadsheet,althoughwithsomedifficulty.Oneinsix,however,saidthey
wouldnotbeabletoworkwithoutthespreadsheet.Developmentwasstrongly
iterative.Therewasahighlevelofconfidencethatthespreadsheetdescribedwas
correct.
Hall(Hall,1996)surveyed106spreadsheetdevelopersinAustralia.Theaveragesize
oftheselectedspreadsheetswas216KB,andbecausemostspreadsheetswere
developedusingLotus123,thisimpliedconsiderablesize.Amongherfindingswere
thefollowing:
1.Thespreadsheetswereimportant.Only7%ofthespreadsheetswereoflow
importance39%wereofhighimportance.27%modifiedexistingcorporate
data49%creatednewcorporatedata.67%wererunonaregularbasis,16%
occasionally,andonly17%onceorafewtimes.Only17%hadresultsusedonly
bythedeveloper,while23%hadresultsusedbymanydepartmentsinthe
organization,and29%hadresultsusedoutsidethefirm.
2.Thespreadsheetswerecomplex.45%usedmacros.36%hadlinkstoother
spreadsheets.21%hadlinkstodatabases.47%usedIFfunctions.66%used
absolutereferencing.
3.Developmentusedonlylimitedsafeguards.Onlyhalfhadmodulardesigns.Only
49%usedcellprotection.Only46%usedcrossfooting.
4.Datatestingwaslimited.71%checkedformulaswithdata,butonly50%didso
withresultsdeterminedaheadoftime,only42%usedtestdatawitherrors,and
only33%usedtestdataatthelimitsofthenormalrange.
5.Inspectionwaslimited.Only17%werecheckedbyanotherdeveloper,5%byan
internalauditor,and6%byanexternalauditor(therewasoverlapinthese
percentages).
6.18%saidthattheirprojecthadbeenrushed.

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

18/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Interestingly,Hallaskedrespondentsiftheyusedanumberofcontrolsandwhether
theyshouldhaveusedthesecontrols.Foralmostallcontrols,moredevelopers
acknowledgedthattheyshouldhaveuseditthansaidthattheyactuallydiduseit.In
general,experienceddevelopersweremorelikelytousemostcontrolsthan
inexperienceddevelopers,butthedifferencewasonlymodest.
Overall,thesestudiesshowthatmanyspreadsheetsarelarge,complex,important,
andaffectmanypeople.Yetdevelopmenttendstobequiteinformal,andeventrivial
controlssuchascellprotectionarenotusedinmostcases.Inprogramming,code
inspectionanddatatestingareneededtoreduceerrorratesafteramoduleis
developed.Yetcodeinspectionisveryinfrequent,andwhiledatatestingisdone,it
lackssuchrigorsastheuseofoutofboundsdata.Ingeneral,enduserdevelopment
inspreadsheetingseemstoresembleprogrammingpracticeinthe1950sand1960s.

CorporatePolicies
Onereasonwhyindividualspreadsheetdevelopersdonotuserigorousdevelopment
disciplinesmaybethefactthatfeworganizationshavepoliciesonenduser
developmentingeneral,muchlessspreadsheetdevelopment.
GallettaandHufnagel(Galletta&Hufnagel,1992)surveyed107MISexecutivesusing
amailquestionnairethatfocusedbroadlyonpoliciesforcontrollingenduser
development.Forrestrictionsonenduserdevelopment,23%saidtheyhadrulesand
58%saidtheyhadguidelines.Yetevenforcorporationswithrulesandguidelines,the
respondentssaidthattherewasfullcomplianceonly27%ofthetime.Whenasked
aboutrequirementsforpostdevelopmentaudits,only15%saidtheyhadrules,and
another34%saidtheyhadguidelines.Therespondentssaidthatwheretheyhad
rulesandguidelinesforpostdevelopmentaudits,furthermore,therewasfull
complianceonly10%ofthetime,andguidancewascompletelyignored41%ofthe
time.
Cale(Cale,1994)surveyed52ISandnonISmanagersin25firms.This
questionnairefocusedontestinganddocumentation.Calefoundthatonly10%ofthe
companieshadwrittenpoliciesfortesting,whileaboutanotherquarterhad"unwritten
policies."Documentationpercentageweresimilar.About70%stronglyagreedthat
thislackoftestingstandardswasproducingseriousproblems.Nonedisagreed.About
90%wouldimposetestingstandardsforspreadsheetstakingaweekormoreto
develop,ifthespreadsheetwereusedbymultipledepartments,oriftheoutputwould
updatedatabasesinthedevelopersdepartmentorotherdepartments.
Floyd,Walls,andMarr(Floyd,etal.,1995),intheirsurveymentionedearlier,found
thatonlyabout15%ofthefirmshaddevelopmentpolicies,40%implementation
policies,andtwothirdsdevelopmentmodificationpolicies.Thosewhosespreadsheets
wereratedasmoreimportantthantheaveragedidnothaveafullersetofpoliciesfor
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

19/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

development.Noneoftherespondentsreportedacomprehensivesetofpoliciesforall
phasesofdevelopment.Furthermore,thepoliciesthatdidexistwerecreatedand
implementedattheworkgrouplevel.Floyd,Walls,andMarrreferredtothe
workgroupcentricnatureofpoliciesa"clanbased"socialpolicybasedprimarilyon
socialization.Norespondent,however,knewofaspreadsheetdisasterintheirfirm.
SpeierandBrown(Speier&Brown,1996)surveyed22membersofthree
departmentsaboutendusercontrolpoliciesingeneral.Thecompanyhadfew
corporatewiderulesexceptforbackuprules,whichwerenotenforcedanyway.Most
"rules"wereinformalanddepartmentalanddifferedwidelyacrossdepartmentsand
withintheperceptionsofdifferentusersineachdepartment.Thisreinforcestheclan
basedcontrolconceptofFloyd,Walls,andMarr(Floydetal.,1995).
Finally,Hall(Hall,1996),inhersurveymentionedearlier,askedrespondentsabout
corporatepolicies.Only11%oftherespondentsclaimedtoknowofacomprehensive
corporatepolicyforspreadsheetdevelopment.Ofthese,furthermore,onlyonein
threeanswered"Yes"whenaskediftheyknewwheretofindthepolicyinwritten
form!In90%ofthecaseswhererulesexisted,furthermore,enforcementwasleftto
thedeveloper.Interestingly,39%saidthattheyhadcompliedwiththeirorganizations
policyfourtimesthepercentagesayingthatsuchapolicyexisted.Perhapsthese
respondentsfeltthattheywerecomplyingwithunwrittenrules.
Overall,formalspreadsheetpoliciesseemtobefairlyrare,andtheirenforcement
seemstobecasual.Whilemorerespondentsseemtobelievethatinformaland
typicallyunwrittenguidelinesexist,theeffectivenessofsuchguidelinescanonlybe
lefttoconjecture.

Overconfidence
Whyisenduserdevelopmentinspreadsheetingsocasual?Theanswermaybethat
bothdevelopersandcorporationsareoverconfidentabouttheaccuracyof
spreadsheets,despitethelargeamountofinformationtothecontrary.Asnoted
earlier,whenBrownandGould(Brown&Gould,1987)gavethreespreadsheet
developmenttaskstoninehighlyexperiencedspreadsheetdevelopers,allmadeat
leastoneerror,and63%ofthespreadsheetsoverallhaderrors.Yetwhenasked
abouttheirconfidenceinthecorrectnessoftheirspreadsheets,theirmedianscore
was"veryconfident."DaviesandIkin(Davies&Ikin,1987),whofounderrorsinfour
ofthe19spreadsheetstheyaudited,alsofoundthatthedeveloperswereextremely
confidentintheaccuracyoftheirspreadsheets.Asnotedabove,Floyd,Wall,andMarr
(Floyd&Pyun,1987)foundthattheir72enduserswereconfidentinthecorrectness
oftheirspreadsheets,despitetheirlargesizes(averaging6,000cells).
Inlargerspreadsheets,thereshouldbeahigherprobabilityofanerrorthaninasmall
spreadsheet.YetwhenReithel,Nichols,andRobinson(Reithel,Nichols,&Robinson,
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

20/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

1996)hadsubjectsexaminespreadsheetsandthenstatetheirconfidenceinthe
spreadsheet,subjectsratedlargewellformattedspreadsheetsmuchhigherthanlarge
plainlyformattedspreadsheetsandalsohigherthansmallspreadsheetsregardlessof
formatting.Inapersonalcommunicationwiththeauthorin1997,Reithelnotedthat
forthelargefancyspreadsheet,72%ofthesubjectsgavethespreadsheetthehighest
confidencerange(80%100%),andonly10%chosethe1%60%range.Forthe
otherthreeconditions,aboutathirdoftherespondentschosethesetworanges.
Unfortunately,thereasonsforthisapparentlyinvertedpatternofconfidenceare
unclear.
Inexperiments,PankoandHalverson(1997)andPankoandSprague(1998)found
highconfidenceamongsubjectsdespitethefactthatthesesubjectsmademany
errors.Inbothstudies,furthermore,confidenceincorrectnesswasunrelatedtothe
actualnumberoferrors.Morerecently,PankoandHalverson(2001)hadsubjects
developaspreadsheetindividuallyorintriads(groupsofthree).Whenaskedto
estimatedtheprobabilitythattheirspreadsheetcontainedanerror,individualshada
meanof18%,whilemembersoftriadshadameanof13%.Theactualfigures
percentageofincorrectspreadsheets86%and27%forindividualsandtriads.
Whilesuchmassivelevelsofoverconfidenceseemunreasonable,overconfidenceis
perhapsthemostwellestablishedphenomenoninthebehavioralsciences.When
askedaboutmanythings,suchasdriving,alargemajorityofallpeoplerate
themselvesasaboveaverage(Brown,1990Gilovich,1991Koriat,Lichtenstein,&
Fischoff,1980).Alongstreamofexperimentshavefoundthatpeopleare
overconfidentinawidevarietyoftasksrangingfromTrivialPursuittypequestionsto
theperceptionofacousticalsignals(Adams&A.,1961Lichtenstein,Fischoff,&
Philips,1982Plous,1993Wagenaar&Keren,1986).Overconfidencehasalsobeen
seenindecisionsupportsystemsuse(Kasper,1996)anddatabasequeries
(Greenblatt&Waxman,1978).Evenexpertsinawidevarietyoffieldsareoften
appallinglyoverconfidentwithintheirdomainsofexpertise(Johnson,1988Shanteau
&Phelps,1977).Overall,inbothcontrivedandrealistictasks,thereseemstobea
pervasivehumantendencytowardoverconfidence.
Oneofthemostinterestingaspectsofthisoverconfidenceisthehardeasyeffect,in
whichoverconfidenceishighestwhenactualperformanceislowest(Clarke,1960
Dunning,Griffin,Milojkovic,&Ross,1990Lichtensteinetal.,1982Plous,1993
Wagenaar&Keren,1986).Inpractice,asdegreeofdifficultyincreases,confidence
falls,butnotproportionately.
Overconfidenceiscorrosivebecauseittendstoblindpeopletotheneedfortaking
stepstoreducerisks.Indriving,forinstance,thereisevidencethatpeopleactasif
theywerealmostblindtolowprobabilitybutlethaldangers(Howarth,1988
Swensen,1977).Infact,whenwetakeunwarranteddrivingrisks,ourriskybehavior
isreinforced,becausewegettooutdestinationsfaster,dolesswork,andsoforth
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

21/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

(Fuller,1990).Evenifwegetintotroubleandhavetotakeevasiveaction,wetendto
interpretthisasvalidationofourskillsratherthanfoolishness(Fuller,1990
Habberley,Shaddick,&Taylor,1986).Inspreadsheeting,inturn,ifwedonotdo
formaltestingandfollowothertediousdisciplines,webenefitbysavingtimeand
avoidingonerouswork.Whenwedocatcherrorsinformally,furthermore,webecome
convincedofourefficacyinerrorcorrection.
Moregenerically,whenwesearchforerrors,weseemtohavethreebasicprocesses
(Allwood,1984),twooftheseappeartobetriggeredbymismatchesbetweenthe
stateoftheworldandwhatwehadexpectedittobe.Thethird,the"standardcheck,"
involvescheckingforerrorsevenwhenwehavenodirectsuspicionthattheyexist.In
otherwords,thestandardcheckislikecodeinspection,althoughitmaybeforonly
partofthespreadsheet.Thestandardcheckisimportantbecausealthoughitisnot
motivatedexceptbyselfdiscipline,itappearstobetheonlywaytocatchcomplex
logicalerrors(Allwood,1984Rizzo,Bagnara,&Visciola,1986).
Howmuchstandarderrorcheckingshouldwedo?Obviously,itwouldbeinsaneto
questioneverythingwedoinlife(Gilovich,1991).Wemusthavesome"stopping
rules,"andthosestoppingruleswilldeterminewhatwefind(Rasmussen,1990).Ifwe
areoverconfident,ourstandarderrorcheckingislikelytostoptoosoon.
Overconfidencetendstobegreatestwhenrealworldsubjectsdonotgetdetailed
feedbackovermanycases(Shanteau&Phelps,1977Wagenaar&Keren,1986).In
experiments,(Arkes,Christensen,Lai,&Blumer,1987Lichtenstein&Fischoff,1980),
too,detailedfeedbackhasbeeneffective.Note,however,thatexperienceisnot
enough.Theremustbeverydetailedfeedbackoneachcase.Forinstance,Wagenaar
andKeren(Wagenaar&Keren,1986)foundthatexpertbridgeplayers,whoreceived
detailedfeedbackaftereachhandandanalyzedit,werewellcalibratedinconfidence.
Incontrast,professionalblackjackdealers,whoseebutdonotanalyzetheresultsof
eachdeal,werenobettercalibratedthanlaypeople.Inspreadsheeting,wehaveseen
thatdetailedcodeinspectionisrare,sofewdevelopersarelikelytobegettingthe
specificfeedbacktheyneedtoreducetheirnaturaloverconfidence.

ImplicationsforDevelopment
Althoughwestillhavefartoolittleknowledgeofspreadsheeterrorstocomeupwitha
definitivelistofwaystoreduceerrors,thesimilarityofspreadsheeterrorsto
programmingerrorssuggeststhat,ingeneral,wewillhavetobeginadopting(yet
adapting)manytraditionalprogrammingdisciplinestospreadsheeting.Ineffect,we
mustteachnewdogsoldtricks.See,forexample,thepaperbyRajalingham,
Chadwick,Knight,andEdwards.
Oneaspectnotedinpaststudies(Brown&Gould,1987Panko&Halverson,1997)is
thetendencyforspreadsheetdeveloperstobeginbuildingwithoutanadequate
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

22/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

preliminarydesign.Inlargespreadsheetsbuiltwithmanyiterations,thisisverylikely
toleadtoproblems.Benham,Delaney,andLuzi(Benham,Delaney,&Luzi,1993),
Ronen,Palley,andLucas(Ronen,Palley,&Lucas,1989),andIsakowitz,Schocken,
andLucas(Isakowitz,Schocken,&Lucas,1995)haveevensuggestedtheimposition
offairlyformalplanningtechniquesusingCASEliketools.Howeverinspreadsheet
development,buildingthespreadsheettypicallyresultsinincrementallygreater
understandingoftheproblem,soitmaybeunreasonabletoexpectfullpriordesign.
Wemanyneedsomethingmorelikeplansinwriting,whichgiveguidanceyetshould
bekept"cheapenough"tobethrownawaywhenobsolete(Hayes&Flower,1980).
Mostprescriptionsforspreadsheetdevelopmenthavefocusedonlatedesignand
developmentphase,includingmodularconstruction,havinganassumptionssection
forallinputs,andusingcellprotection.Thesearecertainlyimportant,buttheyarefar
frombeingenough.Ifpeopledoandfeelgoodaboutsuchthings,infact,theymay
becomeoverconfident.Inaddition,assumptionssections,whiledesirableinmany
ways,mayactuallycreateerrors.TheexperimentsofJanvrinandMorrison(Janvrin&
Morrison,19962000)andLerch(Lerch,1988)suggestthatpeoplemakemanyerrors
whentheypointtodistantcells,asisthecasewhenasassumptionsectionisused.
Hall(Hall,1996)listsmanyotherdevelopmentphasecontrols.
Inprogramming,anewformofdevelopmentiscalledcleanroomdevelopment(Linger,
1994).Inthisapproach,theprogramisproventobecorrectmathematicallybeforeit
iswritten.Inspreadsheeting,ourexperienceobservingdevelopersisthattheyoften
enteraformulawithoutthinkingitthroughbeforehand.Unlesstheresultslook
obviouslywrong,theydonottrytodetermineiftheformulaisincorrect.Thisallows
thesurvivalofmanysubtleyetimportanterrors,evenwhenthedeveloperreally
knowsapproximatelywhatthecorrectanswershouldbe.Inmanyways,the
developersseemtobeactinglikeThorndikes(Thorndike,1898)cats,whouponbeing
putinapuzzleboxsimplytriedthingsuntiltheygotout(foramoderndiscussion,see
Mayer(Mayer,1992)).Oneapproachtoreducingsuchbehavioristorequire
developerstowriteoutandcritiqueallnontrivialformulas,doingcalculationsby
hand,andthenreplicatingthesecalculationswiththeformulatypedintothe
spreadsheet.
Mostprescriptivepapersalsodiscusstheimportanceofdocumentation.Aswehave
alreadyseen,documentationisrareinspreadsheets.
Inprogramming,wehaveseenfromliterallythousandsofstudiesthatprogramswill
haveerrorsinabout5%oftheirlineswhenthedeveloperbelievesthatheorsheis
finished(Panko,2005a).Averyrigoroustestingstageafterthedevelopmentstageis
neededtoreduceerrorratesbyabout80%(Panko,2005a).Whetherthisisdoneby
datatesting,linebylinecodeinspection,orboth,testingisanoneroustaskandis
difficulttodoproperly.Incodeinspection,forinstance,weknowthattheinspection
mustbedonebyteamsratherthanindividualsandthattheremustbesharplimitsfor
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

23/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

modulesizeandforhowmanylinescanbeinspectedperhour(Fagan,1976).We
haveseenabovethatmostdevelopersignorethiscrucialphase.Yetunless
organizationsandindividualsarewillingtoimposetherequirementforcodeinspection
orcomprehensivedatatesting,thereseemslittleprospectforhavingcorrect
spreadsheetsinorganizations.Incomparison,theerrorreductiontechniquesnormally
discussedinprescriptivespreadsheetarticlesprobablyhavefarlowerimpact.
Whateverspecifictechniquesareused,onebroadpolicymustbetheshieldingof
spreadsheeterswhoerrfrompunishment.Inprogramming,ithaslongbeenknown
thatitiscriticaltoavoidblamingincodeinspectionandotherpublicprocesses.For
instance,Fagan(Fagan,1976)emphasizedthatcodeinspectionshouldneverbeused
forperformanceevaluation.AsBeizer(Beizer,1990)hasemphasized,aclimateof
blamingwillpreventdevelopersfromacknowledgingerrors.Edmondson(Edmondson,
1996)foundthatinnursing,too,apunitiveenvironmentwilldiscouragethereporting
oferrors.Quitesimply,althoughtheerrorratesseeninresearchstudiesareappalling,
theyarealsoinlinewiththenormalaccuracylimitsofhumaninformationprocessing.
Wecannotpunishpeoplefornormalhumanfailings.
Finally,wehavebeendiscussingdevelopersinthispaper,butDhebar(Dhebar,1993)
hasemphasizedtheneedforemployeesingeneraltobesmartconsumersof
spreadsheetresultsgeneratedbyothers.Inuniversities,introductoryclassesdeal
withbothstudentswhowillprobablybebothdevelopersandconsumersof
spreadsheets.Ourclasscontentshouldreflectbothroles.
Althoughmanyprescriptivetechniqueshavebeenputforth,onlycodeinspectionhas
beentestedexperimentallyandhasproventobebothsafeandeffective.Individual
codeinspectionhasbeenshowntofindabouthalfofallerrors,sothatgroupcode
inspectionisneeded.AsPanko(1999)hasshown,evengroupinspectionhelpsonly
somewhat,althoughithelpsespeciallyfordifficulttofinderrors.Otherprescriptive
methodshavebeenputforthbuthavenotbeentestedandsomustbeviewed
somewhatskeptically.Forinstance,inunpublishedresultsfoundbytheauthor,when
studentswhoputassumptionssectionsatthestartoftheirspreadsheetwere
comparedwithstudentswhodidnotuseassumptionsections,therewasno
statisticallysignificantdifferenceinerrorrates,andstudentswhodidnothave
assumptionssectionsactuallyhadnumericallylowererrorrates.

Conclusion
Allinall,theresearchdonetodateinspreadsheetdevelopmentpresentsavery
disturbingpicture.Everystudythathasattemptedtomeasureerrors,without
exception,hasfoundthematratesthatwouldbeunacceptableinanyorganization.
Theseerrorrates,furthermore,arecompletelyconsistentwitherrorratesfoundin
otherhumanactivities.Withsuchhighcellerrorrates,mostlargespreadsheetswill
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

24/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

havemultipleerrors,andevenrelativelysmall"scratchpad"spreadsheetswillhavea
significantprobabilityoferror.
Despitetheevidence,individualdevelopersandorganizationsappeartobeinastate
ofdenial.Theydonotregularlyimplementevenfairlysimplecontrolstoreduce
errors,muchlesssuchbitterpillsascomprehensivecodeinspection.Onecorporate
officerprobablysummarizedthesituationbysayingthatheagreedwiththeerrorrate
numbersbutfeltthatcomprehensivecodeinspectionissimplyimpractical.Inother
words,hewassayingthatthecompanyshouldcontinuetobasecriticaldecisionson
badnumbers.
Amajorimpedimenttoimplementingadequatedisciplines,ofcourse,isthatfew
spreadsheetdevelopershavespreadsheetingintheirjobdescriptionsatall,andvery
fewdospreadsheetdevelopmentastheirmaintask.Inaddition,becausespreadsheet
developmentissodispersed,theimplementationofpolicieshastobelefttoindividual
departmentmanagers.Whileorganizationsmightidentifycriticalspreadsheetsand
onlyimposeharddisciplinesonthem(Panko,1988),thiswouldstillmeanthatmany
corporatedecisionswouldcontinuetobemadeonthebasisofquestionableanalyses.

References
Adams,J.K.,&A.,A.P.(1961).RealismofConfidenceJudgements.Psychological
Review,68,3545.
Allwood,C.M.(1984).ErrorDetectionProcessesinStatisticalProblemSolving.
CognitiveScience,8(4),413437.
Arkes,H.R.,Christensen,C.,Lai,C.,&Blumer,C.(1987).TwoMethodsofReducing
Overconfidence.OrganizationalBehaviorandHumanDecisionProcesses,39,134144.
Baars,B.J.(1992).ANewIdeomotorTheoryofVoluntaryControl.InB.J.Baars
(Ed.),ExperimentalSlipsandHumanError(pp.93120).NewYork:Plenum.
Bagnara,S.,Stablum,F.,Rizzo,A.,Fontana,A.,&Ruo,M.(1987,).ErrorDetection
andCorrection:AStudyonHumanComputerInteractioninaHotStripMillPlanning
andProductionSystem.PreprintsoftheFirstEuropeanMeetingonCognitive
EngineeringApproachestoProcessControl,Marcoussis,France.
Basili,V.R.,&Selby,R.W.,Jr.(1986).FourApplicationsofaSoftwareDataCollection
andAnalysisMethodology.InJ.K.Skwirzynski(Ed.),SoftwareSystemDesign
Methdology(pp.333).Berlin:SpringerVerlag.
Beizer,B.(1990).SoftwareTestingTechniques.(2nded.).NewYork:VanNostrand.
Benham,H.,Delaney,M.,&Luzi,A.(1993).StructuredTechniquesforSuccessfulEnd
UserSpreadsheets.JournalofEndUserComputing,5(2),1825.
Brown,I.D.(1990).Drivers'MarginsofSafetyConsideredasaFocusforResearchon
Error.Ergonomics,33(10/11),13071314.
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

25/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Brown,P.S.,&Gould,J.D.(1987).AnExperimentalStudyofPeopleCreating
Spreadsheets.ACMTransactionsonOfficeInformationSystems,5(3),258272.
Butler,R.J.,IsthisSpreadsheetaTaxEvader?HowH.M.Customs&ExciseTaxTest
SpreadsheetApplications,Proceedingsofthe33rdHawaiiInternationalConference
onSystemSciences,Maui,Hawaii,January47,2000.
Butler,Raymond,Taxauditstudyin1992,personalcommunicationsbyelectronic
mail,AugustandSeptember,1996andinAugust,1997.
Cale,E.G.,Jr.(1994).QualityIssuesforEndUserDevelopedSoftware.Journalof
SystemsManagement,45(1),3639.
Clermont,M.,Hanin,C.&Mittermeier,R.(2000,July).Aspreadsheetauditingtool
evaluatedinanindustrialcontext.InProceedingsoftheEuSpRIG2000Symposium,
SpreadsheetRiskstheHiddenCorporateGamble.(pp.3546).Greenwich,England:
GreenwichUniversityPress.
Clarke,F.R.(1960).ConfidenceRatings,SecondChoiceResponses,andConfusion
MatricesinIntelligibilityTests.JournaloftheAcousticalSocietyofAmerica,32,3546.
Coopers&LybrandinLondon.Descriptionavailableat
http://www.planningobjects.com/jungle1.htm.Contactinformationisavailableatthat
webpage.
Cragg,P.G.,&King,M.(1993).SpreadsheetModellingAbuse:AnOpportunityforOR?
JournaloftheOperationalResearchSociety,44(8),743752.
Davies,N.,&Ikin,C.(1987).AuditingSpreadsheets.AustralianAccount,5456.
Davis,G.B.(January1982).Caution:UserDevelopedSystemsCanbeHazardousto
YourOrganization.ProceedingsoftheSeventeenthAnnualHawaiiInternational
ConferenceonSystemSciences,Honolulu,Hawaii.
Dhebar,A.(1993).ManagingtheQualityofQuantitativeAnalysis.SloanManagement
Review,34(2),6975.
Ditlea,S.(1987).SpreadsheetsCanbeHazardoustoYourHealth.Personal
Computing,6069.
Dunning,D.,Griffin,D.W.,Milojkovic,J.D.,&Ross,L.(1990).TheOverconfidence
EffectinSocialPrediction.JournalofPersonalityandSocialPsychology,4,568581.
Edmondson,A.C.(1996).LearningfromMistakesisEasierSaidthanDone:Group
andOrganizationalInfluencesontheDetectionandCorrectionofHumanError.Journal
ofAppliedBehavioralScience,32(1),528.
Fagan,M.E.(1976).DesignandCodeInspectionstoReduceErrorsinProgram
Development.IBMSystemsJournal,15(3),182211.
Ferdinand,A.E.(1993).Systems,Software,andQualityEngineering:ApplyingDefect
BehaviorTheorytoProgramming.NewYork:VanNostrandReinhold.

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

26/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Flower,L.A.,&Hayes,J.R.(1980).TheDynamicsofComposing:MakingPlansand
JugglingConstraints.InL.W.Gregg&E.R.Steinberg(Eds.),CognitiveProcessesin
Writing(pp.3150).Hillsdale,NJ:LawrenceErlbaumAssociates.
Floyd,B.D.,&Pyun,J.(1987,October).ErrorsinSpreadsheetUse(WorkingPaper
167).NewYork:CenterforResearchonInformationSystems,InformationSystems
Department,NewYorkUniversity.
Floyd,B.D.,Walls,J.,&Marr,K.(1995).ManagingSpreadsheetModelDevelopment.
JournalofSystemsManagement,46(1),3843,68.
Freeman,D.(1996).HowtoMakeSpreadsheetsErrorProof.JournalofAccountancy,
181(5),7577.
Fuller,R.(1990).LearningtoMakeErrors:EvidencefromaDrivingSimulation.
Ergonomics,33(10/11),12411250.
Galletta,D.F.,Abraham,D.,ElLouadi,M.,Lekse,W.,Pollailis,Y.A.,&Sampler,J.L.
(1993).AnEmpiricalStudyofSpreadsheetErrorPerformance.JournalofAccounting,
Management,andInformationTechnology,5(34),7995.
Galletta,D.F.,Hartzel,K.S.,Johnson,S.,andJoseph,J.L.(1997,Winter).
SpreadsheetPresentationandErrorDetection:AnExperimentalStudyJournalof
ManagementInformationSystems(13:2),pp.4563.
Galletta,D.F.,&Hufnagel,E.M.(1992).AModelofEndUserComputingPolicy:
Context,Process,ContentandCompliance.InformationandManagement,22(1),1
28.
Gilovich,T.(1991).HowWeKnowWhatIsn'tSo:TheFallibilityofHumanReasonin
EverydayLife.NewYork:FreePress.
Greenblatt,D.,&Waxman,J.(1978).AStudyofThreeDatabaseQueryLanguages.In
B.Schneiderman(Ed.),Databases:ImprovingUsabilityandResponsiveness(pp.77
97).NewYork:AcademicPress.
Habberley,J.S.,Shaddick,C.A.,&Taylor,D.H.(1986).ABehaviouralStudyofthe
CollisionAvoidanceTaskinBridgeWatchkeeping.Southhampton,England:Collegeof
MarineStudies.
Hall,M.J.J.(1996).ARiskandControlOrientedStudyofthePracticesof
SpreadsheetApplicationDevelopers.ProceedingsoftheTwentyNinthHawaii
InternationalConferenceonSystemSciences,Kihei,Maui.
Hayes,J.R.,&Flower,L.S.(1980).IdentifyingtheOrganizationofWritingProcesses.
InL.Gregg&E.Steinbert(Eds.),CognitiveProcessesinWriting(pp.330).Hillsdale,
NJ:LawrenceErlbaumAssociates.
Hendry,D.G.,&Green,T.R.G.(1994).Creating,Comprehending,andExplaining
Spreadsheets:ACognitiveInterpretationofWhatDiscretionaryUsersThinkofthe
SpreadsheetModel.InternationalJournalofHumanComputerStudies,40(6),1033
1065.
Hicks,L.,NYNEX,personalcommunicationwiththefirstauthorviaelectronicmail,
June21,1995.

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

27/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Howarth,C.I.(1988).TheRelationshipBetweenObjectiveRisk,SubjectiveRisk,and
Behaviour.Ergonomics,31,527535.
Isakowitz,T.,Schocken,S.,&Lucas,H.C.J.(1995).TowardaLogical/PhysicalTheory
ofSpreadsheetModeling.ACMTransactionsonInformationSystems,13(1),137.
Janvrin,D.,&Morrison,J.(2000a).FactorsInfluencingRisksandOutcomesinEnd
UserDevelopment.ProceedingsoftheTwentyNinthHawaiiInternationalConference
onSystemSciences,Kihei,Maui,Hawaii.
Janvrin,D.,&Morrison,J.(2000b).UsingaStructuredDesignApproachtoReduce
RisksinEndUserSpreadsheetDevelopment,Information&Management,(37),112.
Johnson,E.J.(1988).ExpertiseandDecisionUnderUncertainty:Performanceand
Process.InM.T.H.Chi,R.Glaser,&M.J.Farr(Eds.),TheNatureofExpertise(pp.
209228).Hillsdale,NJ:LawrenceErlbaumAssociates.
Johnson,P.,&Tjahjono,D.(1997,May).ExploringtheEffectivenessofFormal
TechnicalReviewFactorswithCSRS.PaperpresentedattheProceedingsofthe1997
InternationalConferenceonSoftwareEngineering,Boston,MA.
Kasper,G.M.(1996).ATheoryofDecisionSupportSystemDesignforUser
Calibration.InformationSystemsResearch,7(2),215232.
Koriat,A.,Lichtenstein,S.,&Fischoff,B.(1980).ReasonsforConfidence.Journalof
ExperimentalPsychology:HumanLearningandMemory,6,107117.
KPMGManagementConsulting,"SupportingtheDecisionMakerAGuidetotheValue
ofBusinessModeling,"pressrelease,July30,1998.
http://www.kpmg.co.uk/uk/services/manage/press/970605a.html.
Lawrence,R.J.andLee,J,"FinancialModellingofProjectFinancingTransactions,"
InstituteofActuariesofAustraliaFinancialServicesForum,August26272004.
Lerch,F.J.(1988).ComputerizedFinancialPlanning:DiscoveringCognitiveDifficulties
inKnowledgeBuilding.UnpublishedPh.D.Dissertation,UniversityofMichigan,Ann
Arbor,MI.
Lichtenstein,S.,&Fischoff,B.(1980).TrainingforCalibration.OrganizationalBehavior
andHumanPerformance,26,149171.
Lichtenstein,S.,Fischoff,B.,&Philips,L.D.(1982).CalibrationofProbabilities:The
StateoftheArtto1980.InD.Kahneman,P.Slovic,&A.Tversky(Eds.),Judgment
UnderUncertainty:HeuristicsandBiases(pp.306334).Cambridge,England:
CambridgeUniversityPress.
Linger,R.C.(1994).CleanroomProcessModel.IEEESoftwere,11(2),5058.
Lorge,I.,&Solomon,H.(1955).TwoModelsofGroupBehaviorintheSolutionof
EurekaTypeProblems.Psychometrika,20(2),139148.
Lukasik,Todd,CPS,privatecommunicationwiththeauthor,August10,1998.
Mayer,R.E.(1992).Thinking,ProblemSolving,Cognition.(2nd.ed.).NewYork:W.
H.Freeman.

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

28/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Myers,G.J.(1978).AControlledExperimentinProgramTestingandCode
Walkthroughs/Inspections.CommunicationsoftheACM,21(9,760768.
Nardi,B.A.(1993).ASmallMatterofProgramming:PerspectivesonEndUser
Computing.Cambridge,MA:MITPress.
Nardi,B.A.,&Miller,J.R.(1991).TwinklingLightsandNestedLoops:Distributed
ProblemSolvingandSpreadsheetDevelopment.InternationalJournalofManMachine
Studies,34(1),161168.
Olson,J.R.,&Nilsen,E.(19871988).AnalysisoftheCognitionInvolvedin
SpreadsheetInteraction.HumanComputerInteraction,3(4),309349.
Panko,R.R.(2008a).HumanErrorWebsite
(http://www.cba.hawaii.edu/panko/papers/ss/humanerr.htm).Honolulu,HI:
UniversityofHawaii.
Panko,R.R.(2008b).SpreadsheetResearch(SSR)Website
(http://www.cba.hawaii.edu/panko/ssr/).Honolulu,Hawaii:UniversityofHawaii.
Panko,R.R.(1988).EndUserComputing:Management,Applications,and
Technology,NewYork:Wiley.
Panko,R.R.,"TwoCorpusesofSpreadsheetError,"ProceedingsoftheThirtyThird
HawaiiInternationalConferenceonSystemSciences,Maui,Hawaii,January2000.
(.pdfformat)
Panko,RaymondR.,(1999,Fall)"ApplyingCodeInspectiontoSpreadsheetTesting,
JournalofManagementInformationSystems,(16:2,159176.
Panko,R.R.,andHalverson,R.P.,Jr.(2001,January)"AnExperimentinCollaborative
SpreadsheetDevelopment,JournaloftheAssociationforInformationSystems.
Panko,R.R.,andHalverson,R.P.,Jr.(1997,Spring)."AreTwoHeadsBetterthan
One?(AtReducingErrorsinSpreadsheetModeling?OfficeSystemsResearchJournal
(15:1),pp.2132.
Panko,R.R.,&Halverson,R.P.,Jr.(1996,January).SpreadsheetsonTrial:A
FrameworkforResearchonSpreadsheetRisks.ProceedingsoftheTwentyNinth
HawaiiInternationalConferenceonSystemSciences,Maui,Hawaii.
Panko,R.R.,andSprague,R.H.J.(1999).HittingtheWall:ErrorsinDevelopingand
CodeInspectingaSimpleSpreadsheetModel.DecisionSupportSystems,(22),pp.
337353.
Plous,S.(1993).ThePsychologyofJudgmentandDecsionMaking.Philadelphia:
TempleUniversityPress.
Powell,StephenG.Lawson,BarryandBaker,KennethR.ImpactofErrorsin
OperationalSpreadsheets,EusprigConferenceProceedings2007,Universityof
London,Greenwich2007,5767.(2007a)
Powell,StephenG.Baker,KennethR.andLawson,Barry.(November15,2007).Errorsin
OperationalSpreadsheets,Draft2.0November15,2007.
http://mba.tuck.dartmouth.edu/spreadsheet/product_pubs_files/ErrorsinSpreadsheets.doc.

(2007b)

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

29/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Putnam,L.H.,&Myers,W.(1992).MeasuresforExcellence:ReliableSoftwareon
Time,onBudget.EnglewoodCliffs,NJ:Yourdon.
Rabbit,P.M.A.,&Vyas,S.M.(1970).AnElementaryPreliminaryTaxonomyforSome
ErrorsinLaboratoryChoiceRTTasks.InA.F.Sanbders(Ed.),Attentionand
PerformanceIII(pp.5676).Amsterdam:NorthHolland.
Rajalingham,KamalasenChadwick,DavidKnight,BrianandEdwards,Dilwyn,
(2000,January)"QualityControlinSpreadsheets:ASoftwareEngineeringBased
ApproachtoSpreadsheetDevelopment,"ProceedingsoftheThirtyThirdHawaii
InternationalConferenceonSystemSciences,Maui,Hawaii.(.pdfformat)
Rasmussen,J.(1990).TheRoleofPerceptioninOrganizingBehavior.Ergonomics,
33(10/11),11851199.
Reason,J.T.(1990).HumanError.Cambridge,UK:CambridgeUniversityPress.
Reithel,B.J.,Nichols,D.L.,&Robinson,R.K.(1996).AnExperimentalInvestigation
oftheEffectsofSize,Format,andErrorsonSpreadsheetReliabilityPerception.
JournalofComputerInformationSystems,5464.
Rizzo,A.,Bagnara,S.,&Visciola,M.(1986).HumanErrorDetectionProcesses.InG.
Mancini,D.Woods,&E.Hollnagel(Eds.),CognitiveEngineeringinDynamicWorlds.
Ispra,Italy:CECJointResearchCentre.
Ronen,B.,Palley,M.A.,&Lucas,H.(1989).SpreadsheetAnalysisandDesign.
CommunicationsoftheACM,32(1),8492.
Schultheis,R.,&Sumner,M.(1994).TheRelationshipofApplicationRisksto
ApplicationControls:AStudyofMicrocomputerBasedSpreadsheetApplications.
JournalofEndUserComputing,6(2),1118.
Shanteau,J.,&Phelps,R.H.(1977).JudgmentandSwine:ApproachesandIssuesin
AppliedJudgementAnalysis.InM.F.Kaplan&S.Schwartz(Eds.),HumanJudgment
andDecisionProcessesinAppliedSetting.NewYork:AcademicPress.
Speier,C.,&Brown,C.V.(1996,January).PerceivedRisksandManagementActions:
DifferencesinEndUserApplicationDevelopmentAcrossFunctionalGroups.
ProceedingsoftheTwentyNinthHawaiiInternationalConferenceonSystemScience,
Maui,Hawaii.
Swensen,O.(1977).RisksofRoadTransportationfromaPsychologicalPerspective:A
PilotStudy(ProjectRiskGenerationandRiskAssessmentinaSocialPerspective,
Report377):CommitteeforFutureOrientedResearch,Stockholm,Sweden.
Teo,T.S.H.,&Tan,M.(1997,).QuantitativeandQualitativeErrorsinSpreadsheet
Development.ProceedingsoftheThirtiethHawaiiInternationalConferenceonSystem
Sciences,Maui,Hawaii.
Thorndike,E.L.(1898).AnimalIntelligence:AnExperimentalStudyoftheAssociative
ProcessesofAnimals.PsychologicalMonographs,2(8).
Wagenaar,W.A.,&Keren,G.B.(1986).DoestheExpertKnow?TheReliabilityof
PredictionsandConfidenceRatingsofExperts.InE.Hollnagel,G.Manici,&D.D.
Woods(Eds.),IntelligentDecisionSupportinProcessEnvironments(pp.87103).
Berlin:SpringerVerlag.
http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

30/31

12/28/2016

WhatWeKnowAboutSpreadsheetErrors

Woods,D.D.(1984).SomeResultsonOperatorPerformanceinEmergencyEvents.
InstituteofChemicalEngineersSymposiumSeries,90,2131.

http://panko.shidler.hawaii.edu/My%20Publications/Whatknow.htm

31/31