Anal

ysi
sanddesi
gnofAl
gor
it
hms

I
ntr
oduct
ion
Thetopi csofthedesignandanal y
sisofalgori
thm ar
ev er
yi mpor
tant,asprogrammi ngand
progr
am dev el
opmentcan become compl ex and complicat
ed tasks.Such a scenario,
consideringl
argepr
ojects,
especial
ly,
requi
recoordi
natedeffort
sofmanypeopl e,stor
ingand
accessinglar
gequantiti
esofdataeffi
cient
ly,
andsolvi
ngcompl excomputati
onal pr
oblems.

I
ntui
ti
v el
y,qui
teanumberofsol ut
ionscanbedesi gnedtosolveapart
icul
arpr
oblem.Theset
ofsoluti
ons,usual
lyr
egardedast hesetoffeasiblesolut
ions,pr
ovi
dethebasisfrom which
theoptimalsol
uti
on(i
fitexi
sts)canemergethrought heprocessofst
epwiser
efinement.

- I
nalgor
it
hms’designandanaly
sis,theprocessofref
inementconsi
derstheobjecti
ve
and/
orquanti
tat
ivepr
opert
iesoft healgori
thmsintermsofef f
ici
ency,cor
rectness,
andsometi
messemanticanal
ysisofthetargetal
gor
it
hm.

Def
ini
ti
on
o Anal
gor
it
hm i
sanywel
l
-def
inedcomput
ati
onalpr
ocedur
ethatt
akessomev
alue,or
setofv
alues,asi
nputandpr
oducessomev
alue,orsetofv
alues,asout
put
.An
al
gor
it
hm i
sthusasequenceofcomput
ati
onalst
epst
hatt
ransf
ormt
hei
nputi
ntot
he
out
put
.

o Wecanal
sov
iew anal
gor
it
hm asat
oolf
orsol
vi
ngawel
l
-speci
fedcomput
i ati
onal
pr
obl
em.The st
atementoft
he pr
obl
em speci
fi
es i
n gener
alt
erms t
he desi
red
i
nput
/out
putr
elat
ionshi
p.Theal
gor
it
hm descr
ibesaspeci
fi
ccomput
ati
onalpr
ocedur
e
f
orachi
evi
ngt
hati
nput
/out
putr
elat
ionshi
p

o Anal
gor
it
hm i
sasequenceofunambi
guousi
nst
ruct
ionsf
orsol
vi
ngapr
obl
em,
i.
e.,
for
obt
aini
ngar
equi
redout
putf
oranyl
egi
ti
mat
einputi
naf
ini
teamountoft
ime.

I
ngener
al,ani
nst
anceofapr
obl
em consi
stsoft
hei
nput(
sat
isf
yingwhat
everconst
rai
nts
ar
eimposedi
nthepr
obl
em st
atement
)neededt
ocomput
easol
uti
ont
othepr
obl
em.

Becausemanypr
ogr
amsusesor
ti
ngf
orexampl
easani
nter
medi
atest
ep,sor
ti
ngi
sa
f
undament
aloper
ati
oni
ncomput
ersci
ence.Ther
efor
e,whi
chal
gor
it
hm i
sbestf
oragi
ven
appl
i
cat
iondependson—amongot
herf
act
ors—t
henumberofi
temst
obesor
ted,t
heext
ent
t
owhi
cht
hei
temsar
eal
readysomewhatsor
ted,
possi
bler
est
ri
cti
onsont
hei
tem v
alues,
the
ar
chi
tect
ureoft
hecomput
er,
andt
heki
ndofst
oragedev
icest
obeused:
mai
nmemor
y,di
sks,
orev
ent
apes.
Anal
gor
it
hm i
ssai
dtobecor
recti
f,f
orev
eryi
nputi
nst
ance,i
thal
tswi
tht
hecor
rectout
put
.
Wesayt
hatacor
rectal
gor
ihm sol
t vest
hegi
vencomput
ati
onal
probl
em.

Not
e:acor
rectal
gor
it
hm i
snotonet
hatwor
ksmostoft
het
ime,
butonet
hatwor
kscor
rect
ly
oral
f ll
egi
ti
mat
einput
s.

Ani
ncor
rectal
gor
it
hm mi
ghtnothal
tatal
lonsomei
nputi
nst
ances,ori
tmi
ghthal
t
wi
thani
ncor
rectanswer
.
Cont
rar
ytowhaty
oumi
ghtexpect
,incor
rectal
gor
it
hmscansomet
imesbeusef
ul,i
f
wecancont
rolt
hei
rer
rorr
ate,howev
er,weshal
lbeconcer
nedher
eonl
ywi
thcor
rect
al
gor
it
hms.

Anal
gor
it
hm canbespeci
fi
edi
nEngl
i
sh,asacomput
erpr
ogr
am,orev
enasahar
dwar
e
desi
gn.Theonl
yrequi
rementi
sthatt
hespeci
fi
cat
ionmustpr
ovi
deapr
eci
sedescr
ipt
ionof
t
hecomput
ati
onal
procedur
etobef
oll
owed.

Whatar
ethev
ari
ousst
epst
opl
anAl
gor
it
hm

Foll
owi ngst epsmustbef ol l
owedt opl ananyal gor i
thm:
 
(1)Dev iceAl gorithm:Cr eat i
nganal gor it
hm isanar tinwhi chmaynev erbef ull
yaut omat ed.
Whenwegett hepr oblem, weshoul df i
rstanalyset hegi venpr oblem clearlyandt henwr i
te
downsomest epsont hepaper .
 
(2)ValidateAl gorithm:Onceanal gor i
thm isdev i
sed, i
tisnecessar ytoshowt hati tcomput es
thecor rectanswerf orall possi blelegal i
nputs.Thi spr ocessi sknownasal gorit
hm v alidation.
Theal gor i
thm neednotasy etbeexpr essedasapr ogram.I tissuf f
ici
entt ost ateiti nany
preci
seway .Thepur poseofv ali
dationi st oassur eust hatthisal gori
thm wi l
lwor kcor rectl
y
i
ndependent lyoft hei ssuesconcer ningt hepr ogr ammi ngl anguagei twi llev entuallybe
writ
teni n.Oncet hev alidityoft hemet hodhasbeenshown,apr ogr am canbewr itt
enanda
secondphasebegi ns.Thi sphasei sr ef err
edt oaspr ogr am pr ovi
ngorpr ogram v erif
ication.
 
(3)Anal yseAl gor it
hm:Asanal gori
thm i sexecut ed, itusest hecomput erscent r
alpr ocessi ng
unitt o perfor m oper ati
onsand i tsmemor y( bot hi mmedi ateand auxi li
ary)t o hol dt he
program anddat a.
Anal ysisofal gor i
t hm orper for manceanal ysisr eferst othet askofdet ermi ninghow
muchcomput ingt i
meandst orageanal gori
thm r equires.
Ani mpor tantr esul toft hisstudyi sthati tallowsy out omakequant i
tati
vej udgment s
aboutt hev alueofoneal gor i
thm ov eranot her.
Anot herr esulti st hati tallowsy out opr edictwhet hert hesof twar ewi llmeetany
effi
ciencyconst raintst hatexi st.Analy siscanbemadebyt akingint oconsideration.
 
(4)TestAPr ogram:Test ingapr ogr am consi stsof2phases:debuggi ngandper for mance
management
.
Debuggi
ngi
sthepr
ocessofexecut
ingpr
ogr
amsonsampl
edat
aset
stodet
ermi
ne
whet
herr
esul
tsar
eincor
recti
fsocor
rect
sthem.
 Per
for
mancemanagementi
sthepr
ocessofexecut
ingacor
rectpr
ogr
am ondat
aset
s
andmeasur
ingt
het
imeandspacei
ttakest
ocomput
ether
esul
ts.
Theset
imi
ngf
igur
esar
eusef
uli
nthatt
heymayconf
ir
m apr
evi
ousl
ydoneanal
ysi
s
andpoi
ntoutl
ogi
cal
placest
oper
for
m usef
ulopt
imi
zat
ion.

Fundament
alsofAl
gor
it
hmi
cPr
obl
em Sol
vi
ng

Under
standi
ngt
hePr
obl
em
Readt
hepr
obl
em’
sdescr
ipt
ioncar
eful
l
yandaskquest
ionsi
fyouhav
eanydoubt
saboutt
he
pr
obl
em,doaf
ew smal
lexampl
esbyhand,t
hinkaboutspeci
alcases,andaskquest
ions
agai
nifneeded.Ther
ear
eaf
ewt
ypesofpr
obl
emst
hatar
isei
ncomput
ingappl
i
cat
ionsqui
te
of
ten.I
fthepr
obl
em i
nquest
ioni
soneoft
hem,
youmi
ghtbeabl
etouseaknownal
gor
it
hm
f
orsol
vi
ngi
t.
Ofcour
se,i
thel
pst
ounder
standhow suchanal
gor
it
hm wor
ksandt
oknow i
tsst
rengt
hs
and weaknesses,especi
all
yify
ou hav
eto chooseamong sev
eralav
ail
abl
eal
gor
it
hms
Remembert
hatacor
rectal
gor
it
hm i
snotonet
hatwor
ksmostoft
het
ime,butonet
hat
wor
kscor
rect
l oral
yf ll
egi
ti
mat
einput
s.
Donotski
mpont
hisf
ir
stst
epoft
heal
gor
it
hmi
cpr
obl
em-
sol
vi
ngpr
ocess;
other
wise,
youwi
l
l
r
unt
her
iskofunnecessar
yrewor
k.

Ascer
tai
ningt
heCapabi
li
ti
esoft
heComput
ati
onalDev
ice

Once y
ou compl
etel
y under
stand a pr
obl
em,y
ou need t
o ascer
tai
nthe capabi
l
iti
es oft
he
comput
ati
onaldev
icet
heal
gor
it
hm i
sint
endedf
or.Thev
astmaj
ori
tyofal
gor
it
hmsi
nuset
odayar
e
st
il
ldest
inedt
obepr
ogr
ammedf
oracomput
ercl
osel
yresembl
i
ngt
hev
onNeumannmachi
ne.The
essenceoft
hisar
chi
tect
urei
scapt
uredbyt
heso- edr
cal
l andom-
accessmachi
ne(
RAM)
.
I
tscent
ralassumpt
ioni
sthati
nst
ruct
ionsar
eexecut
edoneaf
teranot
her
,oneoper
ati
onatat
ime.
Accor
dingl
y,al
gor
it
hmsdesi
gnedt
obeexecut
edonsuchmachi
nesar edsequent
ecal
l ialal
gor
it
hms.
Thecent
ralassumpt
ionoft
heRAM modeldoesnothol
dforsomenewercomput
erst
hatcanexecut
e
oper
ati
onsconcur
rent
ly,i
.
e.,i
npar
all
el.Al
gor
it
hmst
hatt
akeadv
ant
ageoft
hiscapabi
l
ityar
ecal
l
ed
par
all
elal
gor
it
hms.
St
il
l,st
udy
ingt
hecl
assi
ctechni
quesf
ordesi
gnandanal
ysi
sofal
gor
it
hmsundert
heRAM model
r
emai
nst
hecor
ner
stoneofal
gor
it
hmi
csf
ort
hef
oreseeabl
efut
ure.Consequent
ly,
inmanysi
tuat
ions
y
ouneednotwor
ryaboutacomput
erbei
ngt
oosl
ow f
ort
het
ask.Ther
ear
eimpor
tantpr
obl
ems,
howev
er,
thatar
ever
ycompl
exbyt
hei
rnat
ure,
orhav
etopr
ocesshugev
olumesofdat
a,ordealwi
th
appl
i
cat
ionswher
ethet
imei
scr
it
ical
.Insuchsi
tuat
ions,
iti
simper
ati
vet
obeawar
eoft
hespeedand
memor
yav
ail
abl
eonapar
ti
cul
arcomput
ersy
stem.
Choosi
ngbet
weenExactandAppr
oxi
mat
ePr
obl
em Sol
vi
ng
The nextpr
inci
paldeci
si
on i
sto choose bet
ween sol
vi
ng t
he pr
obl
em exact
ly orsol
vi
ng i
t
appr
oxi
mat
ely
.
I
nthef
ormercase,
anal
gor
it
hm i edanexactal
scal
l gor
it
hm;
i
nthel
att
ercase,
anal
gor
it
hm i edanappr
scal
l oxi
mat
ional
gor
it
hm.
Whywoul
doneoptf
oranappr
oxi
mat
ional
gor
it
hm?
Fi
rst
,ther
ear
eimpor
tantpr
obl
emst
hatsi
mpl
ycannotbesol
vedexact
lyf
ormostoft
hei
rinst
ances;
exampl
es i
ncl
ude ext
ract
ing squar
eroot
s,sol
vi
ng nonl
i
nearequat
ions,and ev
aluat
ing def
ini
te
i
ntegr
als.
Second,av
ail
abl
eal
gor
it
hmsf
orsol
vi
ngapr
obl
em exact
lycanbeunaccept
abl
ysl
owbecauseoft
he
pr
obl
em’
sint
ri
nsi
ccompl
exi
ty.Thi
shappens,i
npar
ti
cul
ar,f
ormanypr
obl
emsi
nvol
vi
ngav
eryl
arge
numberofchoi
ces;
Thi
rd,anappr
oxi
mat
ional
gor
it
hm canbeapar
tofamor
esophi
sti
cat
edal
gor
it
hm t
hatsol
vesa
pr
obl
em exact
ly

Al
gor
it
hm Desi
gnTechni
ques
Now,wi
thal
lthecomponent
soft
heal
gor
it
hmi
cpr
obl
em sol
vi
ngi
npl
ace,how doy
oudesi
gnan
al
gor
it
hm t
osol
veagi
venpr
obl
em?Thi
sist
hemai
nconcer
nwhi
cht
hisl
ect
ureseekst
oanswerby
t
eachi
ngy
ousev
eral
gener
aldesi
gnt
echni
ques.
Whati
sanal
gor
it
hm desi
gnt
echni
que?
An al
gor
it
hm desi
gn t
echni
que (
or“
str
ategy
”or“
par
adi
gm”
)isa gener
alappr
oach t
o sol
vi
ng
pr
obl
emsal
gor
it
hmi
cal
l
ythati
sappl
i
cabl
etoav
ari
etyofpr
obl
emsf
rom di
ff
erentar
easofcomput
ing.
Checkt
het
abl
eofcont
ent
sofmostt
extont
hissubj
ect
,andy
ouwi
l
lseet
hatamaj
ori
tyoft
hei
r
chapt
ersar
edev
otedt
oindi
vi
dualdesi
gnt
echni
ques.Theydi
sti
l
laf
ewkeyi
deast
hathav
epr
ovent
o
beusef
uli
ndesi
gni
ngal
gor
it
hms.
Lear
ningt
heset
echni
quesi
sofut
mosti
mpor
tancef
ort
hef
oll
owi
ngr
easons.
Fi
rst
,theypr
ovi
degui
dancef
ordesi
gni
ngal
gor
it
hmsf
ornewpr
obl
ems,
i.
e.,
probl
emsf
orwhi
cht
her
e
i
snoknownsat
isf
act
oryal
gor
it
hm.Ther
efor
e,t
ouset
hel
anguageofaf
amouspr
over
b—l
ear
ning
sucht
echni
quesi
saki
ntol
ear
ningt
ofi
shasopposedt
obei
nggi
venaf
ishcaughtbysomebodyel
se.
I
tisnott
rue,
ofcour
se,
thateachoft
hesegener
alt
echni
queswi
l
lbenecessar
il
yappl
i
cabl
eto
ev
erypr
obl
em y
oumayencount
er.Butt
akent
oget
her
,theydoconst
it
uteapower
fulcol
l
ect
ionof
t
ool
sthaty
ouwi
l
lfi
ndqui
tehandyi
nyourst
udi
esandwor
k.
Second,al
gor
it
hms ar
ethe cor
ner
stone ofcomput
ersci
ence.Ev
ery sci
ence i
sint
erest
ed i
n
cl
assi
fyi
ngi
tspr
inci
palsubj
ect
,andcomput
ersci
encei
snoexcept
ion.Al
gor
it
hm desi
gnt
echni
ques
makei
tpossi
blet
ocl
assi
fyal
gor
it
hmsaccor
dingt
oanunder
lyi
ngdesi
gni
dea;t
her
efor
e,t
heycan
ser
veasanat
ural
wayt
obot
hcat
egor
izeandst
udyal
gor
it
hms.

Desi
gni
nganAl
gor
it
hm andDat
aSt
ruct
ures
Ofcour
se,oneshoul
dpaycl
oseat
tent
iont
ochoosi
ngdat
ast
ruct
uresappr
opr
iat
efort
heoper
ati
ons
per
for
med byt
he al
gor
it
hm.Al
so not
ethatsome oft
he al
gor
it
hm desi
gn t
echni
ques depend
i
nti
mat
elyonst
ruct
uri
ngorr
est
ruct
uri
ngdat
aspeci
fyi
ngapr
obl
em’
sinst
ance.
Manyy
ear
sago,ani
nfl
uent
ialt
ext
bookpr
ocl
aimedt
hef
undament
ali
mpor
tanceofbot
hal
gor
it
hms
anddat
ast
ruct
uresf
orcomput
erpr
ogr
ammi
ngbyi
tsv
eryt
it
le:
Al
gor
it
hms+Dat
aSt
ruct
ures=Pr
ogr
ams[
Wir
76]
.
I
nthenewwor
ldofobj
ect
-or
ient
edpr
ogr
ammi
ng,dat
ast
ruct
uresr
emai
ncr
uci
all
yimpor
tantf
orbot
h
desi
gnandanal
ysi
sofal
gor
it
hms.

Met
hodsofSpeci
fyi
nganAl
gor
it
hm
Oncey
ouhav
edesi
gnedanal
gor
it
hm,
youneedt
ospeci
fyi
tinsomef
ashi
on.Togi
vey
ouanexampl
e,
al
gor
it
hm i
sdescr
ibedi
nwor
ds(
inaf
reeandal
soast
ep-
by-
stepf
orm)andi
npseudocode.Thesear
e
t
het
woopt
ionst
hatar
emostwi
del
yusednowaday
sforspeci
fyi
ngal
gor
it
hms.
Usi
nganat
urall
anguagehasanobv
iousappeal
;howev
er,t
hei
nher
entambi
gui
tyofanynat
ural
l
anguagemakesasucci
nctandcl
eardescr
ipt
ionofal
gor
it
hmssur
pri
singl
ydi
ff
icul
t.Nev
ert
hel
ess,
bei
ngabl
etodot
hisi
sani
mpor
tantski
l
lthaty
oushoul
dst
ri
vet
odev
elopi
nthepr
ocessofl
ear
ning
al
gor
it
hms.
Pseudocode i
s a mi
xtur
e ofa nat
urall
anguage and pr
ogr
ammi
ng l
anguage l
i
ke const
ruct
s.
Pseudocodei
susual
l
ymor
epr
eci
set
hannat
urall
anguage,andi
tsusageof
teny
iel
dsmor
esucci
nct
al
gor
it
hm descr
ipt
ions.Sur
pri
singl
y,comput
ersci
ent
ist
shav
enev
eragr
eedonasi
ngl
efor
m of
pseudocode,
leav
ingt
ext
bookaut
hor
swi
thaneedt
odesi
gnt
hei
rown“
dial
ect
s.”
For
tunat
ely
,thesedi
alect
sar
esocl
oset
oeachot
hert
hatany
onef
ami
l
iarwi
thamoder
n
pr
ogr
ammi
ngl
anguageshoul
dbeabl
etounder
standt
hem al
l
.Thest
ateoft
hear
tofcomput
inghas
noty
etr
eached a poi
ntwher
e an al
gor
it
hm’
s descr
ipt
ion—be i
tin a nat
urall
anguage or
pseudocode—canbef
edi
ntoanel
ect
roni
ccomput
erdi
rect
ly.I
nst
ead,
itneedst
obeconv
ert
edi
ntoa
comput
erpr
ogr
am wr
it
teni
napar
ti
cul
arcomput
erl
anguage.Wecanl
ookatsuchapr
ogr
am asy
et
anot
herwayofspeci
fyi
ngt
heal
gor
it
hm,al
thoughi
tispr
efer
abl
etoconsi
deri
tast
heal
gor
it
hm’
s
i
mpl
ement
ati
on.

Pr
ovi
nganAl
gor
it
hm’
sCor
rect
ness

Forsomeal
gor
it
hms,apr
oofofcor
rect
nessi
squi
teeasy
;forot
her
s,i
tcanbequi
tecompl
ex.A
commont
echni
quef
orpr
ovi
ngcor
rect
nessi
stousemat
hemat
icali
nduct
ionbecauseanal
gor
it
hm’
s
i
ter
ati
onspr
ovi
deanat
ural
sequenceofst
epsneededf
orsuchpr
oof
s.
I
tmi
ghtbewor
thment
ioni
ngt
hatal
thought
raci
ngt
heal
gor
it
hm’
sper
for
mancef
oraf
ew speci
fi
c
i
nput
scanbeav
erywor
thwhi
l
eact
ivi
ty,
itcannotpr
ovet
heal
gor
it
hm’
scor
rect
nessconcl
usi
vel
y.But
i
nor
dert
oshowt
hatanal
gor
it
hm i
sincor
rect
,youneedj
ustonei
nst
anceofi
tsi
nputf
orwhi
cht
he
al
gor
it
hm f
ail
s.
Thenot
ionofcor
rect
nessf
orappr
oxi
mat
ional
gor
it
hmsi
slessst
rai
ght
for
war
dthani
tisf
orexact
al
gor
it
hms.Foranappr
oxi
mat
ional
gor
it
hm,weusual
l
ywoul
dli
ket
obeabl
etoshow t
hatt
heer
ror
pr
oducedbyt
heal
gor
it
hm doesnotexceedapr
edef
inedl
i
mit
.

Anal
yzi
nganAl
gor
it
hm
Weusual
l
ywantoural
gor
it
hmst
opossesssev
eralqual
i
ties.Af
tercor
rect
ness,byf
art
hemost
i
mpor
t sef
anti fi
ciency
.Inf
act
,ther
ear
etwoki
ndsofal
gor
it
hm ef
fi
ci :t
ency i
meef
fi
ciency
,indi
cat
ing
howf
astt
heal
gor
it
hm r andspaceef
uns, fi
ciency
,indi
cat
inghowmuchext
ramemor
yituses.
Anot
herdesi
rabl
echar
act
eri
sti
cofanal
gor
it ssi
hm i mpl
ici
ty.Unl
i
keef
fi
ciency
,whi
chcanbepr
eci
sel
y
def
inedandi
nvest
igat
edwi
thmat
hemat
ical
rigor
,si
mpl
i
cit
y,l
i
kebeaut
y,i
stoaconsi
der
abl
edegr
eei
n
t
heey
eoft
hebehol
der
.Why
?Becausesi
mpl
eral
gor
it
hmsar
eeasi
ert
ounder
standandeasi
ert
o
pr
ogr
am;consequent
ly,t
he r
esul
ti
ng pr
ogr
ams usual
l
ycont
ainf
ewerbugs.Ther
eis al
so t
he
undeni
abl
eaest
het
icappealofsi
mpl
i
cit
y.Somet
imessi
mpl
eral
gor
it
hmsar
eal
somor
eef
fi
cientt
han
mor
e compl
i
cat
ed al
ter
nat
ives.Unf
ort
unat
ely
,iti
s notal
way
str
ue,i
n whi
ch case a j
udi
cious
compr
omi
seneedst
obemade.
Yetanot
herdesi
rabl
echar
act
eri
sti
cofanal
gor
it sgener
hm i ali
ty.Ther
ear
e,i
nfact
,twoi
ssuesher
e:
gener
ali
tyoft
hepr
obl
em t
heal
gor
it
hm sol
vesandt
hesetofi
nput
sitaccept
s.
Ont
hef
ir
sti
ssue,
not
ethati
tissomet
imeseasi
ert
odesi
gnanal
gor
it
hm f
orapr
obl
em posed
i
nmor
egener
alt
erms.Ast
othesetofi
nput
s,y
ourmai
nconcer
nshoul
dbedesi
gni
nganal
gor
it
hm
t
hatcanhandl
easetofi
nput
sthati
snat
ural
fort
hepr
obl
em athand.
Forex
ampl
e,excl
udi
ngi
nteger
sequalt
o1aspossi
blei
nput
sforagr
eat
estcommondi
vi
sor
al
gor
it
hm woul
dbequi
teunnat
ural
.Ont
heot
herhand,
alt
hought
hest
andar
dfor
mul
afort
her
oot
sof
aquadr
ati
cequat
ionhol
dsf
orcompl
excoef
fi
cient
s,wewoul
dnor
mal
l
ynoti
mpl
ementi
tont
hisl
evel
ofgener
ali
tyunl
esst
hiscapabi
l
ityi
sexpl
i
cit
lyr
equi
red.
I
fyouar
enotsat
isf
iedwi
tht
heal
gor
it
hm’
sef
fi
ciency
,si
mpl
i
cit
y,orgener
ali
ty,
youmustr
etur
n
t
othedr
awi
ngboar
dandr
edesi
gnt
heal
gor
it
hm.I
nfact
,ev
eni
fyourev
aluat
ioni
sposi
ti
ve,i
tisst
il
l
wor
thsear
chi
ngf
orot
heral
gor
it
hmi
csol
uti
ons.
Gener
all
y,y
oushoul
dnotexpectt
ogett
hebestal
gor
it
hm ont
hef
ir
stt
ry.Att
hev
eryl
east
,youshoul
d
t
ryt
ofi
ne-
tunet
heal
gor
it
hm y
oual
readyhav
e.
Youwi
l
ldowel
li
fyoukeepi
nmi
ndt
hef
oll
owi
ngobser
vat
ionofAnt
oinedeSai
nt-
Exup’
ery
,theFr
ench
wr
it
er,
pil
ot,
andai
rcr
aftdesi :“
gner Adesi
gnerknowshehasar
ri
vedatper
fect
ionnotwhent
her
eisno
l
ongerany
thi
ngt
oadd,
butwhent
her
eisnol
ongerany
thi
ngt
otakeaway
.”1
Codi
nganAl
gor
it
hm
Mostal
gor
it
hmsar
edest
inedt
obei
mpl
ement
edascomput
erpr
ogr
ams.Pr
ogr
ammi
nganal
gor
it
hm
pr
esent
sbot
haper
ilandanoppor
tuni
ty.Theper
ill
i
esi
nthepossi
bil
i
tyofmaki
ngt
het
ransi
ti
onf
rom
anal
gor
it
hm t
oapr
ogr
am ei
theri
ncor
rect
lyorv
eryi
nef
fi
cient
ly.
Some i
nfl
uent
ialcomput
ersci
ent
ist
s st
rongl
ybel
i
evet
hatunl
ess t
he cor
rect
ness ofa
comput
erpr
ogr
am i
spr
ovenwi
thf
ullmat
hemat
icalr
igor
,thepr
ogr
am cannotbeconsi
der
edcor
rect
.
Theyhav
edev
elopedspeci
alt
echni
quesf
ordoi
ngsuchpr
oof
s,butt
hepoweroft
heset
echni
quesof
f
ormal
ver
if
icat
ioni
sli
mit
edsof
art
over
ysmal
lpr
ogr
ams.
Asapr
act
ical
mat
ter
,thev
ali
dit
yofpr
ogr
amsi
sst
il
lest
abl
i
shedbyt
est
ing.
Test
ingofcomput
erpr
ogr
amsi
sanar
trat
hert
hanasci
ence,butt
hatdoesnotmeant
hatt
her
eis
not
hingi
nitt
olear
n.Lookupbooksdev
otedt
otest
inganddebuggi
ng;
evenmor
eimpor
tant
,testand
debugy
ourpr
ogr
am t
hor
oughl
ywhenev
ery
oui
mpl
ementanal
gor
it
hm.
Al
sonot
ethatt
hroughoutt
hebook,
weassumet
hati
nput
stoal
gor
it
hmsbel
ongt
othespeci
fi
edset
s
andhencer
equi
renov
eri
fi
cat
ion.Wheni
mpl
ement
ingal
gor
it
hmsaspr
ogr
amst
obeusedi
nact
ual
appl
i
cat
ions,
youshoul
dpr
ovi
desuchv
eri
fi
cat
ions.
Ofcour
se,i
mpl
ement
inganal
gor
it
hm cor
rect
lyi
snecessar
ybutnotsuf
fi
cient
:youwoul
dnotl
i
ket
o
di
mini
shy
oural
gor
it
hm’
spowerbyani
nef
fi
cienti
mpl
ement
ati
on.
Moder
ncompi
l
ersdopr
ovi
deacer
tai
nsaf
etyneti
nthi
sregar
d,especi
all
ywhent
heyar
eusedi
nthei
r
codeopt
imi
zat
ionmode.St
il
l,y
ouneedt
obeawar
eofsuchst
andar
dtr
icksascomput
ingal
oop’
s
i
nvar
iant(
anexpr
essi
ont
hatdoesnotchangei
tsv
alue)out
sidet
hel
oop,col
l
ect
ingcommonsub
expr
essi
ons,
repl
aci
ngexpensi
veoper
ati
onsbycheapones,
andsoon.
Ty
pical
l
y,suchi
mpr
ovement
scanspeedupapr
ogr
am onl
ybyaconst
antf
act
or,wher
easabet
ter
al
gor
it
hm canmakeadi
ff
erencei
nrunni
ngt
imebyor
der
sofmagni
tude.Butonceanal
gor
it
hm i
s
sel
ect
ed,
a10–50%speedupmaybewor
thanef
for
t.
A wor
king pr
ogr
am pr
ovi
desan addi
ti
onaloppor
tuni
tyi
n al
l
owi
ng an empi
ri
calanal
ysi
soft
he
under
lyi
ngal
gor
it
hm.Suchananal
ysi
sisbasedont
imi
ngt
hepr
ogr
am onsev
erali
nput
sandt
hen
anal
yzi
ngt
her
esul
tsobt
ained.
I
nconcl
usi
on,
letusemphasi
zeagai
nthemai
nlessonoft
hepr
ocessdepi
ctedi
nthef
igur
ebel
ow.
Asar
ule,agoodal
gor
it
hm i
sar
esul
tofr
epeat
edef
for
tandr
ewor
k.Ev
eni
fyouhav
ebeenf
ort
unat
e
enought
ogetanal
gor
it
hmi
cideat
hatseemsper
fect
,youshoul
dst
il
ltr
ytoseewhet
heri
tcanbe
i
mpr
oved.

Ont
heot
herhand,how doesoneknow whent
ost
op?I
nther
ealwor
ld,mor
eof
tent
hannota
pr
oject
’sschedul
eort
hei
mpat
ienceofy
ourbosswi
l
lst
opy
ou.Andsoi
tshoul
dbe:per
fect
ioni
s
expensi
veandi
nfactnotal
way
scal
l
edf
or.Desi
gni
nganal
gor
it
hm i
sanengi
neer
ing-
li
keact
ivi
tyt
hat
cal
l
sforcompr
omi
sesamongcompet
inggoal
sundert
heconst
rai
ntsofav
ail
abl
eresour
ces,
wit
hthe
desi
gner
’st
imebei
ngoneoft
her
esour
ces.
I
ntheacademi
cwor
ld,
thequest
ionl
eadst
oani
nter
est
ingbutusual
l
ydi
ff
icul
tinv
est
igat
ionof
anal
gor
it sopt
hm’ imal
it
y.Act
ual
l
y,t
hisquest
ioni
snotaboutt
heef
fi
ciencyofanal
gor
it
hm butabout
t
hecompl
exi
tyoft
hepr
obl
em i
tsol
ves:
Whati
sthemi
nimum amountofef
f tanyal
or gor
it
hm wi
l
lneedt
oexer
ttosol
vet
hepr
obl
em?
Forsomepr
obl
ems,t
heanswert
othi
squest
ioni
sknown.Forexampl
e,anyal
gor
it
hm t
hatsor
tsan
ar
raybycompar
ingv
aluesofi
tsel sneedsaboutnl
ement og2ncompar
isonsf
orsomear
ray
sofsi
ze
n.Butf
ormanyseemi
ngl
yeasypr
obl
emssuchasi
ntegermul
ti
pli
cat
ion,comput
ersci
ent
ist
sdonot
y
ethav
eaf
inal
answer
.
Anot
heri
mpor
tanti
ssueofal
gor
it
hmi
cpr
obl
em sol
vi
ngi
sthequest
ionofwhet
herornotev
ery
pr
obl
em canbesol
vedbyanal
gor
it
hm.Wear
enott
alki
ngher
eaboutpr
obl
emst
hatdonothav
ea
sol
uti
on,suchasf
indi
ngr
ealr
oot
sofaquadr
ati
cequat
ionwi
thanegat
ivedi
scr
imi
nant
.Forsuch
cases,
anout
puti
ndi
cat
ingt
hatt
hepr
obl
em doesnothav
easol
uti
oni
sal
lwecanandshoul
dexpect
f
rom anal
gor
it
hm.Norar
ewet
alki
ngaboutambi
guousl
yst
atedpr
obl
ems.
Ev
ensome unambi
guouspr
obl
emst
hatmusthav
e a si
mpl
eyesorno answerar
e“undeci
dabl
e,
”i.
e.,
unsol
vabl
ebyanyal
gor
it
hm.For
tunat
ely
,av
astmaj
ori
tyofpr
obl
emsi
npr
act
ical
comput
ingcanbesol
vedbyan
al
gor
it
hm.

Whatki
ndsofpr
obl
emsar
esol
vedbyal
gor
it
hms?
Sor
ti
ngi
snott
heonl
ycomput
ati
onalpr
obl
em f
orwhi
chal
gor
it
hmshav
ebeendev
eloped.Pr
act
ical
appl
i
cat
ionsofal
gor
it
hmsar
eubi
qui
tousandi
ncl
udet
hef
oll
owi
ngexampl
es:
o TheHumanGenomePr
ojecthasmadegr
eatpr
ogr
esst
owar
dthegoal
sofi
dent
if
yingal
lthe
100,
000genesi
nhumanDNA,det
ermi
ningt
hesequencesoft
he3bi
l
li
onchemi
calbasepai
rs
t
hatmakeuphumanDNA,st
ori
ngt
hisi
nfor
mat
ioni
ndat
abases,anddev
elopi
ngt
ool
sfordat
a
 anal
ysi
s.Eachoft
hesest
epsr
equi
ressophi
sti
cat
edal
gor
it
hms.

o TheI
nter
netenabl
espeopl
eal
lar
oundt
hewor
ldt
oqui
ckl
yaccessandr
etr
iev
elar
geamount
sof
i
nfor
mat
ion.Wi
tht
heai
dofcl
everal
gor
it
hms,si
tesont
heI
nter
netar
eabl
etomanageand
mani
pul
atet
hisl
argev
olumeofdat
a.Exampl
eofpr
obl
emst
hatmakeessent
ial
useofal
gor
it
hms
i
ncl
udef
indi
nggoodr
out
esonwhi
cht
hedat
awi
l
ltr
avel
.

o El
ect
roni
ccommer
ceenabl
esgoodsandser
vicest
obenegot
iat
edandexchangedel
ect
roni
cal
l
y,
andi
tdependsont
hepr
ivacyofper
sonali
nfor
mat
ionsuchascr
edi
tcar
dnumber
s,passwor
ds,
andbankst
atement
s.

o Manuf
act
uri
ngandot
hercommer
cialent
erpr
isesof
tenneedt
oal
l
ocat
escar
cer
esour
cesi
nthe
mostbenef
ici
alway
.Exampl
es:Anoi
lcompanymaywi
sht
oknow wher
etopl
acei
tswel
l
sin
or
dert
omaxi
mizei
tsexpect
edpr
ofi
t.AnI
nter
netser
vicepr
ovi
dermaywi
sht
odet
ermi
newher
e
t
opl
aceaddi
ti
onalr
esour
cesi
nor
dert
oser
vei
tscust
omer
smor
eef
fect
ivel
y.Al
loft
hesear
e
ex
ampl
esofpr
obl
emst
hatcanbesol
vedusi
ngwel
ldesi
gnedl
i
nearpr
ogr
ammi
ngal
gor
it
hm.

o Wemaybegi
venar
oadmaponwhi
cht
hedi
stancebet
weeneachpai
rofadj
acenti
nter
sect
ions
i
smar
ked,andwewi
sht
odet
ermi
net
heshor
testr
out
efr
om onei
nter
sect
iont
oanot
her
.The
numberofpossi
bler
out
escanbehuge,ev
eni
fwedi
sal
l
owr
out
est
hatcr
ossov
ert
hemsel
ves.
Howdowechoosewhi
chofal
lpossi
bler
out
esi
stheshor
test
?

o Wemayal
sobegi
vent
woor
der
edsequencesofsy
mbol
s,X=(
x1,x2,
…xm)andY=(
y1,
y2….
,
y
n),
andwewi
sht
ofi
ndal
ongestcommonsubsequenceofXandY.AsubsequenceofXi
sjustX
wi
thsome(
orpossi
blyal
lornone)ofi
tsel
ement
sremov
ed.I
fXhasm sy
mbol
sandYhasn
sy
mbol
s,t e2m and2npossi
henXandYhav blesubsequencesr
espect
ivel
y.Sel
ect
ingal
lpossi
ble
subsequencesofXandY,andmat
chi
ngt
hem upcoul
dtakeapr
ohi
bit
ivel
ylongt
imeunl
essm
andnar
ever
ysmal
l
.

Theabov
eexampl
esar
efarf
rom exhaust
ive,buteachexhi
bitt
wochar
act
eri
sti
cst
hatar
e
commont
omanyi
nter
est
ingal
gor
it
hmi
cpr
obl
ems:
1.Theyhav
emanycandi
dat
esol
uti
ons,t
heov
erwhel
mingmaj
ori
tyofwhi
chdonotsol
ve
t
hepr
obl
em athand.Fi
ndi
ngonet
hatdoes,oronet
hati
s“best
,”canpr
esentqui
tea
chal
l
enge.

2.Theyhav
epr
act
icalappl
icat
ions,Oft
hepr
obl
emsi
ntheabov
eexampl
e,f
indi
ngt
he
shor
testpat
hpr
ovi
dest
heeasi
estexampl
es.At
ranspor
tat
ionf
ir
m,suchasat
rucki
ngor
r
ail
roadcompany
,hasaf
inanci
ali
nter
esti
nfi
ndi
ngshor
testpat
hst
hroughar
oadorr
ail
 net
wor
kbecauset
aki
ngshor
terpat
hsr
esul
tsi
nlowerl
aborandf
uelcost
s.Orar
out
ing
nodeont
heI
nter
netmayneedt
ofi
ndt
heshor
testpat
hthr
ought
henet
wor
kinor
dert
o
r
out
eamessagequi
ckl
y.

I
mpor
tantPr
obl
em Ty
pes
I
nthel
i
mit
lessseaofpr
obl
emsoneencount
ersi
ncomput
ing,
ther
ear
eaf
ewar
east
hathav
e
at
tr
act
edpar
ti
cul
arat
tent
ionf
rom r
esear
cher
s.Byandl
arge,t
hei
rint
eresthasbeendr
iven
ei
therbyt
hepr
obl
em’
spr
act
icali
mpor
tanceorbysomespeci
fi
cchar
act
eri
sti
csmaki
ngt
he
pr
obl
em ani
nter
est
ingr
esear
chsubj
ect
;for
tunat
ely
,theset
womot
ivat
ingf
orcesr
einf
orce
eachot
heri
nmostcases.
I
nthi
ssect
ion,
wear
egoi
ngt
oint
roducet
hemosti
mpor
tantpr
obl
em t
ypes:
Sor
ti
ng
Sear
chi
ng
St
ri
ngpr
ocessi
ng
Gr
aphpr
obl
ems
Combi
nat
ori
alpr
obl
ems
Geomet
ri
cpr
obl
ems
Numer
ical
probl
ems

Sor
ti
ng
Thesor
ti
ngpr
obl
em i
stor
ear
ranget
hei
temsofagi
venl
i
sti
nnon-
decr
easi
ngor
der
.Of
cour
se,
forapr
obl
em t
obemeani
ngf
ul,
thenat
ureoft
hel
i
sti
temsmustal
l
owor
der
ing.
(
Mat
hemat
ici
answoul
dsayt
hatt
her
emustexi
star
elat
ionoft
otal
order
ing.
)
Whywoul
dwewantasor
tedl
i
st?
o Asor
tedl
i
stcanbear
equi
redout
putofat
asksuchasr
anki
ngI
nter
netsear
chr
esul
ts
orr
anki
ngst
udent
sbyt
hei
rGPAscor
es.
o Fur
ther
,sor
ti
ngmakesmanyquest
ionsaboutt
hel
i
steasi
ert
oanswer
.
o Makessear
chi
ngeasy
;thi
siswhydi
cti
onar
ies,
tel
ephonebooks,
classl
i
sts,
andsoon
ar
esor
ted.
Bynow,
comput
ersci
ent
ist
shav
edi
scov
ereddozensofdi
ff
erentsor
ti
ngal
gor
it
hms.Andt
he
huntf
orabet
tersor
ti
ngmouset
rapcont
inues.
Thi
sper
sev
erancei
sfort
hef
oll
owi
ngf
act
s.
Ont
heonehand,
ther
ear
eaf
ewgoodsor
ti
ngal
gor
it
hmst
hatsor
tanar
bit
rar
yar
rayof
zenusi
si ngaboutnl
og2ncompar
isons.
Ont
heot
herhand,
noal
gor
it
hm t
hatsor
tsbykeycompar
isons(
asopposedt
o,say
,
compar
ingsmal
lpi
ecesofkey
s)candosubst
ant
ial
l
ybet
tert
hant
hat
.
Al
thoughsomeal
gor
it
hmsar
eindeedbet
tert
hanot
her
s,t
her
eisnoal
gor
it
hm t
hatwoul
dbe
t
hebestsol
uti
oni
nal
lsi
tuat
ions.
o Someoft
heal
gor
it
hmsar
esi
mpl
ebutr
elat
ivel
ysl
ow,whi
l
eot
her
sar
efast
erbutmor
e
compl
ex;
o somewor
kbet
teronr
andoml
yor
der
edi
nput
s,whi
l
eot
her
sdobet
teronal
most
-sor
ted
l
i
sts;
o somear
esui
tabl
eonl
yforl
i
stsr
esi
dingi
nthef
astmemor
y,whi
l
eot
her
scanbeadapt
ed
f
orsor
ti
ngl
argef
il
esst
oredonadi
sk;
andsoon.

Sear
chi
ng
Thesear
chi
ngpr
obl
em deal
swi
thf
indi
ngagi
venv
alue, edasear
cal
l chkey
,inagi
venset(
oramul
ti
set
,whi
chper
mit
ssev
eral
element
stohav
ethesamev
alue)
.
Ther
ear
epl
ent
yofsear
chi
ngal
gor
it
hmst
ochoosef
rom.Theyr
angef
rom t
hest
rai
ght
for
war
d
sequent
ialsear
cht
oaspect
acul
arl
yef
fi
cientbutl
i
mit
edbi
nar
ysear
chandal
gor
it
hmsbasedon
r
epr
esent
ingt
heunder
lyi
ngseti
nadi
ff
erentf
orm mor
econduci
vet
osear
chi
ng.Thel
att
eral
gor
it
hms
ar
eofpar
ti
cul
ari
mpor
tancef
orr
eal
-wor
ldappl
i
cat
ionsbecauset
heyar
eindi
spensabl
eforst
ori
ngand
r
etr
iev
ingi
nfor
mat
ionf
rom l
argedat
abases.
Forsear
chi
ng,t
oo,t
her
eisnosi
ngl
eal
gor
it
hm t
hatf
it
sal
lsi
tuat
ionsbest
.Someal
gor
it
hms
wor
kfast
ert
hanot
her
sbutr
equi
remor
ememor
y;somear
ever
yfastbutappl
i
cabl
eonl
ytosor
ted
ar
ray
s;andsoon.
Unl
i
ke wi
th sor
ti
ng al
gor
it
hms,t
her
eis no st
abi
l
itypr
obl
em,butdi
ff
erenti
ssues ar
ise.
Speci
fi
cal
l
y,i
nappl
i
cat
ionswher
etheunder
lyi
ngdat
amaychangef
requent
lyr
elat
ivet
othenumberof
sear
ches,
sear
chi
nghast
obeconsi
der
edi
nconj
unct
ionwi
tht
woot
heroper
ati
ons:anaddi
ti
ont
oand
del
eti
onf
rom t
hedat
asetofani
tem.
I
nsuchsi
tuat
ions,
dat
ast
ruct
uresandal
gor
it
hmsshoul
dbechosent
ost
ri
keabal
anceamong
t
her
equi
rement
sofeachoper
ati
on.Al
so,
organi
zi
ngv
eryl
argedat
aset
sforef
fi
cientsear
chi
ngposes
speci
alchal
l
engeswi
thi
mpor
tanti
mpl
i
cat
ionsf
orr
eal
-wor
ldappl
i
cat
ions.

St
ri
ngPr
ocessi
ng
I
nrecentdecades,t
her
api
d pr
oli
fer
ati
on ofappl
i
cat
ionsdeal
i
ng wi
th non numer
icaldat
a has
i
ntensi
fi
edt
hei
nter
estofr
esear
cher
sandcomput
ingpr
act
it
ioner
sinst
ri
ng-
handl
i
ngal
gor
it
hms.
Ast
ri
ngi
sasequenceofchar
act
ersf
rom anal
phabet
.St
ri
ngsofpar
ti
cul
ari
nter
estar
etext
st
ri
ngs,whi
chcompr
isel
ett
ers,number
s,andspeci
alchar
act
ers;bi
tst
ri
ngs,whi
chcompr
isezer
os
andones,
etc.
I
tshoul
dbepoi
ntedout
,howev
er,t
hatst
ri
ng-
processi
ngal
gor
it
hmshav
ebeeni
mpor
tantf
or
comput
ersci
encef
oral
ongt
imei
nconj
unct
ionwi
thcomput
erl
anguagesandcompi
l
ingi
ssues.

Gr
aphPr
obl
ems
Oneoft
heol
destandmosti
nter
est
ingar
easi
nal
gor
it
hmi
csi
sgr
aphal
gor
it
hms.
I
nfor
mal
l
y,agr
aphcanbet
houghtofasacol
l
ect
ionofpoi
ntscal
l
edv
ert
ices,someofwhi
char
e
connect
edbyl
i
nesegment
scal
l
ededges.
Gr
aphsar
eani
nter
est
ingsubj
ectt
ost
udy
,forbot
htheor
eti
cal
andpr
act
ical
reasons.
Gr
aphscanbeusedf
ormodel
l
ingawi
dev
ari
etyofappl
i
cat
ions,i
ncl
udi
ngt
ranspor
tat
ion,
communi
cat
ion,
soci
alandeconomi
cnet
wor
ks,
proj
ectschedul
i
ng,
andgames.
St
udy
ingdi
ff
erentt
echni
calandsoci
alaspect
soft
heI
nter
neti
npar
ti
cul
ari
soneoft
heact
ive
ar
easofcur
rentr
esear
chi
nvol
vi
ngcomput
ersci
ent
ist
s,economi
sts,
andsoci
alsci
ent
ist
s.
Basi
cgr
aphal
gor
it
hmsi
ncl
ude:
o Gr
aph-
tr
aver
sal
algor
it
hms(
howcanoner
eachal
lthepoi
ntsi
nanet
wor
k?)
,
o Shor
test
-pat
hal
gor
it
hms(
whati
sthebestr
out
ebet
weent
woci
ti
es?
),and
o Topol
ogi
calsor
ti
ngf
orgr
aphswi
thdi
rect
ededges(
isasetofcour
seswi
tht
hei
rpr
erequi
sit
es
consi
stentorsel
f-
cont
radi
ctor
y?)
.
Theseal
gor
it
hmscanbeconsi
der
edi
l
lust
rat
ionsofgener
aldesi
gnt
echni
ques.
Somegr
aphpr
obl
emsar
ecomput
ati
onal
l
yver
yhar
d;exampl
e,het
t ravel
li
ngsal
esmanpr
obl
em (
TSP)
i
sthepr
obl
em off
indi
ngt
heshor
testt
ourt
hroughnci
ti
est
hatv
isi
tsev
eryci
tyexact
lyonce.

Combi
nat
ori
alPr
obl
ems
Fr
om amor
eabst
ractper
spect
ive,t
het
rav
eli
ngsal
esmanpr
obl
em andt
hegr
aphcol
ori
ngpr
obl
em
ar esofcombi
eexampl nat
ori
alpr
obl
ems.
Thesear
epr
obl
emst
hatask,expl
i
cit
lyori
mpl
i
cit
ly,t
ofi
ndacombi
nat
ori
alobj
ect
—suchasa
per
mut
ati
on,
acombi
nat
ion,
orasubset
—thatsat
isf
iescer
tai
nconst
rai
nts.
Adesi
redcombi
nat
ori
alobj
ectmayal
sober
equi
redt
ohav
esomeaddi
ti
onalpr
oper
tysuchas
amaxi
mum v
alueorami
nimum cost
.Gener
all
yspeaki
ng,combi
nat
ori
alpr
obl
emsar
ethemost
di
ff
icul
tpr
obl
emsi
ncomput
ing,
from bot
hat
heor
eti
cal
andpr
act
ical
standpoi
nt.
Thei
rdi
ff
icul
tyst
emsf
rom t
hef
oll
owi
ngf
act
s.
Fi
rst
,thenumberofcombi
nat
ori
alobj
ect
sty
pical
l
ygr
owsext
remel
yfastwi
thapr
obl
em’
ssi
ze,
r
eachi
nguni
magi
nabl
emagni
tudesev
enf
ormoder
ate-
sizedi
nst
ances.
Second,t
her
ear
enoknownal
gor
it
hmsf
orsol
vi
ngmostofsuchpr
obl
emsexact
lyi
nan
accept
abl
eamountoft
ime.
Mor
eov
er,mostcomput
ersci
ent
ist
sbel
i
evet
hatsuchal
gor
it
hmsdonotexi
st.Thi
sconj
ect
urehas
beennei
therpr
ovednordi
spr
oved,
andi
tremai
nst
hemosti
mpor
tantunr
esol
vedi
ssuei
ntheor
eti
cal
comput
ersci
ence.
Somecombi
nat
ori
alpr
obl
emscanbesol
vedbyef
fi
ciental
gor
it
hms,butt
heyshoul
dbe
consi
der
edf
ort
unat
eexcept
ionst
other
ule.Theshor
test
-pat
hpr
obl
em ment
ionedear
li
eri
samong
suchexcept
ions.

Geomet
ri
cPr
obl
ems
Geomet
ri
cal
gor
it
hmsdealwi
thgeomet
ri
cobj
ect
ssuchaspoi
nts,
li
nes,
andpol
ygons.
Theanci
entGr
eekswer
ever
ymuchi
nter
est
edi
ndev
elopi
ngpr
ocedur
es(
theydi
dnotcal
lthem
al
gor
it
hms,
ofcour
se)f
orsol
vi
ngav
ari
etyofgeomet
ri
cpr
obl
ems,
incl
udi
ngpr
obl
emsofconst
ruct
ing
si
mpl
egeomet
ri
cshapes—t
ri
angl
es,
cir
cles,
andsoon—wi
thanunmar
kedr
ulerandacompass.
Then,f
orabout2000y
ear
s,i
ntensei
nter
esti
ngeomet
ri
cal
gor
it
hmsdi
sappear
ed,t
o be
r
esur
rect
edi
ntheageofcomput
ers;nomor
erul
ersandcompasses,j
ustbi
ts,by
tes,andgoodol
d
humani
ngenui
ty.
Of cour
se,t
oday peopl
e ar
eint
erest
ed i
n geomet
ri
c al
gor
it
hms wi
th qui
te di
ff
erent
appl
i
cat
ionsi
nmi
nd,
suchascomput
ergr
aphi
cs,
robot
ics,
andt
omogr
aphy
.

Numer
icalPr
obl
ems
Numer
icalpr
obl
ems,anot
herl
arge speci
alar
ea of appl
i
cat
ions,ar
e pr
obl
ems t
hati
nvol
ve
mat
hemat
icalobj
ect
sofcont
inuousnat
ure:sol
vi
ngequat
ionsandsy
stemsofequat
ions,comput
ing
def
ini
tei
ntegr
als,ev
aluat
ingf
unct
ions,andsoon.Themaj
ori
tyofsuchmat
hemat
icalpr
obl
emscan
besol
vedonl
yappr
oxi
mat
ely
.
Themaj
ordi
ff
icul
tyst
emsf
rom t
hef
actt
hatsuchpr
obl
emst
ypi
cal
l
yrequi
remani
pul
ati
ngr
eal
number
s,whi
chcanber
epr
esent
edi
nacomput
eronl
yappr
oxi
mat
ely
.
Mor
eov
er,al
argenumberofar
it
hmet
icoper
ati
onsper
for
medonappr
oxi
mat
elyr
epr
esent
ed
number
scanl
eadt
oanaccumul
ati
onoft
her
ound-
offer
rort
oapoi
ntwher
eitcandr
ast
ical
l
ydi
stor
t
anout
putpr
oducedbyaseemi
ngl
ysoundal
gor
it
hm.
Manysophi
sti
cat
edal
gor
it
hmshav
ebeendev
elopedov
ert
hey
ear
sint
hisar
ea,
andt
heycont
inuet
o
pl
ayacr
it
icalr
olei
nmanysci
ent
if
icandengi
neer
ingappl
i
cat
ions.Buti
nthel
ast30y
ear
sorso,t
he
comput
ingi
ndust
ryhasshi
ft
edi
tsf
ocust
obusi
nessappl
i
cat
ions.Thesenew appl
i
cat
ionsr
equi
re
pr
imar
il
y al
gor
it
hms f
or i
nfor
mat
ion st
orage,r
etr
iev
al,t
ranspor
tat
ion t
hrough net
wor
ks,and
pr
esent
ati
ont
ouser
s.
Asar
esul
toft
hisr
evol
uti
onar
ychange,numer
icalanal
ysi
shasl
osti
tsf
ormer
lydomi
nat
ing
posi
ti
oni
nbot
hindust
ryandcomput
ersci
encepr
ogr
ams.St
il
l,i
tisi
mpor
tantf
oranycomput
er-
l
i
ter
ateper
sont
ohav
eatl
eastar
udi
ment
aryi
deaaboutnumer
ical
algor
it
hms.

Ef
fi
ciencyandPer
for
manceofAl
gor
it
hms
From the f
oregoing di
scussions,we can pi ctur
et hatanalysi
s ofal
gor
it
hm is qui
te
i
mpor t
antincomput erprogr
ammi ng,si
ncewehav earguedthatther
ear
eusual
lysever
al
al
gorit
hmsav ail
ableforapar t
icularappl
i
cation,theimportantandsensi
blet
aski sto
deter
minewhichoptionisthebestf orapart
icularpr
oblem.

- Theanal
ysi
sframeworkforal
gor
it
hmsi
nthef
ir
sti
nst
ance,consi
der
stheef
fi
ciency
andcor
rect
nessofal
gori
thms.

- Theef f
ici
encyconsider
ati
onsusual
lyi nv
esti
gat
et hedualissuesoftimeandspace.
Wheret imeeffi
ciencydealswith how much timean al gori
thm uses,and space
eff
ici
encydealswi thhow muchspace( memoryal l
ocat
ion)thealgorit
hm uses,in
addit
iontothespaceneededfort
hei nputandout
put

Forabet
terunder
standi
ngoft
het
opi
cofef
fi
ciencyandt
hemot
ivat
ionf
orchoosi
ngone
al
gor
it
hm ov
ert
heot
her(
obj
ect
ivel
yandquant
it
ati
vel
y),l
etusconsi
deracompar
ison
 bet
weensequent
ial
andbi
nar
ysear
ch.

- Supposewehav
eami
l
li
onr
ecor
dtosear
ch;

- Wih sequent
t ialsearch,onav
eragewewouldsear
chhal
foft
hel
i
sti
.e.500,
000
r
ecordsbefor
ewef i
ndwhatwearelooki
ngf
or.

o I
ftheit
em wear
elooki
ngf
ori
sthef
ir
stonei
nthel
i
st,t
henwepul
lasi
ngl
e
r
ecor
d;

o i
fiti
sthel
astone,
thenweenduppul
l
ingal
lonemi
l
li
onr
ecor
ds.

- Wi
thbi
nar
ysear
ch,
wel
ookatoner
ecor
d;t
hemi
ddl
eone−per“
tur
n”,
thenspl
i
tthel
i
st
i
nhal
f;so,wegof
rom 1mi
l
li
ont
o½mi
l
li
on,t
o¼mi
l
li
on,t
o125,
000,t
o62,500,
…,30,
t
o15,
7,3,
1,hav
ingamaxi
mum of20“
tur
ns”t
ogot
hroughal
lmi
l
li
onr
ecor
ds.

o Thus,
thebestcase,
forbi
nar
ysear
chcanbewi
tht
hef
ir
str
ecor
d(si
ngl
erecor
d)
whi
l
ethewor
stcasei
stopul
lamaxi
mum of20.

Theabov
e,andsi
mil
arexer
cises,i
stheconcer
nofanal
ysi
sofal
gor
it
hm –wher
e,i
n
t
hecaseofsequent
ialandbi
nar
ysear
ch,wel
ookedatt
henumberof“
tur
ns”whi
ch
t
ransl
atesi
ntot
ime.Anot
hermeasur
ewemaywi
sht
oconsi
deri
sther
espect
ivespace
(
memor
y)eachr
equi
res.

Whenwel
ookateachal
gor
it
hm i
ngener
al,
overar
angeofi
nst
ances,
weobser
vet
hat
t
her
ear
ebest
,wor
st,
andav
eragecasesf
oranal
gor
it
hm.Thesesi
tuat
ionsmayv
arywi
del
y
dependi
ngont
hecl
ass(
es)ofal
gor
it
hms.

Ef
fi
ciency
Di
ff
erental
gor
it
hmsdev
elopedt
osol
vet
hesamepr
obl
em of
tendi
ff
erdr
amat
ical
l
yint
hei
r
ef
fi
ciency
.Thesedi
ff
erencescanbemuchmor
esi
gni
fi
cantt
handi
ff
erencesduet
ohar
dwar
e
andsof
twar
e.
Asanexampl
e,l
ater
,wewi
l
lseehowt
heset
woal
gor
it
hmsf
orsor
ti
ng:
o Thef
ir
sti
,nser
ti
onsor
t,t
akest
imer
oughl oc1n2 t
yequalt osor
tni
tems,
wher
ec1i
sa
const
antt
hatdoesnotdependonn.Thati
s,i
ttakest
imer
oughl
ypr
opor
ti
onal
ton2.
o Thesecondmer
gesor
ttakest
imer
oughl oc2nl
yequalt ogn,wher
elognst
andsf
or
og2nandc2i
l sanot
herconst
antt
hatal
sodoesnotdependonn.
Thei
nser
ti
onsor
tty
pical
l
yhasasmal
l
erconst
antf
act
ort
hanmer
gesor
t,sot
hatc1<c2.
Exper
ience/
obser
vat
ionshav
eshownt
hatt
heconst
antf
act
orsc1andc2hav
efarl
ess
i
mpactont
her
unni
ngt
imet
hant
hedependenceont
hei
nputsi
zen.
I
fwer
ewr
it
ethet
wor
unni
ngt
ime,
wehav
e:
 f
ori
nser
ti
onsor
t:c1.n.
nand
f
ormer
gesor
t:c2.n.
lgn.
wenot
ethatal
thoughi
nser
ti
onsor
tusual
l
yrunsf
ast
ert
hanmer
gesor
tforsmal
li
nputsi
zes,
oncet
hei
nputsi
zenbecomesl
argeenough,
mer
gesor
t’
sadv
ant
ageofl
ognv
s.nwi
l
lmor
e
t
hancompensat
efort
hedi
ff
erencei
nconst
antf
act
ors.
Exampl
e
Letuscompar
eaf
ast
ercomput
er(
comput
erA)r
unni
ngi
nser
ti
onsor
tagai
nsta
sl
owercomput
er(
comput
erB)r
unni
ngmer
gesor
t.Eachmustsor
tanar
rayof10mi
l
li
on
number
s.
Supposef
urt
hert
hatj
ustanav
eragepr
ogr
ammeri
mpl
ement
smer
gesor
t,usi
ngahi
gh
-
lev
ell
anguage wi
th an i
nef
fi
cientcompi
l
er,wi
tht
he r
esul
ti
ng code t
aki
ng 50.
n.l
og.
n
i
nst
ruct
ions.

Tosor
t10mi
l
li
onnumber
s,comput
erAt
akes:
107)
2
2( inst
ruct
ions= 20,
000seconds(
mor
ethan5.
5hour
s)
1010i
nst
ruct
ions/
second
Whi
l
eBt
akes:

Ther
efor
e,usi
nganal
gor
it
hm whoser
unni
ngt
imegr
owsmor
esl
owl
y,ev
enwi
thapoor
compi
l
er,
comput
erBr
unsmor
ethan17t
imesf
ast
ert
hancomput
erA!
Theadv
ant
ageofmer
gesor
tisev
enmor
epr
onouncedwhenwesor
t100mi
l
li
on
number
s:wher
einser
ti
onsor
ttakesmor
ethan23day
s,asagai
nst4hr
sformer
gesor
t.
I
ngener
al,
ast
hepr
obl
em si
zei
ncr
eases,
sodoest
her
elat
iveadv
ant
ageofmer
gesor
t.

Al
gor
it
hmsandot
hert
echnol
ogi
es
Theexampl
eabov
eshowst
hatweshoul
dconsi
deral
gor
it
hms,
li
kecomput
erhar
dwar
e,asa
t
echnol
ogy
.Tot
alsy
stem per
for
mancedependsonchoosi
ngef
fi
ciental
gor
it
hmsasmuchas
onchoosi
ngf
asthar
dwar
e.Justasr
api
dadv
ancesar
ebei
ngmadei
not
hercomput
er
t
echnol
ogi
es,
theyar
ebei
ngmadei
nal
gor
it
hmsaswel
l
Mor
eov
er,ev
en an appl
i
cat
ion t
hatdoes notr
equi
re al
gor
it
hmi
c cont
entatt
he
appl
i
cat
ionl
evel
rel
i
esheav
il
yuponal
gor
it
hmsi
ftheappl
i
cat
ion:
o r
eli
esonf
asthar
dwar
e;t
hehar
dwar
edesi
gnusedal
gor
it
hms.
o r
eli
esonGUI
,thedesi
gnofanyGUIr
eli
esonal
gor
it
hms.
 o r
eli
esonnet
wor
king;
Rout
ingi
nnet
wor
ksr
eli
esheav
il
yonal
gor
it
hms.
o i
swr
it
teni
nal
anguageot
hert
hanmachi
necode,
theni
twaspr
ocessedbyacompi
l
er,
i
nter
pret
er,
orassembl
er,
all
ofwhi
chmakeext
ensi
veuseofal
gor
it
hms.
Ther
efor
e,al
gor
it
hmsar
eatt
hecor
eofmostt
echnol
ogi
esusedi
ncont
empor
arycomput
ers.
Fur
ther
mor
e,wi
tht
heev
er-
incr
easi
ngcapaci
ti
esofcomput
ers,weuset
hem t
osol
ve
l
argerpr
obl
emst
hanev
erbef
ore.Ther
efor
e,hav
ingasol
i
dbaseofal
gor
it
hmi
cknowl
edge
andt
echni
quei
sonechar
act
eri
sti
cthatsepar
atest
het
rul
yski
l
ledpr
ogr
ammer
sfr
om t
he
nov
ices.

Appr
oaches(
met
hods)ofAnal
ysi
sofAl
gor
it
hms

Thissecti
onexplai
nshow t omeasur ei nputsi
zesandr unni
ngt imes.Tounder st
andthe
growt
hr at
eofapr ocedur
e’sr unningt i
me,weneedaf uncti
ont hatmapst hesizeofthe
i
nputstotheprocedur
etot heamountoft i
meittakestoev aluatetheappli
cati
on.Fir
stwe
consi
derhowtomeasurethei nputsize;then,weconsiderhowt omeasur etherunni
ngtime
- Therear
efundamentall
yt woappr oachesforanalysisofalgori
thms.

Thesear e;t hetheoret

icalanal
ysi
s− using proofofcor r
ectness andasymptoti
c
represent
ations( bi
g− ohandr el
atednotat
ions);and usi
ng empiricalanal
ysi
s−
i
nv olvi
ngtestingandmeasurementoverar
angeofi nst
ances(benchmar k)
.

Byconsi deri
ngtheasy mptoticgrowthoff uncti
ons, rat
herthanthei
ract
ualoutputs,theO,W,
andQoper at
orsallowust ohideconst antsandf actorsthatchangedependi
ngont hespeed
ofourprocessor,howdat aisarrangedi nmemor y,andthespecifi
csofhowouri nterpr
eteri
s
i
mpl emented.Instead,wecanconsi dertheessent ialproper
ti
esofhow therunningt i
meof
theproceduresincreaseswit
ht hesizeoft heinput.
Irr
especti
veoftheappr oach,considerat
ionsofef f
iciencycanbeby :
 afuncti
onofI nputsi
ze, measur i
ngr
unni
ngt
ime,
or anal
ysi
s of t
he wor
st-
case,best
-case,andaver
age-
case.

(
1)Ef
fi
ciencyasaf
unct
ionofi
nputsi
ze:

o Pr
ocedur
einput
smaybemanydi
ff
erentt
ypes:Number
s,Li
stsofNumber
s,Li
stsof
Li
sts,Pr
ocedur
es,et
c.Ourgoali
stochar
act
eri
zet
hei
nputsi
zewi
thasi
ngl
enumber
t
hatdoesnotdependont
het
ypesoft
hei
nput
.
o Weuset
heTur
ingmachi
net
omodelacomput
er,
sot
hewayt
omeasur
ethesi
zeoft
he
i
nputi
sthenumberofchar
act
ersneededt
owr
it
ethei
nputont
het
ape.
o Thechar
act
erscanbef
rom anyf
ixed-
sizeal
phabet
,suchast
het
endeci
maldi
git
s,or
t
hel
ett
ersoft
heal
phabet
.
o Thenumberofdi
ff
erentsy
mbol
sint
het
apeal
phabetdoesnotmat
terf
orouranal
ysi
s
si
ncewear
econcer
nedwi
thor
der
sofgr
owt
hnotabsol
utev
alues
Thus,
wemeasur
ethesi
zeoft
hei
nputast
henumberofsy
mbol
srequi
redt
owr
it
ethe
numberonaTur
ingMachi
nei
nputt
ape.
Tof
igur
eoutt
hei
nputsi
zeofagi
vent
ype,
weneedt
othi
nkabouthowmanysy
mbol
sit
woul
drequi
ret
owr
it
edowni
nput
soft
hatt
ype.Fori
nst
ance:
Bool
eans-t
her
ear
eonl
ytwoBool
eanv
alues:t
rueandf
alse.Hence,t
hel
engt
hofaBool
ean
i
nputi
sfi
xed.
Number
s-usi
ngt
hedeci
malnumbersy
stem (
thati
s,10t
apesy
mbol
s),wecanwr
it
ea
numberofmagni
tudenusi
ngl
og10ndi
git
s.Usi
ngt
hebi
nar
ynumbersy
stem (
thati
s,2t
ape
sy
mbol
s),wecanwr
it
eitusi
ngl
og2nbi
ts.Wi
thi
ntheasy
mpt
oti
coper
ator
s,t
hebaseoft
he
l
ogar
it
hm doesnotmat
ter(
asl
ongasi
tisaconst
ant
)si
ncei
tchangest
her
esul
tbya
const
antf
act
or.Wecanseet
hisf
rom t
hear
gumentabov
e— changi
ngt
henumberof
sy
mbol
sint
hei
nputal
phabetchangest
hei
nputl
engt
hbyaconst
antf
act
orwhi
chhasno
i
mpactwi
thi
ntheasy
mpt
oti
coper
ator
s.
Li
sts-i
fthei
nputi
saLi
st,
thesi
zeoft
hei
nputi
srel
atedt
othenumberofel
ement
sint
hel
i
st.
o I
feachel
ementi
saconst
antsi
ze(
forexampl
e,al
i
stofnumber
swher
eeachnumber
i
sbet
ween0and100)
,thesi
zeoft
hei
nputl
i
sti
ssomeconst
antmul
ti
pleoft
he
numberofel
ement
sint
hel
i
st.Hence,
thesi
zeofani
nputt
hati
sal
i
stofnel
ement
sis
cnf
orsomeconst
antc.Si
nceΘ(
cn)=Θ(
n),t
hesi
zeofaLi
sti
nputi
sΘ(
n)wher
eni
s
t
henumberofel
ement
sint
heLi
st.
o I
fLi
stel
ement
scanv
aryi
nsi
ze,
thenweneedt
oaccountf
ort
hati
nthei
nputsi
ze.For
exampl
e,supposet
hei
nputi
saLi
stofLi
sts,
wher
ether
ear
enel
ement
sineachi
nner
Li
st,andt
her
ear
enLi
stel
ement
sint
hemai
nLi
st.Then,t
her en2 t
ear otalel
ement
s
andt
hei
nputsi
zei
si n2)
nΘ( .
I
ngeneral
,moreti
meandspaceareneededtor
unanal
gor
it
hm onbi
ggeri
nput
s(e.
g.mor
e
number
s,lar
gerst
ri
ngs,
lar
gergr
aphs,et
c.)
.

Toanal
yseef
fi
ci
encyasaf
unct
ionofi
nputsi
zen;

- InSearchi
ng/sorti
ng−n=numberofi t
emsi nl
i
st
- InStr
ingprocessing−n=l engt
hofstr
ing(s)
- InMatrixoper
ations−n=dimensionofmat r
ix;nxnmatrxhasn2
i
elements.
- InGraphprocessing-nv =numberofvert
icesandNg =numberofedges.

(
2) Measur
ingRunni
ngTi
me:

Wewantameasur
eoft
her
unni
ngt
imeofapr
ocedur
ethatsat
isf
iest
wopr
oper
ti
es:
(
1)i
tshoul
dber
obustt
oephemer
alpr
oper
ti
esofapar
ti
cul
arexecut
ionorcomput
er,
and
(
2)i
tshoul
dpr
ovi
dei
nsi
ght
sint
ohow l
ongi
ttakesev
aluat
ethepr
ocedur
eonawi
de
r
angeofi
nput
s.
Toest
imat
ether
unni
ngt
imeofanev
aluat
ion,weuset
henumberofst
epsr
equi
redt
o
per
for
mtheev
aluat
ion.Theact
ualnumberofst
epsdependsont
hedet
ail
sofhow much
wor
kcanbedoneoneachst
ep.Foranypar
ti
cul
arpr
ocessor
,bot
hthet
imei
ttakest
o
per
for
m ast
epandt
heamountofwor
kthatcanbedonei
nonest
epv
ari
es.
Whenweanal
yzepr
ocedur
es,howev
er,weusual
l
ydon’
twantt
odealwi
tht
hesedet
ail
s.
I
nst
ead,whatwecar
eabouti
show t
her
unni
ngt
imechangesast
hei
nputsi
zei
ncr
eases.
Thi
smeanswecancountany
thi
ng wewantasa“
step”asl
ong aseachst
ep i
sthe
appr
oxi
mat
elysamesi
zeandt
het
imeast
epr
equi
resdoesnotdependont
hesi
zeoft
he
i
nput
Henceiti
sreasonabletoassumeallt
heev al
uat
ionr
ulest
otakeconst
antt
ime.Thi
sdoesnot
i
ncludeanyaddit
ionaleval
uat
ionr
ulesthatar
eneededtoappl
yonerul
e.

Thet
hingst
odoar
etoi
dent
if
ythemosti
mpor
tantoper
ati
onoft
heal
gor
it
hm,
cal
l
edt
he
basi
coper
ati
on,
theoper
ati
oncont
ri
but
ingt
hemostt
othet
otal
runni
ngt
ime,
andcomput
e
t
henumberoft
imest
hebasi
coper
ati
oni
sexecut
ed.
Ther
efor
e,measur
ingt
imeef
fi
ciencyofal
gor
it
hmsi
nvol
ves:
o I
dent
if
yingt
hebasi
coper
ati
on(
s)cont
ri
but
ingt
hemostt
orunni
ngt
ime.
o Char
act
eri
ze(
descr
ibe)t
henumberoft
imesi
tisper
for
medasaf
unct
ionofi
nput
si
ze.
Basi
cal
l
y,wecanesti
mat
erunni
ngt i
meT(n)by:
T(
n)≈Cop*C( n).
Where:
T(n) - runni
ngti
measafuncti
onofn
Cop - runni
ngti
measasingl
ebasi
coperati
on,and
C(n) - numberofbasi
coper
ati
onsasafuncti
onofn.

Asar
ule,i
tisnotdi
ff
icul
ttoi
dent
if
ythebasi
coper
ati
onofanal
gor
it
hm.I
tisusual
l
ythemostt
ime-
consumi
ngoper
ati
oni
ntheal
gor
it
hm’
sinner
mostl
oop.
Forexampl
e,mostsor
ti
ngal
gor
it
hmswor
kbycompar
ingel
ement
s(key
s)ofal
i
stbei
ngsor
ted
wi
theachot
her
;forsuchal
gor
it
hms,
thebasi
coper
ati
oni
sakeycompar
ison.
Anot
herexampl
e,al
gor
it
hmsf
ormat
hemat
icalpr
obl
emst
ypi
cal
l
yinv
olv
esomeoral
loft
hef
our
ar
it
hmet
icaloper
ati
ons:addi
ti
on,subt
ract
ion,mul
ti
pli
cat
ion,anddi
vi
sion.Oft
hef
our
,themostt
ime-
consumi
ngoper
ati
oni
sdi
vi
sion,
fol
l
owedbymul
ti
pli
cat
ionandt
henaddi
ti
onandsubt
ract
ion,
wit
hthe
l
astt
wousual
l
yconsi
der
edt
oget
her

(
3) Wor st-
case,
Best-
case,andAverage-
caseAnaly
sis
Apr
ocedur
emayhav edi
ffer
entrunni
ngtimesfori
nputsoft
hesamesi
ze.
Forexampl
e,consi
dert
hispr
ocedur
ethatt
akesaLi
stasi
nputandout
put
sthef
ir
stposi
ti
ve
numberi
nthel
i
st:
(
def
i l
ne(i
st-
fi
rst
-posp)
 (
i nul
f( l
?p)(
err
or"
Noposi
ti
veel
ementf
ound"
)
(
if(
>(carp)0)(
carp)(
li
st-
fi
rst
-pos(
cdrp)
)))
)
I
fthef
ir
stel
ementi
nthei
nputl
i
sti
sposi
ti
ve,
eval
uat
ingt
heappl
i
cat
ionofl
i
stf
ir
st-
pos
r
equi
resv
eryl
i
ttl
ewor
k.I
tisnotnecessar
ytoconsi
deranyot
herel
ement
sint
hel
i
sti
fthe
f
ir
stel
ementi
sposi
ti
ve.
Ont
heot
herhand,
ifnoneoft
heel
ement
sar
eposi
ti
ve,
thepr
ocedur
eneedst
otesteach
el
ementi
nthel
i
stunt
ili
treachest
heendoft
hel
i
st(
wher
ethebasecaser
epor
tsaner
ror
).
I
nouranal
ysesweusual
l
yconsi hewor
dert stcasei
nput
.Foragi
vensi
ze,t
hewor
st
casei
nputi
sthei
nputf
orwhi
chev
aluat
ingt
hepr
ocedur
etakest
hemostwor
k.Byf
ocusi
ng
ont
hewor
stcasei
nput
,weknow t
hemaxi
mum r
unni
ngt
imef
ort
hepr
ocedur
e.Wi
thout
knowi
ngsomet
hingaboutt
hepossi
blei
nput
stot
hepr
ocedur
e,i
tissaf
estt
obepessi
mist
ic
aboutt
hei
nputandnotassumeanypr
oper
ti
est
hatar
enotknown(
suchast
hatt
hef
ir
st
numberi
nthel
i
sti
sposi
ti
vef hef
ort ir
st-
posexampl
e).
I
nsomecases,weal
soconsi heav
dert eragecasei
nput
.Si
ncemostpr
ocedur
escant
ake
i
nfi
nit
elymanyi
nput
s,t
hisr
equi
resunder
standi
ngt
hedi
str
ibut
ionofpossi
blei
nput
sto
det
ermi
nean“
aver
age”i
nput
.
Thi
sisof
tennecessar
ywhenwear
eanal
yzi
ngt
her
unni
ngt
imeofapr
ocedur
ethatuses
anot
herhel
perpr
ocedur
e.
I
fweuset
hewor
st-
caser
unni
ngt
imef
ort
hehel
perpr
ocedur
e,wewi
l
lgr
ossl
yov
erest
imat
e
t
her
unni
ngt
imeoft
hemai
npr
ocedur
e.I
nst
ead,si
nceweknow how t
hemai
npr
ocedur
e
usest
hehel
perpr
ocedur
e,wecanmor
epr
eci
sel
yest
imat
etheact
ualr
unni
ngt
imeby
consi
der
ingt
heact
ual
input
s.

Consi
der
ingsequent
ialsear
ch,f
ori
nst
ances,whi
chsear
chesf
orat
argetel
ementi
nan
ar
rayAofnel
ement
s.Thus,
thenumberoft
urnst
omakewi
l
lrunf
rom 0t
on–1[
orf
rom 1
t
on]
.

Wecant
henst
atet
hecaseanal
ysi
sasf
oll
ows:

- Basi
coper at
ions: Thecompar i
soni
ntheloop
- Worstcase:ncompar i
sons
- Bestcase: 1compar i
son
- Averagecase: (
n+1) /
2compari
sons
assumingeachelementi
sequal
lyl
i
kelytobesear
ched.

Ifweconsi
dersequenti
alandbinar
ysear
ch,thei
rrespect
ivebest
,wor
st,
andav
eragecases
canbeoutl
inedforonemill
ioni
temsasfol
lows;

Bestcase WorstCase AverageCase

Sequenti
al Fir
sti
tem i
nli
st Lasti
tem i
nli
st Middleofl
ist
Search (onetur
n) (onemil
li
ontur
ns) (500,
000turns)
 Bi
nar
ySear
ch Middl
eofl
i
st(
one Reduceli
stt
oone Hal
fwaythroughthe
t
urn) orl
ess:20“t
urns” r
educt
ion,10turns.

- Not
ethatt
het
abl
eisspeci
fi
ctoal
i
stofonemi
l
li
oni
tems.Nev
ert
hel
ess,t
het
abl
e
suggest
sthatbi
nar
ysear
chi
sthef
ast
eral
gor
it
hm.Butast
het
abl
eisj
ustf
or1mi
l
li
on
r
ecor
ds,
iti
snotqui
tegener
al(
cannotbegener
ali
zed)
.
- Wet
her
efor
econsi
deramor
esui
tabl
erepr
esent
ati
onast
hesi
zeoft
hei
nput
,sayn,
gr
owsv
eryl
arge.Thi
swi
l
lleadust
oamor
egener
ali
zedt
abl
e(r
elat
ion)
.
- Theabov
etabl
eleadsust
othet
askofgener
ali
zat
ionbyconsi
der
ingt
hegr
owt
hof
al
gor
it
hms,
orwhati
sref
err
edt
oasgr
owt
hoff
unct
ionsi
nmostl
i
ter
atur
e.
- Forsi
mpl
i
cit
y,wer
epl
acet
heonemi
l
li
onr
ecor
dsi
nourt
abl
ebyn;andt
het
abl
e
becomes.
Bestcase Wor
stCase Av
erageCase
Sequent
ial
Search Fir
stitem i
nli
st Lastitem i
nlist Middl
eofl i
st:
(oneturn) n“turns” n/2“t
urns”
Binar
ySearch Middleofli
st(one Reducel i
sttoone Half
wayt hr
ough:
log2
turn) orless:l
og2n (n/
2)”tur
ns”.
“t
urns”

Wecaneasi
l
yext
endt
hisanal
ysi
stoi
nser
ti
onsor
tand/
ormer
gesor
t.Wehav
edel
i
ber
atel
y
f
oll
owedt
heabov
eappr
oach(
discussi
on)i
nor
dert
oreducet
hebur
denofmat
hemat
ics
i
nher
enti
ntheappr
oachesf
oranal
ysi
sofal
gor
it
hms.

Thet
imet
akenbyt
heI
NSERTI
ON-
SORTpr
ocedur
edependsont
hei
nput
:Ingener
al,
thet
ime
t
akenbyanal
gor
it
hm gr
owswi
tht
hesi
zeoft
hei
nput
,soi
tist
radi
ti
onalt
odescr
ibet
he
r
unni
ngt
imeofapr
ogr
am asaf
unct
ionoft
hesi
zeofi
tsi
nput
.
Todoso,
weneedt
odef
inet
het
er r
ms“unni
ngt
ime”and“
sizeofi
nput
”mor
ecar
eful
l
y.
Ther
unni
ngt
imeofanal
gor
it
hm onapar
ti
cul
ari
nputi
sthenumberofpr
imi
ti
veoper
ati
ons
or“
steps”execut
ed.
Fort
hemoment
,letusadoptt
hef
oll
owi
ngv
iew:
o Aconst
antamountoft
imei
srequi
redt
oexecut
eeachl
i
neofourpseudocode.
o Onel
i
nemayt
akeadi
ff
erentamountoft
imet
hananot
herl
i
ne,butweshal
lassume
t
hateachexecut
ionoft
hei
thl
i
net
akest
imeci
,wher
eci
isaconst
ant
.
Thi
svi
ewpoi
nti
sinkeepi
ngwi
tht
heRAM model
,andi
tal
sor
efl
ect
show t
hepseudocode
woul
dbei
mpl
ement
edonmostact
ual
comput
ers.

Forourdi
scussi
on,
let
’sconsi
dert
hef
oll
owi
ngi
nser
ti
on-
sor
tal
gor
it
hm:
Consequentl
y ,therunni
ngt i
meoft healgorit
hm isthesum ofr unni
ngt imesforeach
stat
ementexecut ed;
astat
ementt hatt
akescist
epstoexecuteandexecut
esntimeswillcontri
but
eci
*nto
thetotalr
unningtime.
Tocomput eT( n),t
herunni
ngtimeofINSERTION-SORTonaninputofnv alues,wesum t
he
productsofthecostandtimescol
umns,obtai
ning:

Sincet
healgor i
thm’srunni
ngt i
medependsonwhichinputofthatsizeisgiven;
forthe
INSERTION-SORT, thebestcaseoccur
sift
hear
rayisalreadysorted.Foreachj=2, 3..
,n,we
thenfi
ndthatA[i]<=keyinli
ne5wheni hasi
tsi
nit
ial
valueofj -1.Thustj=1f orj=2,3,
..
,n,
andthebest-caseis:

Thuswecanexpr
esst
hisr
unni
ngt
i orconst
meas:an+bf ant
s:aandb,
thatdependont
hecost
sci
;
i
ti husal
st inearf
unct
ionofn.
Notet hati
fthear
rayisinrev
ersedsortedorder
;thati
s,i
ndecr
easi
ngorder
,theworst
-case
result
s.Thisi
sbecausewemustcompar eeachelementA[j
]wi
theachel
ementint
heent i
re
sortedsubarr
ayA[
1..j
-1]
,andsotj=jforj=2,
3,…,n.
Notet hat
:

and
Thus:
Wecanexpr
esst
hiswor
st-
caser
unni
ngt
imeasan2 +bn+C,f
orconst
ant
sa,bandct
hat
agai
ndependsont
hest
atementcost
sci
;i
ti husaquadr
st ati
cfunct
ionofn.

Wor
st-
caseandav
erage-
caseanal
ysi
s
I
nouranal
ysi
sofi
nser
ti
onsor
t,wel
ookedatbot
hthebestcase,
inwhi
cht
hei
nputar
raywas
al
readysor
ted,
andt
hewor
stcase,
inwhi
cht
hei
nputar
raywasr
ever
sesor
ted.
Fort
her
emai
nderoft
hiscour
se,weshal
lconcent
rat
eonf
indi
ngonl
ythewor
st-
case
r
unni
ngt
ime,
thati
s,t
hel
ongestr
unni
ngt
i oranyi
mef nputofsi
zen.
Ther
easonsf
ort
hisor
ient
ati
onar
e:
1.Thewor
st-
caser
unni
ngt
imeofanal
gor
it
hm gi
vesusanupperboundont
her
unni
ng
t
imef
oranyi
nput
;knowi
ngi
tpr
ovi
desaguar
ant
eet
hatt
heal
gor
it
hm wi
l
lnev
ert
ake
anyl
onger
.
(
Sot
her
eisnoneedt
omakesomeeducat
edguessaboutt
her
unni
ngt
ime
andhopet
hati
twi
l
lnev
erget
smuchwor
se)
.
2.Forsomeal
gor
it
hms,t
hewor
stcaseoccur
sfai
rl
yof
ten.Forexampl
e,i
nsear
chi
nga
dat
abasef
orapar
ti
cul
arpi
eceofi
nfor
mat
ion,t
hesear
chi
ngal
gor
it
hm’
swor
stcase
wi
l
lof
tenoccurwhent
hei
nfor
mat
ioni
snotpr
esenti
nthedat
abase.
I
nsomeappl
i
cat
ions,
sear
chesf
orabsenti
nfor
mat
ionar
efr
equent
.
3.The“
aver
agecase”i
sof
tenr
oughl
yasbadast
hewor
stcase.Onav
erage,t
her
efor
e,
wecheckhal
foft
hesubar
rayA[
1,.
.,j
-1]
,andsot
jisaboutj
/2.Ther
esul
ti
ngav
erage-
caser
unni
ngt
imet
urnsoutt
obeaquadr
ati
cfunct
ionoft
hei
nputsi
ze,j
ustl
i
ket
he
wor
st-
caser
unni
ngt
ime.

Det
ermi
ningt
heOr
der(
Rat
e)ofgr
owt
h
Si
nceourgoali
stounder
standhowt
her
unni
ngt
imeofanappl
i
cat
ion’
spr
ocedur
eisr
elat
ed
t
othesi
zeoft
hei
nput
,wewantt
o dev
isea f
unct
ion t
hatt
akesasi
nputa number
r
epr
esent
ingt
hei
nputsi
zeandout
put
sthemaxi
mum numberofst
epsr
equi
redt
ocompl
ete
t
heev
aluat
iononani
nputoft
hatsi
ze.Sy
mbol
i
cal
l
y,wecant
hinkoft
hisf
unct
ionas:
Max-
StepsProc:Number Number
Wher
ePr
oci
sthenameoft
hepr
ocedur
ewear
eanal
yzi
ng.
Becauset
heout
putr
epr
esent
sthemaxi
mum numberofst
epsr
equi
red,
weneedt
oconsi
der
t
hewor
st-
casei
nputoft
hegi
vensi
ze.
Si
nceofal
lthei
ssueswi
thcount
ingst
epsexact
ly,
andt
heuncer
tai
ntyabouthowmuchwor
k
canbedonei
nonest
eponapar
ti
cul
armachi
ne,weusual
l
ycannotdet
ermi
net
heexact
f
unct
i orMax
onf -St
epsProc.
I
nst
ead,wechar
act
eri
ze(
descr
ibe)t
her
unni
ngt
imeofapr
ocedur
ewi
thasetof
f
unct
ionsr
efer
red(
denot
ed)byanasy
mpt
oti
coper
ator
.
Foroper
ator
ssuchas(
O,Ω,andΘ)
,theact
ualt
imeneededf
oreachst
epdoesnot
mat
tersi
ncet
heconst
antf
act
orsar
ehi
ddenbyt
heoper
ator
;whatmat
ter
sishow t
he
numberofst
epsr
equi
red,
growsast
hesi
zeoft
hei
nputgr
ows.

Not
ice t
hat we used some si
mpl
i
fyi
ng abst
ract
ions t
o ease our anal
ysi
s of t
he
I
NSERTI
ONSORTpr
ocedur
e.
o Fi
rst
,wei
gnor
edt
heact
ualcostofeachst
atement
,rat
herusedconst
ant
scit
o
r
epr
esentt
hesecost
s.Then,weobser
vedt
hatev
ent
heseconst
ant
sgi
veusmor
e
det
ailt
hanwer
eal
l
yneed:Wet
husi
gnor
ednotonl
ytheact
ualst
atementcost
s,but
al
sot
heabst
ractcost
sci
.
Anot
hersi
mpl
i
fyi
ngabst
ract
ioni
sther
ateofgr
owt
h(oror
derofgr
owt
h),oft
her
unni
ng
t
ime:
o Weconsi
deronl
ythel
eadi
ngt
erm ofaf
ormul
a(f
unct
ion)(
e. ,an2)
g. ,si
ncet
hel
ower
-
or
dert
ermsar
erel
ati
vel
yinsi
gni
fi
cantf
orl
argev
aluesofn.
o Weal
soi
gnor
ethel
eadi
ngt
erm’
sconst
antcoef
fi
cient
,si
nceconst
antf
act
orsar
eless
si
gni
fi
cantt
hant
her
ateofgr
owt
hindet
ermi
ningcomput
ati
onalef
fi
ciencyf
orl
arge
i
nput
s.
o Fori
nser
ti
onsor
t,whenwei
gnor
ethel
ower
-or
dert
ermsandt
hel
eadi
ngt
erm’
s
const
antcoef
fi
cient
,wear
elef
twi
tht
hef orofn2f
act rom t
hel
eadi
ngt
erm.
o We t
her
efor
e wr
it
ethati
nser
ti
on sor
thas a wor
st-
case r
unni
ng t
ime of
n2)
Θ( (pr
onounced“
thet
aofn-
squar
ed”
).
o Usual
l
yweconsi
deroneal
gor
it
hm t
obemor
eef
fi
cientt
hananot
heri
fit
swor
st-
case
r
unni
ngt
imehasal
oweror
derofgr
owt
h.

Not
ethatforl
argeenoughi
nput
s,aΘn2al
gor
it
hm,
forexampl
e,wi
l
lrunmor
equi
ckl
yint
he
3
wor
stcasethanaΘn algor
it
hm.

Asy
mpt
oti
canal
ysi
sofef
fi
ciencyofal
gor
it
hms

Al
so i
nourear
li
erconsi
der
ati
on(
compar
ati
veanal
ysi
s)ofsequent
ialsear
chandbi
nar
y
sear
ch,wesaw t
hat
,int
uit
ivel
y,wecoul
dcompar
ethei
rper
for
mancesov
erani
nputsi
zeof
onemi
l
li
onr
ecor
ds.Howev
er,wer
emar
kedt
hatsuchsi
mpl
i
sti
ccompar
isonswi
l
lbedi
ff
icul
t
ast
hei
nputsi
ze(
s)gr
ow(
s),
whi
chmakesgener
ali
zat
iondi
ff
icul
ty.

- I
nor
dert
oov
ercomet
heabov
ement
ionedconst
rai
ntsast
hei
nputsi
zesgr
ow,t
he
anal
yti
cal
appr
oacht
ermedasy
mpt
oti
cef
fi
ciencyofal
gor
it
hmsi
smostcommonl
yused.
Asy
mpt
oti
cef
fi
ci
encyofal
gor
it
hmsmakesi
tpossi
blet
osi
mpl
yconsi
deri
nputsi
zesl
arge
enought
omakeonl
ytheor
derofgr
owt
hoft
her
unni
ngt
imer
elev
ant
.

Thus,t
heemphasi
sisonl
yonhowt
her
unni
ngt
imeofanal
gor
it
hm i
ncr
easeswi
tht
he
i
nputsi
ze;
ast
hesi
zeoft
hei
nputi
ncr
easeswi
thoutbound.

- Asymptoti
canaly
sisofeff
ici
encyofal
gori
thms,t
her
efor
e,i
samet
hodofdescr
ibi
ng
l
i
mi t
ingbehavi
ourbyusi
ngasymptot
icnot
ati
ons.

- Stateddi
ff
erent
ly,t
heAsy mptot
icnotat
ions(al
socal
l
edBachmann-
Landaunotati
on)
;
descri
beli
miti
ngbehaviour
soffuncti
onsasargument
softhef
unct
ionsgr
owort endsto
i
nfini
ty.

- Conci
sel
y,t
heasymptot
esprov
ideawayofcompar
ingf
unct
ionsbyi
gnor
ingconst
ant
f
actor
sandsmall
inputsi
zes.

- I
ngeneral
,thebasi
coper
ati
oncountcanbeappr
oxi
mat
edasCg(
n),wher
eg(
n)i
sthe
or
derofgr
owth.

Thef
oll
owi
ngt abl
eshow someoftheappr
oxi
mat
eval
uesofaf
ew f
unct
ionsconsi
der
ed
i
mpor
tantf
oranaly
sisofal
gor
it
hms.

n Log2n N og2n
nl n2 n3 2n n!
10 3.
3 101 3x101
3. 102 103 103 6x106
3.
102 6.
6 102 6x102
6. 104 106 3x1030
1. 6x10157
9.
103 10 103 0x103
1. 106 109
104 13 104 3x104
1. 108 1012
105 17 105 7x105
1. 1010 1015
106 20 106 0x106
2. 1012 1018
Themagni
tudeoft
henumber
sint
heTabl
ehasapr
ofoundsi
gni
fi
cancef
ort
heanal
ysi
sof
al
gor
it
hms.Thef
unct
iongr
owi
ngt
hesl
owestamongt
hesei
sthel
ogar
it
hmi
cfunct
ion.I
t
gr
owssosl
owl
y,i
nfact
,thatweshoul
dexpectapr
ogr
am i
mpl
ement
inganal
gor
it
hm wi
tha
l
ogar
it
hmi
cbasi
c-oper
ati
oncountt
orunpr
act
ical
l
yinst
ant
aneousl
yoni
nput
sofal
lreal
i
sti
c
si
zes.

Al
sonot
ethatal
thoughspeci
fi
cval
uesofsuchacountdepend,
ofcour
se,
ont
hel
ogar
it
hm’
s
base,
thef
ormul
a:

makesi
tpossi
blet
oswi
tchf
rom onebaset
oanot
her
,leav
ingt
hecountl
ogar
it
hmi
cwi
tha
newmul
ti
pli
cat
iveconst
ant
.
Thi
siswhyweomi
tal
ogar
it
hm’
sbaseandwr
it
esi
mpl
ylogni
nsi
tuat
ionswher
ewe
ar
eint
erest
edj
usti
naf
unct
ion’
sor
derofgr
owt
hwi
thi
namul
ti
pli
cat
iveconst
ant
.
Ont
heot
herendoft
het
abl
ear
etheexponent
ial
funct
ion2nandt
hef
act
ori
alf
unct
ionn!
Theset
wof
unct
ionsgr
owsof
astt
hatt
hei
rval
uesbecomeast
ronomi
cal
l
ylar
geev
en
f
orsmal
lv uesofn.(
al Thi
saccount
sforournoti
ncl
udi
ngt
hei
rval orn>=102 i
uesf nthe
Tabl
e.)
Forex
ampl
e,i
twoul
dt 1010 y
akeabout4* ear
sforacomput
erper
for
mingat
ri
ll
i
on
1012)oper
( ati
onspersecondt e2100 oper
oexecut ati
ons.Thought
hisi
sincompar
abl
yfast
er
t
hani
twoul
dhav
etakent
oexecut
e100!Oper
ati
ons,
iti
sst
il
ll
ongert
han4.
5bi
l
lon(
i 4.5*109)
y
ear
s—t
heest
imat
edageoft
hepl
anetEar
th.
Thought
her
eisat
remendousdi
ff
erencebet
weent
heor
der
sofgr
owt
hoft
hef
unct
ions2n
and n!Yetbot
h ar
e of
ten r
efer
red t
o as “
exponent
ial
-gr
owt
hfunct
ions”(
orsi
mpl
y
“
exponent
ial
”)despi
tet
hef
actt
hat
,st
ri
ctl
yspeaki
ng,
onl
ythef
ormershoul
dber
efer
redt
oas
such.Thebot
tom l
i
ne,whi
chi
simpor
tantt
oremember
,ist
his:Al
gor
it
hmst
hatr
equi
rean
exponent
ialnumberofoper
ati
onsar
epr
act
icalf
orsol
vi
ngonl
ypr
obl
emsofv
erysmal
linput
si
zes.
Anot
herwayt
oappr
eci
atet
hequal
i
tat
ivedi
ff
erenceamongt
heor
der
sofgr
owt
hoft
he
f
unct
ionsi
ntheTabl
eist
oconsi
derhowt
heyr
eactt
o,say
,at
wof
oldi
ncr
easei
nthev
alueof
t
hei gumentn.
rar
Thef
unct
i og2ni
onl ncr
easesi
nval
uebyj
ust1(
becausel
og2 2n=l og2n=1+
og2 2+l
og2 n)
l ;thel
i
nearf
unct
ioni
ncr
easest
wof
old,t
hel
i
near
it
hmi
cfunct
ionnl
og2 ni
ncr
eases
sl
i
ght
lymor
ethant
wof
old,whi
l
ethequadr
ati
cfunct
ionn2 andcubi
cfunct
ionn3 i
ncr
ease
2
=4n2and( 3
f
our
fol
dandei
ght
fol
d,r
espect
ivel
y(because(
2n) 2n)=8n3)

The contentoft he abovetabl

e aretypi
calvalue ofthe mat
hemat
icalexpressi
ons
(f
uncti
ons)writt
enint ermsofn( i
.e.g(
n))(whichassisti
nknowingwhichexpressi
on
gi
veshighestval
ues).

Recapi
tul
ati
onoft
heAnal
ysi
sFr
amewor
k
Bef
orewel
eav
ethi
ssect
ion,
letussummar
izet
hemai
npoi
ntsoft
hef
ramewor
kout
li
nedabov
e.
1.Bot
hti
meandspaceef
fi
cienci
esar
emeasur
edasf
unct
ionsoft
heal
gor
it
hm’
sinputsi
ze.
2.Ti
meef
fi
ciencyi
smeasur
edbycount
ingt
henumberoft
imest
heal
gor
it
hm’
sbasi
coper
ati
on
i
sexecut
ed.Spaceef
fi
ciencyi
smeasur
edbycount
ingt
henumberofext
ramemor
yuni
ts
consumedbyt
heal
gor
it
hm.
3.Theef
fi
cienci
esofsomeal
gor
it
hmsmaydi
ff
ersi
gni
fi
cant
lyf
ori
nput
soft
hesamesi
ze.For
suchal
gor
it
hms,
weneedt
odi
sti
ngui
shbet
weent
hewor
st-
case,
aver
age-
case,
andbest
-case
ef
fi
cienci
es.
4.Thef
ramewor
k’spr
imar
yint
erestl
i
esi
ntheor
derofgr
owt
hoft
heal
gor
it
hm’
srunni
ngt
ime
andext
ramemor
yuni
tsconsumedasi
tsi
nputsi
zegoest
oinf
ini
ty.
Asy
mpt
oti
cNot
ati
onsandBasi
cEf
fi
ciencyCl
asses
Asy
mpt
oti
cAnal
ysi
s

Theasy
mpt
oti
cbehav
iourofaf
unct
ionf
(n)r
efer
stot
hegr
owt
hoff
(n)asnget
slar
ge.
Ty
pical
l
ywei
gnor
esmal
lval
uesofn,
sincewear
emor
eof
teni
nter
est
edi
nest
imat
ing
howsl
owt
hepr
ogr
am r
unbeonl
argei
nput
s.
A goodr
uleoft
humbi
sthatt
hesl
owert
heasy
mpt
oti
cgr
owt
hrat
e,t
hebet
tert
he
al
gor
it
hm,
thoughi
t’
snotal
way
str
ue.
Forexampl
e,a l
i
nearal
gor
it
hm f
(n)
=d*
n+k i
s al
way
s asy
mpt
oti
cal
l
y bet
tert
han a
quadr
ati
cone,
f( n2+q.
n)=c*

Aspoi
ntedouti
nthepr
evi
oussect
ion,
theef
fi
ciencyanal
ysi
sfr
amewor
kconcent
rat
esont
he
or
derofgr
owt
h ofan al
gor
it
hm’
sbasi
coper
ati
on countast
hemai
nindi
cat
oroft
he
al
gor
it
hm’
sef
fi
ciency
.
Tocompar
eandr
anksuchor
der
sofgr
owt
h,comput
ersci
ent
ist
suset
hreenot
ati
ons:
O(
bigoh)
,_(
bigomega)
,and_(
bigt
het
a).

Wei
ntr
oducet
heset
hreef
unct
ionsempl
oyt
ocapt
uret
hei
mpor
tantpr
oper
ti
esofhow
r
equi
redr
esour
cesgr
owwi
thi
nputsi
ze.
Eachf
unct
iont
akesasi
nputaf
unct
ion,
andpr
oducesasout
putasetoff
unct
ions:
O(f)(
“bi
goh”
)-Thesetoff
unct
ionst ownof
hatgr ast
ert
hanfgr
ows(
<=)
.
Θ(f)(
thet
a)-Thesetoff
unct
ionst owasf
hatgr astasfgr
ows(
=).
Ω(f)(
omega)-Thesetoff
unct
ionst ownosl
hatgr owert
hanfgr
ows(
>=)
.
Thesef
unct
ionscapt
uret
heasy
mpt
oti
cbehav
iouroff
unct
ions,ast
hei
nput
sgetar
bit
rar
il
y
l
arge.
Thef
igur
ebel
owdepi
ctst
heset
sO,
ΘandΩf
orsomef
unct
ionf
.

Fi
gur
e:Vi
sual
i
zat
ionoft
heset
sO(f)
,Ω(f)
,andΘ(f)
.(Sour
ce:
int
roduct
iont
ocomput
ing)
Let
snowdef
ineeachf
unct
ionandpr
ovi
desomeexampl
es.

(
1) TheO–not
ati
on

TheO–not
ati
on(
pronouncedbi
gOh)-t
akesasi
nputaf
unct
ion,
andpr
oducesasout
putt
he
setofal
lfunct
ionst
hatgr
ownof
ast
ert
hant
hei
nputf
unct
ion.ThesetO(
f)i
sthesetofal
l
f
unct
ionst
hatgr
owasf
astas,
orsl
owert
han,
fgr
ows.
I
nFi
gur
e,t
heO(
f)seti
srepr
esent
edbyev
ery
thi
ngi
nsi
det
heout
erci
rcl
e.
- I
tis used when we hav
e onl
y an asy
mpt
oti
c upperbound.i
tis wr
it
ten:as
expr
O( essi
on)
, eexpr
wher essi
oni
sint
ermsofn.

Todef
inet
hemeani
ngofOpr
eci
sel
y,weneedt
oconsi
derwhati
tmeansf
oraf
unct
iont
o
gr
ow.Wewantt
ocapt
urehow t
heout
putoft
hef
unct
ioni
ncr
easesast
hei
nputt
othe
f
unct
ioni
ncr
eases.Fi
rst
,weconsi
deraf
ewexampl
es;t
henwepr
ovi
deaf
ormaldef
ini
ti
onof
O.
Let
sconsi
dert
hef
oll
owi
ng:
1. f
(n)=n+12andg(
n)=n-7
Nomat
terwhatnv
alueweuse,
thev
alueoff
(n)i
sgr
eat
ert
hant
hev
alueofg(
n).
Thought
hisdoesn’
tmat
terf
ort
hegr
owt
hrat
es.Whatmat
ter
sishowt
hedi
ff
erence
bet
weeng(
n)andf(
n)changesast
hei
nputv
aluesi
ncr
ease.
Nomat
terwhatv
alueswechoosef
orn,weknow g(
n)-f
(n)=-
19.Thus,t
hegr
owt
h
r
atesoffandgar
eident
icalandn-7i
sint
hesetO(
n+12)
,andn+12i
sint
heset
O(
n-7)
.
2. f(
n)=2nandg(
n)=3n
Thedi
ff
erencebet
weeng(
n)andf
(n)i
sn.Thi
sdi
ff
erencei
ncr
easesast
hei
nputv
alue
ni
ncr
eases,
buti
tincr
easesbyt
hesameamountasni
ncr
eases.
So,
thegr
owt
hrat
easni
ncr
easesi
sn/
n=1.Thev
alueof2ni
sal
way
swi
thi
na
const
antmul
ti
pleof3n,
sot
heygr
owasy
mpt
oti
cal
l
yatt
hesamer
ate.
Hence,
2ni
sint
hesetO(
3n)and3ni
sint
hesetO(
2n)
.
3. f
( n)=n2
n)=nandg(
Thedi
ff
erencebet
weeng(
n)andf
(n)i
sn2-
n=n(
n-1)
.Thegr
owt
hrat
easni
ncr
eases
i
sn(
n-1)
/n=n-1.Thev
alueofn-1i
ncr
easesasni
ncr
eases,
soggr
owsf
ast
ert
hanf
.
smeansn2i
Thi snoti
nO( ncen2gr
n)si owsf
ast
ert
hann.
Thef
unct
ionni
si n2)si
nO( ncengr
owssl
owert
hann2gr
ows.
4. f
(n)=Fi
bonacci
(n)andg(
n)=n
 TheFi
bonaccif
unct
iongr
owsv
eryr
api
dly
.Thev
alueofFi
bonacci
(n+2)i
smor
ethan
doubl
ethev
alueofFi
bonacci
(n)si
nce:
Fi
bonacci
(n+2)=Fi
bonacci
(n+1)+Fi
bonacci
(n)
,andFi
bonacci
(n+1)>Fi
bonacci
(n)
.
Ther
ateofi
ncr
easei
smul
ti
pli
cat
ive,
andmustbeatl
eastaf
act
orof√2≈1.
414(
since
i
ncr
easi
ngbyonet
wicemor
ethandoubl
est
hev
alue)
.(I
nfact
,ther
ateofi
ncr
easei
sa
f orofϕ=(
act 1+√5)
/2≈1.
618,
alsoknownast
he“
gol
denr
ati
o”.
Thi
sismuchf
ast
ert
hant
hegr
owt
hrat
eofn,
whi
chi
ncr
easesbyonewhenwe
i
ncr
easenbyone.So,
nisi
nt Fi
hesetO( bonacci
(n)
),butFi
bonacci
(n)i
snoti
ntheset
O(
n).

Thef
ormal
def
ini
ti
onofO–not
ati
on

Thefunct
iongisamemberofthesetO(
f)i
fandonl
yifther
eexi
stposi
ti
veconst
ant
scandn0
suchthat
: f
oral
lval
uesn>=n0,g(
n)<=cf(n).
Or
Foragivenfunct
ionwi
thv
alueg(n)
,O(
g(n)
)={f(n):t
her
eexi
stposi
ti
veconst
ant
sCandn0
suchthat
: 0≤f (
n)≤Cg(
n)foral
ln≥n0}
Or
Af
uncti
onf (
n)i
ssai
dt obeinO( g(
n)),
denotedf(n)∈O(g(
n))
,iff
(n)i
sboundedabovebysome
constantmul
ti
pleofg(n)foral
llar
gen, i
.e.
,i
fthereexi
stsomeposit
iveconst
antcand
somenonnegativei
ntegern0sucht hat
: f (
n)≤cg(n)foral
ln≥n0.

Thi
sisi
l
lust
rat
edgr
aphi
cal
l
yas:

OR
Not
ethatdif
fer
entf
unct
ionswi
tht
hesamegr
owt
hrat
emayber
epr
esent
edusi
ngt
hesame
O–notat
ion.
O-not
ationisf
requent
lyusedt odescr
ibehowclosel
yafini
teser
iesappr
oxi
matesagiven
funct
ion,byomitt
ingconstantfact
orsandlowerordert
erms.TheO-not
ati
onspeci
fi
call
y
descri
besworst-
casescenario.
Let
’snowshowt
hatt
hecl
aimedpr
oper
ti
esar
etr
ueusi
ngt
hef
ormal
def
ini
ti
on.
1. n-7i
sinO(
n+12)
Choosec=1andn0=1.Then,
weneedt
oshown-7<=1(
n+12)f
oral
lval
uesn>=1.Thi
sis
t
rue,
sincen-7<n+12f
oral
lval
uesn.
2. n+12i
sinO(
n-7)
Choosec=2andn0=26.Then,
weneedt
oshown+12<=2(
n-7)f
oral
lval
uesn>=26.The
equat
ionsi
mpl
i
fiest
on+12<=2n-
14,
whi
chsi
mpl
i
fiest
o26<=n.Thi
sist
ri
vi
all
ytr
uef
oral
l
v
aluesn>=26.
3. 2ni
sinO(
3n)
Choosec=1andn0=1.Then,
2n<=3nf
oral
lval
uesn>=1.
4. 3ni
sinO(
2n)
Choosec=2andn0=1.Then,
3n<=2(
2n)si
mpl
i
fiest
on<=4n/
3whi
chi
str
uef
oral
lval
uesn
>=1.
5. n2i
snoti
nO(
n)
Weneedt
oshowt
hatnomat
terwhatv
aluesar
echosenf
orcandn0,
ther
ear
eval
uesofn>=
n0sucht
hatt
hei
nequal
i
tyn2>cndoesnothol
d.Foranyv
al wecanmaken2>cnby
ueofc,
choosi
ng n>c.
6. ni
si Fi
nO( bonacci
(n)
)
Choosec=1andn0=3.Thenn<=Fi
bonacci
(n)f
oral
lval
uesn>=n0.

Ot
herAppl
i
cat
ionofO-
not
ati
on:

(
1) Usi
ngO-
not
ati
ont
osi
mpl
i
fyf
unct
ions:
 Consi
der
:fx)=6x4–2x3+5
(
- Recallthatasymptot
icnotat
ionignoressmal linputv
aluesandconst antfactors.
Oft heter
msi nf (
x)above,theonewi t
ht hehighestgrowthrateistheonewi t
ht he
4 4
l
argestexponentasaf uncti
onofx,namel y6x,whi chistheproductof6andx.The
fi
rstfact
ordoesnotdependonx.t hi
sleavesuswi ththesi
mpl i
fi
edform asx4.

- Thus,
f( x4)(
x)=O( i.
e.f
(x)i
sbi x4)
gOof( )

- Appl
yi
ngformaldefi
nit
ionoft
heO-
not
ati
on,t
hest
atementt
hatf
( x4)i
x)=O( s
equi
val
entt
oit
sexpressi
on;

f
(x)
≤mg(
x)f
orsomesui
tabl
echoi
cesofx
0andm f
oral
lx>x0

[
Thebur
denofpr
oofandappl
i
cat
ioni
sonst
udent
s].

Exer
cise1:

Foreachoft
hegf
unct
ionsbel
ow,
answerwhet
herornotgi
sint
he

a.g(
n)=n+5
b.g(
n)=0.
1n
c.g(
n)=150n+√n
n)=n15
d.g(
Supposeanal
gori
thm isbeingdevel
opedtooperat
eonasetofn-
element
s,whereiti
s
r
equir
edtodetermi
neti
meT( n)fort
healgor
it
hm torunint
ermsoft
henumberofelements
i
ntheinputsi
ze.
Thealgor
it
hm wor
ksbyf ir
stcall
i
ngasubr out
inetosor
tel
ementsinthesetandthen
per
formsitsownoper
ations.
Thesortpr
ocedurehasaknownti
mecomplexi
t n2)
yofO( ,andaft
ert
hesubr
out
iner
uns,
t
heal
gori
thm musttakeanaddi
ti 55n3+2n+10t
onal imebef
oreitt
ermi
nat
es.
Thust
heov
eral
lti
mecompl
exi
tyoft
heal
gor
it
hmsi
sT( n2)+55n3+2n+10.
n)=O(
Usi
ngt
heO-
not
ati
on,
simpl
i
fyT(
n).

TheΩ–not
ati
on
TheΩ–not
ati
on(
pronounced“
big-
omegaofgofn”
)pr
ovi
desanasy
mpt
oti
clowerbound,
j
ustast
heO-not
ati
onpr
ovi
desanasy
mpt
oti
cupperboundonaf
unct
ion.

Foragi
venf
unct
iong(
n),
theΩ–not
ati
oni
susual
l
ydenot
edas:
Ω–not
ati
on(
g(n)
),and
f
ormal
l
ydef
inedas:
Ω(
g(n))={f
(n)
:Ther
eexi
stposi
ti
veconst
antcandn0sucht
hat0≤cg(
n)≤f
(n)f
oral
ln≥n0}
Or
Thefuncti
ongisamemberoft hesetΩ(f)ifandonlyi
fther
eexistposi
ti
veconstantscand
n0suchthat,
foral
lval
uesn>=n0,
g(
n)>=cf(n).
or
Afunctont(
i n)issai
dtobeinΩ(g(n)
), edt(
denot n)∈Ω(g(n)
),ft(
i n)isboundedbelowby
someposi
ti
veconst
antmul
ti
pleofg(
n)f
oral
ll
argen,
i.
e.,
ift
her
eexi
stsomeposi
ti
ve
antcandsomenonnegat
const ivei
ntegern0sucht
hat
t
(n)≥cg(
n)f
oral
ln≥n0.
So,
afunct
iongi
sinΩ(
f)i
fggr
owsasf
astasforf
ast
er.Cont
rastt
hiswi
thO(f)
,thesetof
al
lfunct
ionst ownof
hatgr ast
ert
hanfgr
ows.
I
nFi
gur
eabov
e,Ω(f)i
sthesetofal
lfunct
ionsout
sidet
hedar
kerci
rcl
e.
Thef
ormal
def
ini
ti
onofΩ(
f)i
snear
lyi
dent
ical
tot
hedef
ini
ti
onofO(f)
:theonl
ydi
ff
erencei
s
t
he<=compar
isoni
schangedt
o>=.

Thi
scanbei
l
lust
rat
edgr
aphi
cal
l
yas:

OR
- From t
heabov
efi
gur
e,f
oral
lval
uenatort
other
ightofn0t
hev
alueoff
(n)i
sonor
abovecg(
n).
Exampl
e
(
1) Consi
deranal
gor
it
hm wi
tht
imecompl
exi
t n)=an2+bn+c,
yT( i n2)
sΩ( .
Thi
simpl
i
est
hatt
her
eexi
stsaconst hatC1n2≤an2+bn+c.
antC1>0sucht
Topr
ovet
his:
LetC1=a/
2oranynegat
ivev
alue,
weonl
yneedt
oshowt
hata/
2(n2)+bn+c≥0f
or
al
la>0.
NB Asy
mpt
ote–Al
i
net
hatt
endst
owar
dsacur
vewi
thoutmeet
ingi
tatanyf
ini
te
t
ime.
Exampl
es:
Wer
epeatsel
ect
edexampl
esf
rom t
hepr
evi
oussect
ionwi
thΩi
nst
eadofO.Thest
rat
egyi
s
si
mil
ar:
weshowgi
sinΩ(f)usi
ngt
hedef
ini
ti
onofΩ(f)bychoosi
ngposi
ti
veconst
ant
sfor
t
hev
aluesofcandn0,
andshowi
ngt
hatt
hepr
oper
tyg(
n)≥cf(
n)hol
dsf
oral
lval
uesn≥n0.
Toshowgi
snoti
nΩ(f)
,weneedt
oexpl
ainhow,
foranychoi
cesofcandn0,
wecan
f
indachoi
cef
orn≥n0sucht
hatg(
n)<cf(
n).

1. n-7i
sinΩ(
n+12)
Choosec=1/
2andn0=26.Then,
weneedt
oshown-7≥1/
2(n+12)f
oral
lval
uesn
≥26.Thi
sist
rue,
sincet
hei
nequal
i
tysi
mpl
i
fiesn/
2≥13whi
chhol
dsf
oral
lval
uesn≥
26.
2. 2ni
sinΩ(
3n)
Choosec=1/
3andn0=1.Then,
2n≥1/
3(3n)si
mpl
i
fiest
on≥0whi
chhol
dsf
oral
lval
ues
n≥1.
3. ni
snoti n2)
nΩ(
What
everv
aluesar
echosenf
orcandn0, hatn≥cn2does
wecanchoosen≥n0sucht
nothol
d.
Choosen>1/
c(not
ethatcmustbel
esst
han1f
ort
hei
nequal
i
tyt
ohol
dforany
posi
ti
ven,
soi
fci
snotl
esst
han1wecanj
ustchoosen≥2)
.Then,
ther
ightsi
deoft
he
i
nequal
i
tycn2wi
l
lbegr
eat
ert
hann,
andt
heneededi
nequal
i
tyn≥cn2doesnothol
d.

Exer
cise2:RepeatExer
cise1usi
ngΩi
nst
eadofO.

Exer
cise3:Foreachpart,i
denti
fyafuncti
ongthatsat
isf
iest
hepr
oper
ty.
a.gisinO(n2)butnoti
nΩ( n2)
.
b.gisnoti
nO( n2)buti
sinΩ( n2)
.

TheΘ–not
ati
on

Thef
unct
ionΘ(f)denot
est
hesetoff
unct
ionst
hatgr
owatt
hesamer
ateasf.
I
tist
hei
nter
sect
ionoft
heset
sO(f)andΩ(f)
.Hence,
afunct
iongi
sinΘ(f)i
fandonl
yifg
i
sinO(f)andgi
sinΩ(f)
.InFi
gur
ebel
ow,Θ(f)i
sther
ingbet
weent
heout
erandi
nner
ci
rcl
es.

St
ateddi
ff
erent
ly:

TheΘ(
f(n)
)orΘ(
g(n)
)ist
hesetofal
lfunct
ionswi
tht
hesameor
derofgr
owt
hasf
(n)org(
n),
ast
hecasemaybe.

Def
ini
ti
ons:
a. Foragi
venf
unct
iong(
n),
denot
ebyΘ(
g(n)
)ist
hesetoff
unct
ions:
Θ(
g(n)
)={
f(n)
:ift
her
eexi
stposi
ti
veconst
ant
sC1,
C2,
andn0sucht
hat
0≤C1g(
n)≤f
(n)C2g(
n)f
oral
ln≥n0}
b. Thef
unct
iongi
samemberoft
hesetΘ(f)i
fanyonl
yift
her
eexi
stposi
ti
veconst
ant
s
c1,
c2,
andn0sucht
hat
,foral
lval
uesn≥n0,
c1f(
n)≥g(
n)≥c2f(
n).

Thi
simpl
i
est
hatt
hef
unct
ionf
(n)bel
ongst
othesetΘ(
g(n)
)ift
her
eexi
stposi
ti
veconst
ant
s
C1andC2sucht
hati
tcanbe“
sandwi
ched”bet
weenC1g(
n)andC2g(
n),
forsuf
fi
cient
lyl
argen.

BecauseΘ(
g(n)
)isaset
,weshoul
dwr
it
e“f(
n)εΘ(
g(n)
)”t
oindi
cat
ethatf(
n)i
samemberof
Θ(
g(n)
).I
nst
ead,
wewi
l
lusual
l
ywr
it
e:
“
f(n)=Θ(
g(n)
)”t
oexpr
esst
hesamenot
ion.

Θ-
not
ati
oni
sil
l
ust
rat
edgr
aphi
cal
l
yas:

OR
Wei
ntr
oducedear
li
erani
nfor
malnot
ionofΘ-
not
ati
ont
hatamount
edt
othr
owi
ngawayl
ower
-
ordert
ermsandi
gnor
ingt
hel
eadi
ngcoef
fi
cientoft
hehi
ghest
-or
dert
erm.

Letusbr
ief
lyj
ust
if
ythi
sbyusi
ngt
hef
ormal
def
ini
ti
ont
oshowt
hat
Todoso,
wemustdet
ermi
neposi
ti
veconst
ant
sc1,
c2,
andn0sucht
hat
:
Not
e: T(
n)=Θ(
f(n)
)isequi
val
entt
ost
ati
ngt
hatT(
n)=Ω(
f(n)
)butnott
heconv
erse.
- I
fT(
n)=Ω(
f(n)
)andT(
n)=O(
f(n)
).
ThenT(
n)=Θ(
f(n)
).
 Onuspr
oofandexampl
eisonst
udent
s.
- TheΘ-notati
onall
owsusimmediatel
y(r
apidl
y)t
odet
ermi
net
heal
gor
it
hm t
hatcanr
un
f
aster
,allt
hingsbei
ngequal(
cet
erispari
bus).

General
ly
,asy mptoti
cnot at
ionsaremachine-i
ndependent(i.
e.actualmachine,compil
er,
progr
ammi nglanguage)notati
onf
orrunni
ngtimes,andperhapsothereff
ici
encypar
ameters,
ofanalgori
thm.

I
ntui
ti
vely
,thel
ower-
orderter
msofanasy mptot
ical
l
yposit
ivefuncti
oncanbei gnor
edi
n
det
ermini
ngasy
mptoti
call
yti
ghtboundsbecauset
heyarei
nsi
gnif
icantforl
argen.

Whenni slar
ge,evenatinyf
ract
ionofthehighest
-ordert
erm suff
icestodomi
natethel
ower-
orderterms.Thus,setti
ngc1t oav al
uet hatissli
ghtl
ysmal l
erthanthecoef
fi
cientofthe
highest
-orderter
m andsetti
ngc2t oav al
uethatissli
ghtl
ylarger
,permit
sthei
nequali
ti
esin
thedefini
ti
onofΘ- not
ati
ontobesatisf
ied.

Thecoeffi
cientofthehighest
-or
derterm canli
kewi
sebei
gnor
ed,
sincei
tonl
ychangesc1
andc2byaconst antf
actorequalt
ot hecoeff
ici
ent
.
Examples:
Exer
cises3:

a.RepeatEx
erci
se1usi
ngΘi
nst
eadofO.
b.Showt n2–n)i
hatΘ( sequi
val
entt n2)
oΘ(
c.i
sΘ(
2n)equi
val
entt
oΘ(
3n)
?Ei
therpr
ovet
heyi
dent
ical
,orpr
ovet
heyar
edi
ff
erent
.