1

Chapter 1. Foundations

We use omputer algorithms to solve problems, e.g., to ompute the maximum of a

set of real numbers or to ompute the produ t of two integers. A problem P onsists
of innitely many problem instan es. An instan e of the maximum problem is e.g.,
to ompute the maximum of the following ve numbers 2; 7; 3; 9; 8. An instan e of
the multipli ation problem is, e.g., to ompute the produ t of 257 and 123. We
asso iate with every problem instan e p 2 P a natural number g(p), its size.
Sometimes, the size will be a tuple of natural numbers; e.g., we measure the size of
a graph by a pair onsisting of the number of nodes and the number of edges. In the
maximum problem we an dene the size as the ardinality of the input set (5 in
our example), in the multipli ation problem we an dene the size as the sum of the
lengths of the de imal representations of the fa tors (6 in our example). Although
the denition of size is arbitrary, there is usually a natural hoi e.
Exe ution of a program on a ma hine requires resour es, e.g., time and spa e.
Resour e requirements depend on the input. We use T (p) to denote the running
A

time of algorithm A on problem instan e p. We an determine T (p) by experiment

A

and measure it in millise onds.

Global information about the resour e requirements of an algorithm is in gen-
eral more expressive than information about resour e requirements on parti ular
instan es. Global information su h as maximal running time on an input of size n
annot be determined by experiment. Two abstra tions are generally used: worst
ase and average ase behavior.
Worst ase behavior is the maximal running time on any input of a parti ular
size. We use T (n) to denote the worst ase running time (or simply running time)
A

of algorithm A on an input of size n, i.e.,

T (n) = supfT (p); p 2 P
A A and g(p) = ng:
Worst ase behavior onsiders algorithms from a pessimisti point of view. For
every n we single out the input with maximal running time.
Sometimes, we are given a probability distribution on the set of problem in-
stan es. We an then talk about average ase behavior (or expe ted behavior);
it is dened as the expe tation of the running time for problems of a parti ular size,
i.e.,

T (n) = E fT (p); p 2 P and g(p) = ng :
av
A A

In this book (Chapters 2 and 3), omputing expe tations is always redu ed to
omputing nite sums. Of ourse, average ase running time is never larger than
worst ase running time and sometimes mu h smaller. However, an average ase
analysis always poses the following question: does the a tual use of the algorithm
onform to the probability distribution on whi h our analysis is based?
We an now formulate one goal of this book. Determine T (n) for important
A

algorithms A. More generally, develop methods for determining T (n). Unfortu-

A

nately, this goal is beyond our rea h for many algorithms at the moment. We have
Version: 18.10.99 Time: 17:53 {1{
2
to onne ourselves to determine upper and lower bounds for T (n), i.e., to asymp-
A

toti analysis. A typi al laim will be: T (n) is bounded above by some quadrati
fun tion. We write T (n) = O(n ) whi h means that T (n) 
n for onstants
2 2

> 0 and n and all n  n . Or we laim that T (n) grows at least as fast as n log n.
0 0
We write T (n) =
(n log n) whi h means that there are onstants > 0 and n 0
su h that T (n) 
n log n for all n  n (log denotes log to base two throughout
0
this book). We ome ba k to this notation in Se tion 1.6.
We an also ompare two algorithms A and A for the same problem. We say
1 2
that A is faster than A if T (n)  T (n) for all n and that A is asymptot-
1 2 A1 A2 1
i ally faster than A if lim !1 T (n)=T (n) = 0. Of ourse, if A is asymp-
2 n A1 A2 1
toti ally faster than A then A may still be more eÆ ient than A on instan es of
2 2 1
small size. This trivial observation is worth being exemplied.
Let us assume that we have 4 algorithms A; B; C; D for solving problem P with
running times T (n) = 1000n, T (n) = 200n log n, T (n) = 10n and T (n) = 2
A B C
2
D
n

millise onds. Then D is fastest for 1  n  9, C is fastest for 10  n  100 and

A is fastest for n  101. Algorithm B is never the most eÆ ient. How large is the
maximal problem instan e whi h we an solve in one hour of omputing time? The
answer is 3600 (1600, 600, 21) for algorithm A (B; C; D). If the maximal solvable
problem size is too small we an do either one of two things. Buy a larger ma hine
or swit h to a more eÆ ient algorithm. Assume rst that we buy a ma hine whi h
is ten times as fast as the present one, or alternatively that we are willing to spend
10 hours of omputing time. Then the maximal solvable problem size in reases to
36000 (13500, 1900, 25) for algorithms A (B; C; D). We infer from this example
that buying a faster ma hine hardly helps if we use a very ineÆ ient algorithm
(algorithm D) and that swit hing to a faster algorithm has a more drasti ee t on
the maximally solvable problem size. More generally, we infer from this example
that asymptoti analysis is a useful on ept and that spe ial onsiderations are
required for small instan es ( f. Se tions 2.1.5 and 5.4).
So far, we dis ussed the omplexity of algorithms, sometimes, we will also
talk about the omplexity of problems. An upper bound on the omplexity of a
problem is established by devising and analyzing an algorithm; i.e., a problem P has
omplexity O(n ) if there is an algorithm for P whose running time is bounded by
2

a quadrati fun tion. Lower bounds are more diÆ ult to obtain. A problem P has
omplexity
(n ) if every algorithm for P has running time at least
(n ). Lower
2 2

bound proofs require the dis ussion of an entire lass of algorithms and are usually
very diÆ ult to obtain. Lower bounds are only available in very rare ir umstan es
( f. Se tions 2.1.6, 2.3, 3.4 and 5.7).
We will next dene running time and storage spa e in pre ise terms. To do so
we have to introdu e a ma hine model. We want this ma hine model to abstra t
the most important features of existing omputers, so as to make our analysis
meaningful for every-day omputing, and to make it simple enough, to make analysis
possible.

Version: 18.10.99 Time: 17:53 {2{

 1.1. Ma hine Models: RAM and RASP 3
1.1. Ma hine Models: RAM and RASP

A random a ess ma hine (RAM) onsists of 4 registers, the a umulator 

and the index registers , , (the hoi e of three index registers is arbitrary),
1 2 3
and an innite set of storage lo ations numbered 0; 1; 2; : : :, f. Figure 1.

 a umulator 0
1 index register 1 1
2 index register 2 2
3 index register 3 3
.. ..
. .
Figure 1. A RAM
The instru tion set of a RAM onsists of the following list of one address
instru tions. We use reg to denote an arbitrary register , , , , i to denote a
1 2 3
non-negative integer, op to denote an operand of the form i, (i) or reg, and mop
to denote a modied operand of the form (i + ). In applied position operand
j

i evaluates to number i, (i) evaluates to the ontent of lo ation i, reg evaluates

to the ontent of reg and (i + ) evaluates to the ontent of lo ation numbered
j

(i + ontent of ). Modied operands are the only means of address al ulation in

j

RAMs. We dis uss other possibilities at the end of this se tion. The instru tion set
onsists of four groups:
Load and store instru tions:
reg op , e.g., 1 (2),
 mop , e.g.,  ( 2 + 3),
op reg , e.g., (3) ,
mop  , e.g., ( 2 + 3) .
Jump instru tions:
goto k , k 2 N0 ,
if reg  0 then goto k , k 2 N0 ,
where  2 f=,6=,<,,>,g is a omparison operator.
Arithmeti instru tions:
   op,    mop,
where  2 f+, ,,div,modg is an arithmeti operator.
Version: 18.10.99 Time: 17:53 {3{
4
Indexregister instru tions:
j i j , 1j3 , i2N :
0

A RAM program is a sequen e of instru tions numbered 0; 1; 2; : : : . Integer k in

jump instru tions refers to this numbering. Flow of ontrol runs through the pro-
gram a ording to this numbering ex ept for jump instru tions.
Example: Program 1 shows a RAM program for omputing 2 . We assume that
n

n is initially stored in lo ation 0. The output is stored in lo ation 1. The right

olumn shows the number of exe utions of ea h instru tion on input n.

0: 1 (0) 1
1:  1 1
2: if 1 = 0 then goto 6 n+1
3:  2 n
4: 1 1 1 n
5: goto 2 n
6: (1)  1
Program 1

In our RAMs there is only one data type: integer. It is straightforward to

extend RAMs to other data types su h as boolean and reals; but no additional
insight is gained by doing so. Registers and lo ations an store arbitrary integers,
an unrealisti assumption. We balan e this unrealisti assumption by a areful
denition of exe ution time. Exe ution time of an instru tion onsists of two parts:
storage a ess time and exe ution time of the instru tion proper. We distinguish
two ost measures: unit ost and logarithmi ost.
In the unit ost measure we abstra t from the size of the operands and
harge one time unit for ea h storage a ess and instru tion exe ution. The unit
ost measure is reasonable whenever algorithms use only numbers whi h t into
single lo ations of real omputers. All algorithms in this book (ex ept Chapter 6)
are of this kind for pra ti al problem sizes and we will therefore always use the unit
ost measure outside Chapter 6. However, the reader should be warned. Whenever
he analyzes an algorithm in the unit ost measure, he should give areful thought
to the size of the operands involved.
In the logarithmi ost measure we expli itely a ount for the size of the
operands and al ulate the osts a ording to their length L. If binary representa-
tion is used then 
1
L(n) = blog if n = 0;
n + 1 otherwise.
This explains the name \logarithmi ost measure". The logarithmi ost measure
has to be used if the numbers involved do not t into single storage lo ations
Version: 18.10.99 Time: 17:53 {4{
 1.1. Ma hine Models: RAM and RASP 5
anymore. In the following table we use m to denote the number moved in load
and store instru tions, and m and m to denote the numbers operated on in an
1 2
arithmeti instru tion. The meaning of all other quantities is obvious from the
instru tion format.
Costs for Storage A ess:
Operand Unit Cost Logarithmi Cost
i 0 0
reg 0 0
(i) 1 L(i)
(i + j ) 1 L(i) + L( )j

Cost for Exe uting the Instru tion Proper:

Unit Cost Logarithmi Cost
Load and Stores 1 1 + L(m)
Jumps 1 1 + L(k)
Arithmeti 1 1 + L(m ) + L(m ) 1 2
Index 1 1 + L( ) + L(i) j

The ost of a onditional jump if reg  0 then goto k is independent of the ontent
of reg be ause all omparison operators require only that one he ks some few bits
of the binary representation of reg.
Under the unit ost measure the ost of an instru tion is 1 + # of storage a -
esses (we use the symbol # with the meaning of \number"), under the logarithmi
ost measure it is 1+ sum of the lengths of the addresses and numbers involved.
Thus the exe ution time of an instru tion is independent of the data in the unit
ost measure, but it depends on the data in the logarithmi ost measure.
Example ( ontinued): The instru tions of program 1 have the following osts:
Instru tion Unit Cost Logarithmi Cost
0 2 L(0) + 1 + L((0))
1 1 1 + L(1)
2 1 1 + L(6)
3 1 1 + L() + L(2)
4 1 1 + L( ) + L(1) 1
5 1 1 + L(2)
6 2 L(1) + 1 + L()
We thus have total ost 4n + 6 under the unit ost measure and (n ) under the 2

logarithmi ost measure ( ompare Se tion 1.6 for a denition of ). Note that the
ost of line 3 is P (1+ L(2 )+ L(2)) = (n ). Note also, that one an redu e the
n
i=0
1 i 2

ost of omputing 2 to (log n) under the unit ost measure and to (n) under
n

the logarithmi ost measure ( f. Exer ise 1).

Version: 18.10.99 Time: 17:53 {5{
6
We infer from this example that the ost of a program an dier drasti ally under
the two measures. It is therefore important to always he k whether the unit ost
measure an be reasonably used. This will be the ase in Chapters 2, 3, 4, 5, 7and 8.
Analogously, we use unit and logarithmi ost measure for storage spa e also. In the
unit ost measure we ount the number of storage lo ations and registers whi h are
used in the omputation and forget about the a tual ontents, in the logarithmi
ost measure we sum the lengths of the binary representations of the ontents of
registers and storage lo ations and maximize over time.
Example ( ontinued): Program 1 for omputing 2 uses registers  and and
n
1
lo ations 0 and 1. Hen e its spa e omplexity is 4 under the unit ost measure.
The ontent of all 4 ells is bounded by 2 , two of them a tually a hieve that value.
n

Hen e spa e omplexity is (L(2 )) = (n) under the logarithmi ost measure.
n

Address modi ation by index registers is the only means of address modi ation
in RAMs. Most realisti omputers allow two other te hniques for address mod-
i ation: general address substitution and dire t address al ulation. General
address substitution allows us to use any lo ation as an index register, i.e., mod-
ied operands of the form (i + (j )) an be used also. The ost of fet hing su h
an operand is 2 in the unit ost and L(i) + L(j ) + L((j )) in the logarithmi ost
measure. It is not too hard to simulate the enlarged instru tion set by our original
instru tion set with only a onstant in rease in ost. Let us for example onsider
the instru tion  (i + (j )). It is simulated by
1 (j )
 (i + ) 1

However, the ontent of is destroyed by this pie e of ode. We therefore have

1
to save the ontent of before exe uting it. Let us assume that lo ation 0 is not
1
used (Exer ise 2 dis usses this assumption in detail) in the program whi h is to be
simulated. Then we only have to bra ket the above pie e of ode by (0) and 1
1 (0). We obtain
New Instru tion Unit Cost Logarithmi Cost
 (i + (j )) 3 1 + L(i) + L(j ) + L((j ))+
L((i + (j )))

Simulating Program
(0) 1 8 6 + L(i) + L(j ) + 2L((j ))+
(j )
1 L((i + (j ))) + 2L( ) 1
 (i + ) 1
(0)
1

The ost of the simulating program is only larger by a onstant fa tor in the unit
ost measure. We thus have
Version: 18.10.99 Time: 17:53 {6{
 1.1. Ma hine Models: RAM and RASP 7
Lemma 1. General address substitution redu es the ost of RAM programs by
only a onstant fa tor in the unit ost measure.
The situation is slightly more ompli ated in the logarithmi ost measure. Fa tor
L( ) annot be estimated in a simple way. We therefore hange the simulation
1
method and arrive at a simple onne tion between the ost of the original program
and the ost of the simulating program. We want lo ation 0 to always ontain the
ontent of . We a hieve this goal by inserting (0) after every instru tion
1 1
whi h modies and by inserting (0) before every instru tion whi h uses .
1 1 1
For example, we repla e  (i + ) by (0);  (i + ) and (i) by
1 1 1 1
1 (i); (0) . This modi ation in reases the ost only by a onstant fa tor
1
under both measures. Finally, we repla e the instru tions using general address
substitution as des ribed above, i.e., we repla e, e.g.,  (i + (j )) by 1
(j );  (i + ). Note that we do not have to in lude this pie e of ode into
1
bra kets (0) and 1 (0) as before be ause we took are of saving
1 1
elsewhere. We thus have (details are left to Exer ise 2)
Theorem 1. General address substitution an redu e the time omplexity of RAM
programs by at most a onstant fa tor in both ost measures.

Next we dis uss dire t address al ulation. We extend the RAM model by a
program store PS and all the extended model RASP (Random A ess Stored
Program Ma hine). The program store onsists of innitely many lo ations num-
bered 0; 1; 2; : : : . Ea h lo ation has two parts. The rst part ontains the name of
the instru tion (the op ode), the se ond part ontains the operand, i.e., either an
address or the number of an instru tion.
Example ( ontinued): The RASP-version of our example program is shown in
Figure 2. We use the RAM-instru tion as the op ode, data addresses are repla ed
by symbol a and instru tion addresses are repla ed by k.
Op ode Address
0 (a) 1 0
1  a 1
2 if = 0 then goto k
1 6
3  a 2
4 a 1 1 1
5 goto k 2
6 (a)  1
Figure 2. A RASP program for omputing 2 n

The number of op odes is nite be ause there are only four registers and only
a nite number of instru tions. For the sequel, we assume a xed bije tion between
op odes and some initial segment of the natural numbers. We use Num to denote
that bije tion.
Version: 18.10.99 Time: 17:53 {7{
8
In addition to the RAM instru tion set, the RASP instru tion set ontains so- alled
-instru tions. -instru tions operate on the program store. They are
9
  (i) >
> i 2 N0 ;
h
>
  (i + ) >
=
h 2 f1; 2g;
h j

 (i)  >
>
> j 2 f1; 2; 3g:
h
>
;
 (i + )
h j 

Instru tion   (i) loads the h-th omponent of lo ation i of PS into the a u-
h

mulator . If h = 1 then mapping Num is applied additionally. The semanti s of

all other instru tions is dened similarly.
Exe ution times of RASP instru tions are dened as in the RAM ase ex ept
one hange. RASP programs an grow during exe ution and therefore the time
required to modify the instru tion ounter annot be negle ted any longer. We
therefore add L(k) to the ost of an instru tion stored in ell k in the logarithmi
ost measure. We have the following relations in both ost measures.
Theorem 2. Exe uting a RAM program of time omplexity T (n) on a RASP takes

T (n) time units, where 2 R is a onstant depending on the RAM program
but not on the input.

Theorem 3. There is a > 0 su h that every RASP program of time omplexity

T (n) an be simulated in 
T (n) time units on a RAM.

Theorem 2 follows immediately from the observation, that a RAM program uses
only a xed number of storage lo ations of the program store and that therefore the
additive fa tor L(k) (k being the ontent of the program ounter) an be bounded
by a onstant whi h is independent of the parti ular input. Thus the \RASP ost"
of a RAM instru tion is at most times the \RAM ost" where = 1 + L(length
of RAM program to be exe uted on a RASP).
Theorem 3 is more diÆ ult to prove. One has to write a RAM program whi h
interprets RASP programs. Data store, program store and registers of the RASP
are stored in the data store of the RAM, more pre isely, we use lo ation 1 for the
a umulator, lo ations 2, 3 and 4 for the index registers, lo ations 5, 8, 11, 14, : : :
for the data store, and lo ations 6, 7, 9, 10, 12, 13, 15, 16, : : : for the program
store. Two adja ent ells are used to hold the two omponents of a lo ation of the
program store of the RASP. Lo ation 0 is used as an instru tion ounter; it always
ontains the number of the RASP instru tion to be exe uted next. The interpreter
has the following stru ture:
Version: 18.10.99 Time: 17:53 {8{
 1.2. Randomized Computations 9
(1) loop: load the op ode of the RASP-instru tion to be exe uted into the
a umulator;
(2) de ode the op ode and transfer ontrol to a modul whi h simulates
the instru tion;
(3) simulate the instru tion and hange the instru tion ounter.
(4) goto loop.
We leave the details to the reader (Exer ise 3). A ording to Theorems 2 and 3,
time omplexities on RAMs and RASPs dier only by a onstant fa tor. Sin e
we will negle t onstant fa tors anyway in most of what follows, the hoi e of the
ma hine model is not ru ial. We prefer the RAM model be ause of its simpli ity.
So far, RAMs (and RASPs) have no ability to intera t with their environment,
i.e., there are no I/O-fa ilities. The details of the I/O-fa ilities are not important
ex ept for Chapter 6 and we therefore always assume that the input (and output) is
stored in the memory in some natural way. For Chapter VI on NP- ompleteness we
have to be more areful. We equip our ma hines with two semi-innite tapes, a read
only input tape and a write only output tape. The input tape ontains a sequen e
of integers. There is one head on the input tape whi h is positioned initially on
the rst element of the input sequen e. Exe ution of the instru tion  Input
transfers the integer under the input head into the a umulator and advan es the
input head by one position. The ost of instru tion  Input is 1 in the unit
ost measure and 1 + L(n) in the logarithmi ost measure where n is the integer
to be read in. Similarly, the statement Output  transfers the ontent of 
onto the output tape. Whenever a RAM attempts to read from the input tape and
there is left no element on the input tape, the omputation blo ks. We will then
say that the output is undened and that the time omplexity of that parti ular
omputation is the number of time units onsumed until blo king o urred.

1.2. Randomized Computations

There are two important extensions of RAMs whi h we have to dis uss: randomized
RAMs and nondeterministi RAMs. We dis uss randomized RAMs now and put
o the dis ussion of nondeterministi RAMs to Chapter 6.
A randomized RAM (RRAM) has the ability to toss a perfe t oin and to
make further omputation dependent on the out ome of the oin toss, i.e., there is
an additional instru tion
 random
whi h assigns to  either 0 or 1 with probability 1=2 ea h. The ost of this in-
stru tion is 1 in both measures. We illustrate this new on ept by a very simple
example, an RRAM whi h omputes onstant 0.
1:  random
2: if  6= 0 then goto 1
Version: 18.10.99 Time: 17:53 {9{
10
Apparently, the ontent of  is 0 when the program stops. However, the running
time of the algorithm depends on the out ome of the oin tosses. More pre isely,
if the random hoi e omes out 0 at the k-th toss for the rst time, k  1, then
the running time is 2k in the unit ost measure. Sin e the oin is assumed to be
fair,
P the probability of this ase is 2 and therefore the average running time is
k

 2
2k = 4 ( f. appendix, formula S1). Note that the average running time

k
1
is small, although there is a han e that the program never halts.
k

The notion of RRAM is most easily made pre ise by redu ing it to ordinary
RAMs with two input fa ilities. The rst input fa ility re ords the a tual input
p for the randomized omputation (as above, we leave the exa t nature of that
input fa ility unspe ied), the se ond input fa ility is a read only input tape whi h
ontains a sequen e of 0's and 1's. Exe ution of  random reads the next element
(if there is one) from the input tape and transfers it into the a umulator .
Let A be a RAM program. For s a sequen e of 0's and 1's it thus makes sense
to talk about A(p; s), the output of A on input p and sequen e s of oin tosses, and
T (p; s), the running time of A on input p and sequen e s of oin tosses. Again,
A

we leave it unspe ied, whether the output is written into the memory or onto an
output tape. The expe ted running time of randomized algorithm A on input p is
then dened by X
T (p) = lim 2 T (p; s): k

!1 A
k
s 2f0 1gk
;
A

T (p) is well dened be ause of

A

Lemma 1. For all k and p:

X X
2 k
T (p; s)
A  2 k 1
T (p; t):
A

s 2f0 1gk
; 2f0 1gk+1
t ;

Proof : Let s 2 f0; 1g , and let t = s0 or t = s1. If the omputation of A on input p

k

and sequen e s of oin tosses stops regularly, i.e., is not blo ked be ause sequen e s
is exhausted, then T (p; s) = T (p; t). If it is not blo ked but never halts then
A A

T (p; s) = 1 = T (p; t). If it is blo ked then T (p; s)  T (p; t).

A A A A

We an now dene T (n) and T (n) as des ribed above. Of ourse T (n) is only
A
av av

dened with respe t to a probability distribution on the inputs.

A A

What do we understand by saying that a randomized algorithm A omputes a

fun tion f : P 7! Y ? The answer to this question is not evident sin e the output of
A an depend on the parti ular sequen e s of oin tosses used in the omputation.
Denition: Let f : P 7! Y and " : N 7! R be fun tions. The randomized
algorithm A omputes f with error probability at most " if for all p 2 P
lim jfs 2 f0; 1g ; f (p) = A(p; s)gj  1 "(g(p));
k

!1k 2 k

Version: 18.10.99 Time: 17:53 {10{

 1.2. Randomized Computations 11
where g(p) is the size of input p.

An argument similar to the one used in Lemma 1 shows that the limit in the
denition above always exists. Of ourse, only the ase "(n) < 1=2 is interesting.
Then A gives the desired output with probability larger than 1=2. A randomized
algorithm A is alled Las Vegas algorithm for fun tion f if it omputes f with
error probability 0 of error, i.e., "(n) = 0 in the above denition. In parti ular,
whenever A(p; s) stops and is dened then A(p; s) = f (p). Las Vegas algorithms
are a parti ularly suitable lass of randomized algorithms be ause the output is
ompletely reliable. We will see examples of Las Vegas algorithms in Se tions 2.1.3
and 3.1.2.
Of ourse, we want the error probability as small as possible. Suppose, that we
have a randomized algorithm A whi h omputes f : P 7! Y with error probability
at most "(n). If "(n) is too large we might just run A several times on the same
input and then determine the output by a majority vote. This should strengthen
our onden e in the output.

Lemma 2. Let Æ > 0. If the randomized algorithm A omputes f : P 7! Y with

error probability at most "(n) =  < 12 in time T (n) and if T Æ g is omputable in
A A

time O(T ) then there is a randomized algorithm B whi h omputes f with error
A

probability at most Æ in time
m
( 12 ) 1
T (n). Here m = 2d(log Æ)= log(1 ( 12

A

)2 )e and is a onstant. Moreover, B always halts within
m
( 12 ) 1
T (n) A

time units.

Proof : Consider any p 2 P . Let n = g(p), T = d(4=(1 2))
T (n)e and m = A

2d(log Æ)= log(1 ( ) )e. On input p, B omputes T , hooses m random sequen es

1 2

s ; s ; : : : ; s of length T ea h, and simulates A on inputs (p; s ); : : : ; (p; s ) for up

2
1 2 m 1 m

to T time units ea h. It then outputs whatever the majority of the simulated runs of
A outputs. Apparently B runs for at most O((1+ m)
T ) = O(m
( )
T (n)) 1 1
A

time units. Moreover, f (p) 6= B (p; s ; : : : ; s ) i A(p; s ) 6= f (p) for at least m=2
2
1 m i

distin t i's. Next note that

 
fs 2 f0; 1g ; f (p) 6= A(p; s)g
 T

2 T

 lim jfs 2 f0; 1g ; f (p) 6= A(p; s)gj + jfs 2 f0; 1g ; T (p; s) > T gj
k T
A

!1
k 2 k
2 T

  + ( )=21
2

= 1
2
( );
1
2
1
2

sin e A omputes f with error at most  (and therefore the rst term is bounded
Version: 18.10.99 Time: 17:53 {11{
12
by ) and sin e for p 2 P with g(p) = n we have
T (n)  T (p)
A A

 jfT (p; s); 2s 2 f0; 1g gj

X A
T

 jfT (p; s); s 2 f0; 1g2 and T (p; s) > T gj

X A
T
A
T

 
fs 2 f0; 1g ; T (p; s)  T g
T

>T
; A

2 T

and therefore the se ond term is bounded by T (n)=T  (1 2)=4. Let = A

1
2
1
2
( ). Then
1
2
 
fs


s 2 f0; 1g
; B (p; s ; : : : ; s ) 6= f (p)g
1 m
T m
1 m

2
T m

 
X
m
m
 i
i
(1 ) m i

i=m=2

 
X
m
m
 i
m=2
(1 ) m=2
(sin e < 1=2)
i=m=2

 2 (1 )
m m=2 m=2

= (4 (1 )) m=2

 Æ: (by denition of m)
It is worth illustrating Lemma 2 by an example. Assume that  = 0:49 and Æ = 0:03.
Then m = 2d(log Æ=) log(1 ( ) )e = 2dlog 0:03= log 0:9999e = 70128. Thus we
1 2

have to repeat a omputation whi h gives the orr t answer with probability 0.51
2

about 70000 times in order to raise the level of onden e to 0.97 . If we start with a
more reliable ma hine, say  = 0:25, then m redu es to 2dlog 0:03= log 0:9375e = 110.
By this example we see that bringing the error down from 0.49 to 0.25 is the diÆ ult
part, in reasing the level of onden e further is easy.
Randomized algorithms have a fair oin available and deterministi algorithms
have not. It is therefore important to know how well a randomized algorithm an be
simulated by a deterministi algorithm. We approa h this problem from two sides.
First we show that for xed problem size one an always repla e the fair oin by a
xed sequen e of 0's and 1's of reasonable length (Theorem 1), and then we show
how to use good pseudo-random number generators in randomized omputations.
Theorem 1. Let n 2 N, N = jfp 2 P ; g(p)  ngj, Æ = 1=(N + 1) and let A,
B ,  and f be dened as in Lemma 2. Then there is a sequen e s 2 f0; 1g
, 0
T m

Version: 18.10.99 Time: 17:53 {12{

 1.2. Randomized Computations 13
m and T as in the proof of Lemma 2 (under the additional assumption that T is A

non-de reasing), su h that f (p) = B (p; s0 ) for all p 2 P . n

Proof : B omputes f with error probability at most Æ. Let P = fp 2 P ; g(p)  ng. n

Then for all p 2 P n

 
fs 2 f0; 1g
; B (p; s) 6= f (p)g  Æ
2

T m
= 2
=(N + 1); T m T m

and therefore X X
if B (p; s) 6= f (p) then 1 else 0
s 2f0 1gT
m
; p 2 n
P

X X
= if B (p; s) 6= f (p) then 1 else 0
p2Pn s2f0;1gT
m

 N
2
=(N + 1) T m

< 2
: T m

Thus there is at least one s 2 f0; 1g
su h that

0
T m

X
if B (p; s0 ) 6= f (p) then 1 else 0 < 2
=2
= 1: T m T m

p2 n
P

Hen e B (p; s ) = f (p) for all p 2 P .

0 n

We illustrate Theorem 1 by an example. Assume P = f0; 1g and g(p) = jpj,

the length of bit string p. Then jP j  2 . Assume also that we start with
n
n+1

a randomized ma hine A with  = 1=4 and running time T (n) = n for some k. A
k

Taking Æ = 1=(2 +1), Lemma 2 yields a ma hine B with worst ase running time
n+1

T (n) = O(( log Æ)
T (n)) = O(n ) and error probability at most Æ. Moreover,
B A
k +1

by Theorem 1, there is a xed 0-1 sequen e s of length O(n ) whi h an be 0

k +1

used by B instead of a true random number generator. Unfortunately, the proof of

Theorem 1 does not suggest an eÆ ient method for nding a suitable s . 0
The question now arises whether we an use a pseudo-random number genera-
tor (say built-in pro edure Random on your favorite omputer) to generate oin toss
sequen es for randomized algorithms. A typi al pseudo-random number generator
works as follows. It onsists of a fun tion T : f0; 1g 7! f0; 1g whi h is designed
m m

su h that there is no \obvious" onne tion between argument x and value T (x).
The most popular hoi e of fun tion T is
T (x) = (a
x + ) mod 2 m

where the argument x is interpreted as a number between 0 and 2 1, and m

the numbers a and are of the same range. The result is nally trun ated to
the last m bits. A user of a pseudo-random number generator provides a \seed"
Version: 18.10.99 Time: 17:53 {13{
14
x0 2 f0; 1g and uses the transformation T to generate a sequen e x ; x ; : : : ; x
m
1 2 k

with x = T (x ). Thus a pseudo-random number generator takes a bit sequen e x

i+1 i 0
of length m and produ es a bit sequen e (take the on atenation of x ; : : : ; x ) of
1 k

length k
m for some k.

We an therefore dene a pseudo-random number generator as a mapping  :
f0; 1g 7! f0; 1g
m E (m)
where E (m)  m. It takes a seed x 2 f0; 1g and produ es
m

a sequen e (x) of length E (m).

The hoi e of the seed is left to the user and we will not dis uss it any further.
He might use a physi al devi e or a tually toss a oin. However, we will dis uss
the desirable properties of mapping  in more detail. The mapping  takes a bit
string of length m and produ es a bit string of length E (m). If E (m) > m then
(x) is ertainly not a random string (in the sense that all strings of length E (m)
are equally likely) even if x is a random string of length m. After all, only 2 out
m

of the 2 possible strings of length E (m) are in the range of .

E (m)

Suppose now we an generate random strings x of length m, is it then safe to

use the pseudo-random strings (x) of length E (m) in a randomized algorithm? At
rst glan e the answer seems \No" be ause pseudo-random strings are not random
strings. However, they might be \random enough" to be used anyway instead of
a true random sequen e. In order to make this pre ise we need to introdu e a
measure of quality for pseudo-random number generators. We do so by introdu ing
the on ept of a statisti al test.
Consider for example the fun tion h : f0; 1g 7! f0; 1g whi h yields one if
the number of zeroes and ones in the argument diers by at most 10%. Then h
applied to a random bit string yields one with very high probability and we might
require the same for a random element in the range of . If this were the ase then
the statisti al test h annot distinguish between true random sequen es and the
sequen es obtained by applying  to shorter random sequen es. If this were true
for all statisti al tests (a notion whi h still needs to be dened) then the sequen es
generated by  are rightly alled pseudo-random.
In general, we dene a statisti al test to be any fun tion h : f0; 1g 7! f0; 1g
whi h yields a one for at least half of the arguments of any xed length.
A randomized algorithm A an easily be turned into a statisti al test h . The
test h alls a sequen e s 2 f0; 1g \good" (i.e., yields a one) if the running time
A

of algorithm A when using sequen e s of oin tosses does not ex eed its expe ted
running time by more than a fa tor of, say, two. Then most random sequen es of
any xed length are good, i.e., h is a statisti al test and has polynomial running
A

time if A has (we restri t our attention to polynomial time bounded algorithms
be ause we saw in the beginning of this hapter that algorithms with exponential
running time are hopelessly ineÆ ient).
We say that a pseudo-random number generator  passes test h if \many" (a
pre ise denition is given below) sequen es in the range of  are good.
Suppose now that  passes all statisti al tests of polynomial time omplexity.
Then  passes also test h if A is a polynomial time bounded algorithm and hen e
A

we an hope to use the pseudo-random sequen es generated by  instead of true

Version: 18.10.99 Time: 17:53 {14{
 1.2. Randomized Computations 15
random sequen es for operating algorithm A.
We will now make these on epts pre ise. Part ) of the denition below
denes in pre ise terms what we mean by the phrase that the mapping  passes the
statisti al test h. We give two variants of the denition whi h are geared towards the
two appli ations of pseudo-random number generators to be des ribed later: fast
simulation of randomized algorithms by deterministi algorithms and redu tion of
the number of oin tosses in randomized omputations.
Denition:
a) A fun tion h : f0; 1g 7! f0; 1g is polynomial time omputable if there is
a deterministi algorithm omputing h whose running time is bounded by a
polynomial.
b) A statisti al test is a fun tion h : f0; 1g 7! f0; 1g with
jfx 2 f0; 1g ; h(x) = 1gj  2
k k 1
for all k:
) Let E : N 7! N be a fun tion, let  : f0; 1g 7! f0; 1g be su h that j (x)j =
E (m) for jxj = m and let m : N 7! N be a fun tion. Let h be a statisti al
0
test and let h be omputable in time t
n for some t where n is the size of the
t

input. Then  passes test h if for all m  m (t)0


x 2 f0; 1g ; x 2 range() and h(x) = 1 6= ;:
E (m)

Furthermore,  passes test h well if for all m  m (t) 0

 
fx 2 f0; 1g ; x 2 range () and h(x) = 1g  2 =8:
E (m) m

Remark: If  passes test h well then a random element in the range of  satises h
with probability ex eeding 1=8 while a true random element of f0; 1g satises h
E ( m)

with probability ex eeding 1=2. The hoi e of ut-points a = 1=2 and a = 1=8 is
1 2
arbitrary; however 0 < a  a is essential.
2 1

d) A mapping  is a good (very good) pseudo-random number generator if

it passes all polynomial time omputable statisti al tests (well).
The reader should pause at this point and should try to grasp the intuition behind
this denition. We dened a statisti al test to be any predi ate on bit strings whi h
at least half of the strings of any xed length satisfy (part b)). Furthermore, we
restri t our attention to simple ( = polynomial time omputable) predi ates (part
a) and )). A pseudo-random number generator  passes all statisti al tests if the
range of  has no simple stru ture, i.e., if there is no large and omputationally
simple subset of f0; 1g , namely a set fx 2 f0; 1g ; h(x) = 1g for some
E (m) E (m)

statisti al test h, whi h  either misses ompletely or does not hit with suÆ iently
high probability. In other words, the properties of a random element in range() are
Version: 18.10.99 Time: 17:53 {15{
16
diÆ ult to predi t, and hen e the elements produ ed by  are rightly alled pseudo-
random sequen es. Note that \being random" is the same as \being diÆ ult to
predi t".
It is not known whether (very) good random number generators in the sense
of this denition exist. However, it an be shown that very good random number
generators with E (m) = m for any k omputable in polynomial time exist if any
k

one of the following number theoreti problems is hard: the dis rete logarithm
problem or the problem of fa toring integers. We have to refer the reader to the
literature for a dis ussion of these results ( f. A.C. Yao: \Theory and Appli ations
of Trapdoor Fun tions", IEEE FOCS 1982, 80{91).
We pro eed on the assumption that a (very) good polynomial time omputable
pseudo-random number generator  exists with, say, E (m) = m . We show that 2

good pseudo-random number generators an be used to speed up the simulation of

randomized algorithms by deterministi algorithms and that very good generators
an be used to redu e the required number of true random hoi es. The latter
onsequen e is important if generation of truely random bits ever be ame possible,
yet would be expensive.
For on reteness and simpli ity, let A be a Las Vegas algorithm with polynomial
running time, i.e., T (n)  t
n for some t 2 N and let  be a good pseudo-
A
t

random number generator with E (m) = m . Let p 2 P , n = g(p), be su h that

p 2

2t
n  m (t). Then h : f0; 1g 7! f0; 1g with

t
0 p

n
1 if T (p; s)  2t
n ;
h (s) = A
t

0 otherwise
p

is omputable in time O(t
n ) and we have h (s) = 1 for at least fty per ent of
t
p

the bit strings of length 2t
n . This follows from the fa t that the running time of
t

T on p and s an ex eed twi e thep expe ted value for at most half of the sequen es
A

of oin tosses. Hen e for all m  2t
n t

fs 2 f0; 1g E (m)
; s 2 range( ) and h (s) = 1g 6= ;
p

or
fs 2 f0; 1g ; s 2 range () und T (p; s)  2t
n g 6= ;:
E (m)
A
t

This relationship dire tly leads to a deterministi simulation of probabilisti ma-

hinespwhi h is more eÆ ient than the naive one. Let p 2 P , g(p) = n and let
m = 2t
n . Consider Program 2.
t

Sin e A is a Las Vegas algorithm and sin e there is an s 2 (f0; 1g ) with m

T (p; s)  2t
n , the simulation always

A
t
p
produ e the orre t answer. Also the
running time of the simulation is O(2 t
(t
n + q(m))), where q is the poly-
2t n t

nomial bound on the time needed for omputing . Thus the existen e of good
pseudo-random number generators leads to more eÆ ient simulations of probabilis-
ti ma hinest by deterministi ma hines. Note that the naive simulation has running
time O(2

t
n ).
t n t

Version: 18.10.99 Time: 17:53 {16{

 1.2. Randomized Computations 17

for all x 2 f0; 1g m

do s (x);
run A on p and s for up to 2t
n steps; t

if A halts within that number of steps

then output, whatever A outputs and halt
fi
od.
Program 2

Assume now that  is a very good pseudo-random number generator. Then

 
 fs 2 f0; 1g E (m)
; s 2 range () and T (p; s)  2t
n g  2 =8;
A
t m

where m, n and p are dened as above. In other words, a random x 2 f0; 1g m

produ es an s = (x) su h that T (p; s)  2t
n with probability at least 1=8. This

A
t

observation leads to the following implementation of algorithm A in Program 3

whi h uses fewer oin tosses than A.
repeat generate a random sequen e of m bits and all it x;
s (x)
until A on p and s halts within 2t
n steps ; t

output, whatever A outputs on p and s.

Program 3

Sin e a random x 2 f0; 1g produ es an s = (x) 2 f0; 1g with T (p; s) 

m E (m)
A

2t
n having probability at least 1=8 only 8 iterations of the loop are required on
t

the average. pHen e the algorithm above simulates A in time O(T (n)) = O(n ) and A
t

uses only O( n ) random bits. Thus if true random hoi es are possible but ostly,
t

this is a very signi ant improvement. This ompletes our dis ussion on the use of
(very) good pseudo-random number generators in randomized omputations.
We end this se tion with a remark on the relation between the expe ted time
omplexity of deterministi algorithms and the running time of probabilisti algo-
rithms.
Let A be a Las Vegas algorithm whi h omputes fun tion f : P 7! Y . Let us
assume for simpli ity that A makes at most a(n) oin tosses for every n 2 N on any
input p 2 P with g(p) = n. Then
X
T (p)
A = T (p; s)=2
A
a(n)
:
s 2f0 1ga(n)
;

Suppose also that we are given a probability distribution  on the set P of problem
instan es. Let B be a deterministi algorithm for f , whose expe ted running time
Version: 18.10.99 Time: 17:53 {17{
18
on inputs of size n is minimal (B is ertain to exist if P , the set of problem instan es n

of size n is nite), i.e., for all deterministi algorithms C

E (fT (p); p 2 P
B and g(p) = ng)  E (fT (p); p 2 P and g(p) = ng); C

where expe tations are al ulated with respe t to probability distribution . For ev-
ery xed s 2 f0; 1g algorithm A with sequen e s of oin tosses is a deterministi
a(n)

algorithm and hen e

X X
(p)
T (p)
B  (p)
T (p; s):
A

p2 P n 2
p P n

Sin e this inequality holds for every s we have

X X X
(p)
T (p) B  (p)
T (p; s)=2
A
a(n)

p 2 P n s 2f0 1ga(n) 2
; p P n
X X
 (p) T (p; s)=2
A
a(n)

p 2 P n s 2f0 1ga(n)
;

X
 (p)
T (p): A

p 2 P n

Thus the expe ted running time of Las Vegas algorithm A on inputs of size n an
not be better than the expe ted running time of the best deterministi algorithm.
We will use this fa t to derive lower bounds on the randomized omplexity of some
sorting problems in Chapter 2. For sorting and some related problems we will
derive
(n log n) lower bounds on the expe ted running time of a large lass of
deterministi algorithms in Chapter 2. The inequality derived above immediately
extends that lower bound to the orresponding lass of Las Vegas algorithms.

1.3. A High Level Programming Language

Only a few algorithms in this book are formulated in RAM ode, most algorithms
are formulated in a high level ALGOL-like programming language. We feel free
to introdu e additional statements into the language, whenever the need arises
and whenever translation into RAM ode is obvious. Also we make frequent use
of omplex data stru tures su h as lists, sta ks, queues, trees and graphs. We
hoose this very high level des ription for many algorithms be ause it allows us
to emphasize the prin iples more learly. Most statements of our programming
language are known from ALGOL-like languages. In parti ular, we use
Version: 18.10.99 Time: 17:53 {18{
 1.3. A High Level Programming Language 19
the onditional statement:
if h onditioni then hstatementi else hstatementi fi;
the iterative statement:
while h onditioni do hstatementi od;
the for -loop:
for i from hexpressioni step hexpressioni to hexpressioni
do hstatementi od
If the step size is unspe ied then it is 1 by default. We also use a se ond form of
the for -loop:
for i 2 hseti do hstatementi od
with the following semanti s. The statement is exe uted jhsetij-times; i runs
through the members of the set in some unspe ied order. Assignments are written
in the form hvariablei hexpressioni. Translation of all statements above into
RAM ode is simple. We des ribe the translation in the ase of a while-statement
while B do S od:
Let P be a RAM program for B , i.e., P evaluates expression B and leaves a 0 (1)
1 1
in the a umulator, if B evaluates to false (true). Let P be a RAM program for S .
2
Then the RAM ode of Program 4 realizes the while-loop.

P1
if  = 0 then goto exit;
P2
goto rst instru tion of P1 ;
exit:
Program 4

Program 4 also denes the omplexity of the while-loop; it is the sum of the
time units spent on the repeated testing of the ondition and the exe ution of the
body.
Variables in our programs ontain unstru tered elementary values or stru tured
values. The elementary data types are integer, real, boolean, pointer and
an additional unspe ied data type. We use this additional data type in sorting
algorithms; the data to be sorted will be of the unspe ied type. The operations
dened on this additional data type are given on a ase by ase basis. Stru tured
data types are strings over some alphabet, re ords, arrays, lists, sta ks, queues,
trees and graphs. Strings are treated in 2.2, graphs in 4.1, all other stru tured types
are treated in 1.4. The type of a variable will be obvious from the ontext in most
ases; in this ase we will not de lare variables expli itely.
Version: 18.10.99 Time: 17:53 {19{
20
Pro edures play a major role in our programs. Parameters are restri ted to
elementary types. Then parameter passing takes onstant time ( f. Se tion I.5).
Non-re ursive pro edures are easily redu ed to RAM ode; one only has to substi-
tute the pro edure body at the pla e of the all. The situation is more ompli ated
for re ursive pro edures. We treat re ursive pro edures in 1.5.
Comments are bra keted by o and o .

1.4. Stru tured Data Types

Re ords and arrays are the most simple ways of stru turing data.
An array A of n elements onsists of n variables A[1℄; : : : ; A[n℄ of the same
type. An array is stored in n onse utive storage lo ations, e.g., in lo ations BA +1,
BA + 2; : : :,BA + n. Here BA stands for base address. If x is a variable stored in
lo ation i then a essing array element A[x℄ is realized by means of index registers.
The following pie e of ode
1 (i);
 (BA + ) 1

loads A[x℄ into the a umulator for a ost of 4 in the unit ost measure and 2+L(i)+
L(BA) + 2L(x) + L(A[x℄) in the logarithmi ost measure. Again the logarithmi
ost measure is proportional to the length of the numbers involved.
Re ords are xed size olle tions of variables of dierent type, e.g.,
re ord age : integer; in ome : real end:
A variable x of this re ord type is easily simulated by two simple variables, a variable
x:age of type integer and a variable x:in ome of type real.
Queues, sta ks, lists and trees are treated in the se tions below. They are all
redu ed to arrays.

1.4.1. Queues and Sta ks

Queues and sta ks are used to represent sequen es of elements whi h an be modied
by insertions and deletions. In the ase of queues insertions are restri ted to the
end of the sequen e and deletions are restri ted to the front of the sequen e. A
typi al example is a waiting line in a student afeteria. Queues are also known
under the name FIFO store (rst in - rst out). In the ase of sta ks, insertions
and deletions are restri ted to the end of the sequen e: LIFO store (last in - rst
out). Very often, the names Push and Pop are used instead of insertion into and
deletion from a sta k.
A sta k K is most easily realized by an innite array K [1℄, K [2℄, : : : and an
index Top of type integer. The sta k onsists of elements K [1℄,: : :,K [Top ℄; K [Top ℄
is the top element of the sta k. The following pie e of ode realizes operation
Push (K; a)

Version: 18.10.99 Time: 17:53 {20{

 1.4.2. Lists 21
Top Top + 1;
K [Top ℄ a.
The next pie e of ode deletes an element from the sta k and assigns it to variable x,
i.e., it realizes x Pop (K )
if Top = 0 then error fi;
x K [Top ℄;
Top Top 1.
Of ourse, innite arrays are rarely available. Instead we have to use a nite array
of, say, n elements. In this ase a push-operation should also he k whether over ow
o urs. In either ase the sta k operations Push and Pop take onstant time in the
unit ost measure.
A queue S is also realized by an array. We immediately treat the ase of a
nite array S [1 : : n℄. We on eptually think of array S as a losed line, i.e., S [1℄
follows S [n℄, and use two indi es Front and End to denote the borders of the queue.
More pre isely, if Front < End then the queue onsists of S [Front ℄; : : : ; S [End 1℄,
if Front > End then the queue onsists of S [Front ℄; : : : ; S [N ℄; S [1℄; : : : ; S [End 1℄,
and if Front = End then the queue is empty. Then deleting an element from S and
assigning it to x is realized by
if Front = End then error fi;
x S [Front ℄;
Front 1 + (Front mod n)
and inserting an element a into the queue is realized by
S [End ℄ a;
End 1 + (End mod n );
if Front = End then error fi.
Insertions into and deletions from queues take onstant time in the unit ost mea-
sure.

1.4.2. Lists

Linear lists are used to represent sequen es whi h an be modied anywhere. In

linear lists the elements of a sequen e are not stored in onse utive storage lo ations,
rather ea h element expli itly points to its su essor ( f. Fig. 3).

Head : element element element element

1 2 3 4
Figure 3. A linear list
Version: 18.10.99 Time: 17:53 {21{
22
There are many versions of linear lists: singly linked, doubly linked, ir ular,
et . We dis uss singly linked lists here and leave the others for the exer ises. In
singly linked linear lists ea h element points to its su essor. There are two realiza-
tions of linear lists: one by re ords and one by arrays. We dis uss both, although
internally the re ord representation boils down to the array representation. We use
both representations throughout the book and always hoose the one whi h is more
onvenient.
In the re ord representation an element of a linear list is a re ord of
type element = re ord ont : real; next : "element end
and Head is a variable of type "element. Head always points to the rst element of
the list. The pi torial representation is as given in Figure 3.
The realization by two arrays is loser to RAM ode. Real array Content [1 : : n℄
ontains the ontents of the elements and integer array next [1 : : n℄ ontains the
pointers. Head is an integer variable. Our example list an be stored as shown in
Figure 4.
Head = 2 Content next
1 element 4 0
2 element 1 4
3 element 3 1
4 element 2 3
Figure 4. Realization by arrays
Here Head = 2 means that row 2 of the array ontains the rst element of the
list, the se ond element is stored in row 4, et . The last element of the list is stored
in row 1; next [1℄ = 0 indi ates that this element has no su essor.
We des ribe now insertion into, deletion from and reation of linear lists. We
give two versions of ea h program, one using re ords and one using arrays. In either
ase we assume that there is a supply of unused elements and that a all of pro edure
Newr (var p :"element) resp. Newa (var p : integer) takes a node from the supply
and makes p point to it in the re ord (array) version and that a all of pro edure
Disposer (var p :"element) resp. Disposea (var p : integer)) takes the node pointed
to by p and returns it to the supply. SuÆxes r and a distinguish between the
representations. We dis uss later how the supply of elements is implemented. In
our example a all Newa (p) might assign 5 to p be ause the fth row is unused and
a all Newr (p) results in the situation depi ted in Figure 5.
p:
After all of Newr
Figure 5.
Pro edure Create takes one parameter and makes it the head of an empty list.
Again we use suÆxes r and a to distinguish the re ord and the array version.
Version: 18.10.99 Time: 17:53 {22{
 1.4.2. Lists 23
pro edure Creater (var Head : pro edure Createa (var Head :
"element); integer);
Head nil Head 0
end. end.

Pro edure Insert takes two parameters, a pointer to the element after whi h we
want to insert and the ontent of the new element.
pro edure Insertr (p :"element, a:real); pro edure Inserta (p :integer; a:real);
var q : "element; var q : integer;
Newr (q); Newa (q);
q": ont a; Content [q℄ a;
q":next p":next; next [q℄ next [p℄;
p ":next q next [p℄ q
end. end.

This pro edure fails for empty lists. (The following pro edure Delete fails already
for lists onsisting of only one element.) A real implementation must take this into
a ount and add tests for spe ial ases. In general, the des riptions of our algorithms
skip sometimes su h spe ial ases to enhan e the readability of the book. Figure 6
illustrates one all of Insertr (p; a).
Pro edure Delete takes one parameter, a pointer to the element whi h pre edes
the element we want to delete.
pro edure Deleter (p :"element); pro edure Deletea (p : integer);
var q :"element; var q : integer;
q p":next; q next [p℄;
p":next q":next; next [p℄ next [q℄;
Disposer (q) Disposea (q)
end. end.

Finally, we have a fun tion to test whether a list is empty.

fun tion Emptyr (Head : "element); fun tion Emptya (Head : integer);
Emptyr (Head = nil) Emptya (Head = 0)
end. end.

It remains to dis uss the supply of unused nodes in more detail. Supply is again a
linear list with head Free . We will only des ribe the array version of pro edures New
and Dispose be ause these pro edures are usually built-in fun tions in programming
languages whi h ontain re ords. Internally, re ords are always realized by arrays
and therefore Newr and Disposer are identi al to Newa and Disposea . A supply of
n elements is reated by
Version: 18.10.99 Time: 17:53 {23{
24




before the all
p:

after exe uting (1)
p: q:

after exe uting (2) and (3)
p: q: a

after exe uting (4)
p: q: a

Figure 6. Snapshots during exe ution of Insertr (p; a)

pro edure Inita(n : integer);

var i : integer;
Free 1;
for i from 1 to n 1
do next [i℄ i + 1 od;
next [n℄ 0
end.

Newa and Disposea are realized by

Version: 18.10.99 Time: 17:53 {24{
 1.4.3. Trees 25
pro edure Newa(q : integer);
q Free ;
if Free = 0 then supply exhausted fi;
Free next [Free ℄;
next [q℄ 0
end.
and
pro edure Disposea(var q : integer);
next [q℄ Free ;
Free q;
q 0
end.
We summarize in
Theorem 1. Creating a supply of n nodes takes time O(n) in the unit ost measure,
reation of an empty list, insertion into, deletion from a linear list given that the
positions of insertion or deletion is known and testing for emptiness take time O(1)
in the unit ost measure.

One often uses linear lists to realize sta ks and queues. In parti ular, several sta ks
and queues may share the same supply of unused nodes. This will guarantee high
storage utilization if we have knowlegde of the total length of all sta ks and queues
but no knowlegde of individual length. Typi al examples an be found in Chapter 6.
We store a graph by listing the set of its su essors for ea h node. In a graph of n
nodes and m edges these lists have total length m, but nothing is known in advan e
about the length of individual lists.

1.4.3. Trees

Trees onsist of nodes (bran hing points) and leaves. Let V = fv ; v ; : : :g be an

1 2
innite set of nodes and let B = fb ; b ; b ; : : :g be an innite set of leaves. We
1 2 3
dene the set of trees over V and B indu tively.
Denition:
a) Ea h element b 2 B is a tree. Then b is also the root of the tree.
i i

b) If T1 ; : : : ; T (m  1) are trees with pairwise disjoint sets of nodes and leaves

m

and v 2 V is a new node then the (m + 1)-tuple T = hv; T1 ; : : : ; T i is a m

tree. Node v is the root of the tree, (v) = m is its degree and T is the i-th
i

subtree of T .
In the graph-theoreti literature trees as dened above are usually alled ordered
rooted trees. We always draw trees with the root at the top and the leaves at the
Version: 18.10.99 Time: 17:53 {25{
26
bottom. As shown in the example tree of Figure 7 nodes are drawn as ir les and
leaves are drawn as re tangles.
v1

v2 v3

b1 v4 b2 b3

b4 b5
Figure 7. Tree T with nodes and leaves
Ex

We use the following terms when we talk about trees. Let T be a tree with
root v and subtrees T ; 1  i  m. Let w = root(T ). Then w is the i-th son of v
i i i i

and v is the father of w . Des endant (an estor) denotes the re exive, transitive
i

losure of relation son (father). w is brother of w ; j 6= i. In the tree of Figure 7

j i

b and v are brothers, v is father of v and b is des endant of v .

1 4 1 3 5 2

Denition (depth): Let v be a node or leaf of tree T . If v is the root of T then

depth(v; T ) = 0. If v is not the root of T then v belongs to T for some i. Then i

depth(v; T ) = 1 + depth(v; T ). We mostly drop the se ond argument of depth if it

i

is lear from the ontext.

Denition (height of a tree): Let T be a tree. Then
height(T ) = maxfdepth(b; T ); b is leaf of T g:
In Figure 7 we have depth(v ) = 1, depth(v ) = 2 and height(T ) = 3.
3 4 Ex

Denition: Tree T is a binary tree if all nodes of T have degree exa tly 2.
Our example tree T is a binary tree. A binary tree with n nodes has n +1 leaves.
Ex

The 1st (2nd) subtree is also alled left (right) subtree.

Information an be stored in the leaves and nodes of a tree. In some appli ations
we use only one possibility. A binary tree is realized by three arrays Lson , Rson
and Content or equivalently by re ords with three elds. Figure 8 gives the storage
representation of our example tree T . Ex

We have asso iated rows with nodes and leaves in some arbitrary way. If
information is only stored in the nodes and not in the leaves then leaves do not
have to be stored expli itly. All rows orresponding to leaves an be deleted and
pointers pointing to leaves are set to 0. A 0-pointer then represents a subtree
Version: 18.10.99 Time: 17:53 {26{
 1.4.3. Trees 27
root = 5 Content Lson Rson
1
Content of v 8 4 4
2
Content of v 3 2 1
3
Content of b 0 1 0
4
Content of b 0 5 0
5
Content of v 2 1 7
6
Content of b 0 2 0
7
Content of v 6 3 9
8
Content of b 0 4 0
9
Content of b 0 3 0
Figure 8. Realization of T by arrays Ex

onsisting of a single leaf. In the diagrams we will not draw leaves in this ase ( f.
Figures 9 and 10).
v1

v2 v3

v4
Figure 9. T Ex without leaves

root = 3 Content Lson Rson

1 Content of v 0 2 4
2 Content of v 0 3 0
3 Content of v 1 1 2
4 Content of v 0 4 0
Figure 10. Realization of T without leaves by arrays
Ex

Systemati exploration of a tree is needed frequently. A binary tree onsists of

three omponents: a root, a left subtree and a right subtree. Thus three methods
of tree traversal ome to mind naturally:
Preorder traversal: visit the root, traverse the left subtree, traverse the right
subtree: root, L, R.
Postorder traversal: traverse the left subtree, traverse the right subtree, visit the
root: L, R, root.
Symmetri traversal: traverse the left subtree, visit the root, traverse the right
subtree: L, root, R.
Version: 18.10.99 Time: 17:53 {27{
28
Symmetri al variants are obtained by inter hanging L and R. Pro edure Symord of
Program 5 traverses a tree in symmetri al order and prints the ontent of all nodes
and leaves.

pro edure Symord (v);

(1) if v is leaf
(2) then print(Content [v℄)
(3) else Symord (Lson [v℄);
(4) print(Content [v℄);
(5) Symord (Rson [v℄)
(6) fi
end.
Program 5

1.5. Re ursion

Re ursive pro edure Symord traverses a binary tree in symmetri al order. Before
we an estimate time and spa e omplexity of Symord we need to take a look at
the implementation of re ursive pro edures in RAM or RASP ode. Re ursive pro-
edures are realized by means of a sta k. We asso iate with ea h all (in arnation,
a tivation) of a pro edure an element of the sta k, alled a tivation re ord. The
a tivation re ord ontains omplete information about the all, i.e.,
a) the values of the a tual parameters,
b) the return address,
) the lo al variables.
If a pro edure has n parameters and m lo al variables then the a tivation re ord
onsists of n + 1 + m storage lo ations. The i-th ell, 1  i  n, ontains the value
of the i-th a tual parameter, the (n +1)-st ell ontains the return address, i.e., the
address of the instru tion whi h follows the pro edure all in the alling program,
and the (n + 1 + j )-th ell ontains the j -th lo al variable. Parameters and lo al
variables are addressed indire tly via the a tivation re ord. More pre isely, if Top is
the address of the storage lo ation immediately pre eding the a tivation re ord then
lo ation Top + i ontains the i-th a tual parameter and ell Top + n +1+ j ontains
the j -th lo al variable. Top is best stored in an index register. After ompletion of
the pro edure all ontrol is transfered to the address stored in lo ation Top + n +1.
Also Top is redu ed after ompletion, i.e., the a tivation re ord is deleted from the
sta k. Parameters and return address are omputed by the alling program, i.e.,
the rst n + 1 ells of an a tivation re ord are initialized by the alling program.
We are now ready to give the non-re ursive version of Symord by Program 6. Array
K [1 : : 1℄ is used as the sta k.
Version: 18.10.99 Time: 17:53 {28{
 1.5. Re ursion 29

(1') begin o the main program alls Symord (root) o

(2') Top 0;
(3') K [1℄ root;
(4') K [2℄ \HP";
(5') goto Symord;
(6') HP : Halt;
(7') Symord : o here omes the ode for Symord ;
node v is stored in K [Top + 1℄,
and return address is stored in K [Top + 2℄ o
(8') if Lson [K [Top + 1℄℄ = Rson [K [Top + 1℄℄ = 0
then o K [Top + 1℄ is a leaf o
(9') print(Content [K [Top +1℄℄);
(10') goto Finish
else o all Symord (Lson [v℄) o
(11') Top Top + 2;
(12') K [Top + 1℄ Lson [K [Top 1℄℄;
(13') K [Top + 2℄ \M1";
(14') goto Symord;
(15') M1 : Top Top 2;
(16') print(Content [K [Top + 1℄℄);
o all Symord (Rson [v℄) o
(17') Top Top + 2;
(18') K [Top + 1℄ Rson [K [Top 1℄℄;
(19') K [Top + 2℄ \M2";
(20') goto Symord;
(21') M2 : Top Top 2;
(22') goto Finish
fi;
(23') Finish : goto K [Top + 2℄
end.
Program 6

Call Symord (Lson [v℄) (line (3) of the re ursive program) is simulated by lines
(11'){(15') of the non-re ursive program. In (11') storage spa e for the a tivation
re ord is reserved. The a tivation re ord is initialized at lines (12') and (13');
at (12') the value of the a tual parameter and at (13') the return address \M1"
is stored. At line (14') ontrol is transfered to Symord and the re ursive all is
started. Upon ompletion of the all ontrol is transfered to label M1 (line (15')).
The spa e for the a tivation re ord is released and exe ution of Symord is resumed.
Analogously, one an asso iate the other instru tions of the non-re ursive program
with the instru tions of the re ursive program.
Program 6 is pra ti ally a RASP program. Line (23') uses a -instru tion. It
an be turned into a RAM program either by the method des ribed in the proof of
Version: 18.10.99 Time: 17:53 {29{
30
Theorem 3 of Se tion 1 or more simply by repla ing line (23') by Program 7.

if K [Top + 2℄ = \M1"
then goto M1;
if K [Top + 2℄ = \M2"
then goto M2;
goto HP
Program 7

We have des ribed a simplied method for translating re ursive programs into
non-re ursive ones. The method suÆ es if global variables an only ome out of
the main program. This will be the ase throughout this book. In the general ase
a single register Top does not suÆ e; one has to repla e it by a set of registers, one
for ea h a tivation re ord whi h an ontain global variables. We refer the reader
to a book on ompiler onstru tion for details, e.g., Gries (71)).
We are now ready to analyze the time omplexity of re ursive pro edures in
the unit ost measure. A onstant number of instru tions is required to set up a
re ursive all, namely to in rease Top , to store the n a tual parameters and the
return address. Note that n is xed and in dependent of the input; n an be
infered from the program text. Also, we allow only elementary types in parameter
position and hen e a single parameter an be passed in onstant time. Upon return
from a re ursive pro edure Top has to be de reased. Altogether, the administrative
overhead for a pro edure all is bounded by a onstant.
The following method for omputing the time omplexity of re ursive pro e-
dures is often useful. We asso iate with every pro edure all the ost of exe uting
the pro edure body, in luding the administrative ost for initiating further re ur-
sive alls but ex luding the time spent inside re ursive alls. Then we sum over all
alls and obtain the total ost in this way. We illustrate this method on pro edure
Symord . Ea h line in the body of Symord takes onstant time. Re all that we
only ount the time required to set up the re ursive alls at lines (3) and (5) of
Program 5 but that we do not ount the time needed to exe ute the re ursive alls.
Symord is alled on e for ea h leaf and node of a binary tree. Thus the total ost
of Symord is O(n) where n is the number of leaves and nodes of the binary tree.
The summation over all alls of a pro edure an be done more formally by
means of re ursion equations. Let T (v) be the running time of all Symord (v).
Then 
if v is a leaf;
T (v) = + T (Lson [v℄) + T (Rson [v℄) otherwise
1
2

for suitable onstants and . We an now show by indu tion on the height of v
1 2
(the tree with root v) that T (v) =
# nodes in the tree with root v +
# leaves
2 1
in the tree with root v. Although the indu tion proof is very simple we in lude it
for dida ti purposes.
Version: 18.10.99 Time: 17:53 {30{
 1.6. Order of Growth 31
Height(v) = 0: Then v is a leaf and we have T (v) = . 1
Height(v) > 0: Then v is a node and we have
T (v) = + T (Lson [v℄) + T (Rson [v℄)
2

= +
# nodes(Lson [v℄) +
# leaves(Lson [v℄)

2 2 1

+
# nodes(Rson [v℄) +
# leaves(Rson [v℄)

2 1

=
# nodes(v) +
# leaves(v):
2 1

A more detailed dis ussion of re ursion equations an be found in Se tion 2.1.3.

1.6. Order of Growth

In most ases we will not be able to derive exa t expressions for the running time
of algorithms; rather we have to be ontent with order of growth analysis. We use
the following notation.
Denition: Let f : N 7! N be a fun tion. Then O(f ),
(f ) and (f ) denote
0 0
the following sets of fun tions
O(f ) = fg : N ! N ; 9 > 0 9n : g(n) 
f (n)
0 0 0

for all n  n g 0

(f ) = fg : N ! N ; 9 > 0 9n : g(n) 
f (n)
0 0 0

for all n  n g 0

(f ) = fg : N ! N ; 9 > 0 9n : (1= )
f (n)  g(n) 
f (n)

0 0 0

for all n  n g 0

We use the O-notation in proofs of upper bounds, the

-notation in proofs of lower
bounds and the -notation whenever we an determine the order of growth exa tly.
It is ustomary to use the notations above together with the equality sign
instead of the symbols 2,  for the relations `element of' and `subset of', i.e., we
write n + 5n = n + O(n) = O(n ) instead of n + 5n 2 n + O(n)  O(n ).
2 2 2 2 2 2

More pre isely, if Æ is an n-ary operation on fun tions and A ; : : : ; A are sets of 1 n

fun tions, then Æ(A ; : : : ; A ) denotes the natural extension to sets of fun tions, e.g.,
1 n

A + A = fa + a ; a 2 A and a 2 A g. Singleton sets are denoted by their

1 2 1 2 1 1 2 2
single member. Then expressions  and  whi h ontain O-expressions denote sets
of fun tions and  =  stands for    . Equalities ontaining O-expressions an
only be read from left to right. The terms of sequen e A = A = A =


= A 1 2 3 k

represent larger and larger sets of fun tions; the bounds be ome oarser and oarser
from left to right.
Version: 18.10.99 Time: 17:53 {31{
32
1.7. Se ondary Storage

On a few o asions we onsider algorithms whi h use se ondary storage. We make

the following assumption. Data is transported in blo ks (pages) between main and
se ondary memory. A page onsists of a xed (say 2 = 1024) number of storage
10

lo ations. It takes 5000 time units to transport a page from one memory to the
other. This assumption approximates the behavior of modern disk memory.

1.8. Exer ises

1) Develop a RAM program for omputing 2 whi h runs in O(log n) time units
n

in the unitP ost measure and O(n) time units in the logarithm- ost measure. [Hint:
Let n = k
a
2 , a 2 f0; 1g, be the binary representation of n. Note that
i
i
i

2 = (


((2 k )
2 k )


2 )
2 .℄
i=0
n a 2 a 1 2 a1 2 a0

2) Give a detailed proof of Theorem 1 of Se tion 1.1. In parti ular, dis uss the
assumption that a RAM program does not use ell 0. Show that one an add 1 to
all addresses dynami ally and hen e free ell 0.
3) Give a detailed proof of Theorem 3 of Se tion 1.1.
4) Show how to translate onditional and iterative statements into RAM ode.
5) In doubly linked lists every list element points to its su essor and its prede es-
sor. Do Se tion 1.4.2. for doubly linked lists.
6) A tree is k-ary if every node has exa tly k sons. Show that a k-ary tree with n
nodes has exa tly k + (k 1)(n 1) leaves.
7) Prove
f (n) = O(f (n))
O(f (n)) + O(f (n)) = O(f (n))

O(f (n)) = O(f (n))
O(f (n))
O(g(n)) = O(f (n)
g(n))

1.9. Bibliographi Notes

RAMs and RASPs were introdu ed by Shepherdson/Sturgis (63) and Elgot/Robin-

son (64). Our notation follows Hotz (72). The theorems in Se tion 1.1. are taken
over from Cook/Re khow (73). The dis ussion of randomized algorithms is based
on Adleman (78), Reif (82), Yao (77) und Yao (82). A more detailed a ount of
linear lists an be found in Maurer (74) and Knuth (68). For a detailed dis ussion
of the implementation of re ursive pro edures we re ommend Gries (71).
Version: 18.10.99 Time: 17:53 {32{