i i

“main” — 2011/1/13 — 9:10 — page 162 — #184

i i

162 Chapter 4. Analysis Tools

4.2 Analysis of Algorithms

In a classic story, the famous mathematician Archimedes was asked to determine if

a golden crown commissioned by the king was indeed pure gold, and not part silver,
as an informant had claimed. Archimedes discovered a way to perform this analysis
while stepping into a (Greek) bath. He noted that water spilled out of the bath in
proportion to the amount of him that went in. Realizing the implications of this
fact, he immediately got out of the bath and ran naked through the city shouting,
“Eureka, eureka!,” for he had discovered an analysis tool (displacement), which,
when combined with a simple scale, could determine if the king’s new crown was
good or not. That is, Archimedes could dip the crown and an equal-weight amount
of gold into a bowl of water to see if they both displaced the same amount. This
discovery was unfortunate for the goldsmith, however, for when Archimedes did
his analysis, the crown displaced more water than an equal-weight lump of pure
gold, indicating that the crown was not, in fact, pure gold.
In this book, we are interested in the design of “good” data structures and algo-
rithms. Simply put, a data structure is a systematic way of organizing and access-
ing data, and an algorithm is a step-by-step procedure for performing some task in
a finite amount of time. These concepts are central to computing, but to be able to
classify some data structures and algorithms as “good,” we must have precise ways
of analyzing them.
The primary analysis tool we use in this book involves characterizing the run-
ning times of algorithms and data structure operations, with space usage also being
of interest. Running time is a natural measure of “goodness,” since time is a pre-
cious resource—computer solutions should run as fast as possible.
In general, the running time of an algorithm or data structure method increases
with the input size, although it may also vary for different inputs of the same size.
Also, the running time is affected by the hardware environment (as reflected in the
processor, clock rate, memory, disk, etc.) and software environment (as reflected in
the operating system, programming language, compiler, interpreter, etc.) in which
the algorithm is implemented, compiled, and executed. All other factors being
equal, the running time of the same algorithm on the same input data is smaller if
the computer has, say, a much faster processor or if the implementation is done in a
program compiled into native machine code instead of an interpreted implementa-
tion run on a virtual machine. Nevertheless, in spite of the possible variations that
come from different environmental factors, we would like to focus on the relation-
ship between the running time of an algorithm and the size of its input.
We are interested in characterizing an algorithm’s running time as a function of
the input size. But what is the proper way of measuring it?

i i

i i
 i i

“main” — 2011/1/13 — 9:10 — page 163 — #185

i i

4.2. Analysis of Algorithms 163

4.2.1 Experimental Studies

If an algorithm has been implemented, we can study its running time by executing
it on various test inputs and recording the actual time spent in each execution. For-
tunately, such measurements can be taken in an accurate manner by using system
calls that are built into the language or operating system (for example, by using the
clock() function or calling the run-time environment with profiling enabled). Such
tests assign a specific running time to a specific input size, but we are interested in
determining the general dependence of running time on the size of the input. In or-
der to determine this dependence, we should perform several experiments on many
different test inputs of various sizes. Then we can visualize the results of such
experiments by plotting the performance of each run of the algorithm as a point
with x-coordinate equal to the input size, n, and y-coordinate equal to the running
time, t. (See Figure 4.3.) From this visualization and the data that supports it, we
can perform a statistical analysis that seeks to fit the best function of the input size
to the experimental data. To be meaningful, this analysis requires that we choose
good sample inputs and test enough of them to be able to make sound statistical
claims about the algorithm’s running time.

t (ms)
60

50

40

30

20

10

n
0 50 100

Figure 4.3: Results of an experimental study on the running time of an algorithm.

A dot with coordinates (n,t) indicates that on an input of size n, the running time
of the algorithm is t milliseconds (ms).

i i

i i
 i i

“main” — 2011/1/13 — 9:10 — page 164 — #186

i i

164 Chapter 4. Analysis Tools

While experimental studies of running times are useful, they have three major
limitations:
• Experiments can be done only on a limited set of test inputs; hence, they
leave out the running times of inputs not included in the experiment (and
these inputs may be important).
• We have difficulty comparing the experimental running times of two algo-
rithms unless the experiments were performed in the same hardware and
software environments.
• We have to fully implement and execute an algorithm in order to study its
running time experimentally.
This last requirement is obvious, but it is probably the most time consuming aspect
of performing an experimental analysis of an algorithm. The other limitations im-
pose serious hurdles too, of course. Thus, we would ideally like to have an analysis
tool that allows us to avoid performing experiments.
In the rest of this chapter, we develop a general way of analyzing the running
times of algorithms that:
• Takes into account all possible inputs.
• Allows us to evaluate the relative efficiency of any two algorithms in a way
that is independent from the hardware and software environment.
• Can be performed by studying a high-level description of the algorithm with-
out actually implementing it or running experiments on it.
This methodology aims at associating, with each algorithm, a function f (n) that
characterizes the running time of the algorithm as a function of the input size n.
Typical functions that are encountered include the seven functions mentioned ear-
lier in this chapter.

4.2.2 Primitive Operations

As noted above, experimental analysis is valuable, but it has its limitations. If
we wish to analyze a particular algorithm without performing experiments on its
running time, we can perform an analysis directly on the high-level pseudo-code
instead. We define a set of primitive operations such as the following:
• Assigning a value to a variable
• Calling a function
• Performing an arithmetic operation (for example, adding two numbers)
• Comparing two numbers
• Indexing into an array
• Following an object reference
• Returning from a function

i i

i i
 i i

“main” — 2011/1/13 — 9:10 — page 165 — #187

i i

4.2. Analysis of Algorithms 165

Counting Primitive Operations

Specifically, a primitive operation corresponds to a low-level instruction with an ex-

ecution time that is constant. Instead of trying to determine the specific execution
time of each primitive operation, we simply count how many primitive operations
are executed, and use this number t as a measure of the running time of the algo-
rithm.

This operation count correlates to an actual running time in a specific computer,

since each primitive operation corresponds to a constant-time instruction, and there
are only a fixed number of primitive operations. The implicit assumption in this
approach is that the running times of different primitive operations is fairly similar.
Thus, the number, t, of primitive operations an algorithm performs is proportional
to the actual running time of that algorithm.

An algorithm may run faster on some inputs than it does on others of the same
size. Thus, we may wish to express the running time of an algorithm as the function
of the input size obtained by taking the average over all possible inputs of the same
size. Unfortunately, such an average-case analysis is typically quite challenging.
It requires us to define a probability distribution on the set of inputs, which is often
a difficult task. Figure 4.4 schematically shows how, depending on the input distri-
bution, the running time of an algorithm can be anywhere between the worst-case
time and the best-case time. For example, what if inputs are really only of types
“A” or “D”?

5 ms worst-case time

4 ms

} average-case time?
Running Time

3 ms
best-case time
2 ms

1 ms

A B C D E F G
Input Instance

Figure 4.4: The difference between best-case and worst-case time. Each bar repre-
sents the running time of some algorithm on a different possible input.

i i

i i
 i i

“main” — 2011/1/13 — 9:10 — page 166 — #188

i i

166 Chapter 4. Analysis Tools

Focusing on the Worst Case
An average-case analysis usually requires that we calculate expected running times
based on a given input distribution, which usually involves sophisticated probability
theory. Therefore, for the remainder of this book, unless we specify otherwise, we
characterize running times in terms of the worst case, as a function of the input
size, n, of the algorithm.
Worst-case analysis is much easier than average-case analysis, as it requires
only the ability to identify the worst-case input, which is often simple. Also, this
approach typically leads to better algorithms. Making the standard of success for
an algorithm to perform well in the worst case necessarily requires that it does well
on every input. That is, designing for the worst case leads to stronger algorithmic
“muscles,” much like a track star who always practices by running up an incline.

4.2.3 Asymptotic Notation

In general, each basic step in a pseudo-code description or a high-level language
implementation corresponds to a small number of primitive operations (except for
function calls, of course). Thus, we can perform a simple analysis of an algorithm
written in pseudo-code that estimates the number of primitive operations executed
up to a constant factor, by pseudo-code steps (but we must be careful, since a single
line of pseudo-code may denote a number of steps in some cases).
In algorithm analysis, we focus on the growth rate of the running time as a
function of the input size n, taking a “big-picture” approach. It is often enough just
to know that the running time of an algorithm such as arrayMax, shown in Code
Fragment 4.1, grows proportionally to n, with its true running time being n times a
constant factor that depends on the specific computer.
We analyze algorithms using a mathematical notation for functions that disre-
gards constant factors. Namely, we characterize the running times of algorithms by
using functions that map the size of the input, n, to values that correspond to the
main factor that determines the growth rate in terms of n. This approach allows us
to focus on the “big-picture” aspects of an algorithm’s running time.
Algorithm arrayMax(A, n):
Input: An array A storing n ≥ 1 integers.
Output: The maximum element in A.
currMax ← A[0]
for i ← 1 to n − 1 do
if currMax < A[i] then
currMax ← A[i]
return currMax
Code Fragment 4.1: Algorithm arrayMax.

i i

i i