CS2210

Data Structures and Algorithms

Lecture 2:
Analysis of Algorithms
Asymptotic notation

Instructor: Olga Veksler

© 2004 Goodrich, Tamassia

 Outline
 Comparing algorithms
 Pseudocode
 Theoretical Analysis of Running time
 Primitive Operations
 Counting primitive operations
 Asymptotic analysis of running time

Analysis of Algorithms 2
 Comparing Algorithms
 Given 2 or more algorithms to solve the same
problem, how do we select the best one?
 Some criteria for selecting an algorithm
1) Is it easy to implement, understand, modify?
2) How long does it take to run it to completion?
3) How much of computer memory does it use?
 Software engineering is primarily concerned
with the first criteria
 In this course we are interested in the second
and third criteria

Analysis of Algorithms 3
 Comparing Algorithms
 Time complexity
 The amount of time that an algorithm needs to run to
completion
 Space complexity
 The amount of memory an algorithm needs to run
 We will occasionally look at space complexity,
but we are mostly interested in time complexity
in this course
 Thus in this course the better algorithm is the
one which runs faster (has smaller time
complexity)

Analysis of Algorithms 4
 How to Calculate Running time
 Most algorithms transform input objects into output
objects
sorting
5 3 1 2 1 2 3 5
algorithm
input object output object

 The running time of an algorithm typically grows

with the input size
 idea: analyze running time as a function of input size

Analysis of Algorithms 5
 How to Calculate Running Time
 Even on inputs of the same size, running time can
be very different
 Example: algorithm that finds the first prime number in an
array by scanning it left to right
 Idea: analyze running time in the
 best case
 worst case
 average case

Analysis of Algorithms 6
 How to Calculate Running Time

 Best case running time best case

is usually useless average case
worst case
 Average case time is 120

very useful but often 100

difficult to determine

Running Time
80

 We focus on the worst 60

case running time 40

 Easier to analyze 20

 Crucial to applications 0
1000 2000 3000 4000
such as games, finance Input Size
and robotics

Analysis of Algorithms 7
 Experimental Evaluation of Running Time

9000
 Write a program 8000
implementing the algorithm 7000

 Run the program with inputs 6000

Time (ms)
of varying size and 5000

composition 4000

 Use a method like 3000

System.currentTimeMillis() to get 2000

an accurate measure of the 1000

actual running time 0

0 50 100
 Plot the results Input Size

Analysis of Algorithms 8
 Limitations of Experiments
 Experimental evaluation of running time is
very useful but
 It is necessary to implement the algorithm,
which may be difficult
 Results may not be indicative of the running
time on other inputs not included in the
experiment
 In order to compare two algorithms, the same
hardware and software environments must be
used
Analysis of Algorithms 9
 Theoretical Analysis of Running Time

 Uses a pseudo-code description of the

algorithm instead of an implementation
 Characterizes running time as a
function of the input size, n
 Takes into account all possible inputs
 Allows us to evaluate the speed of an
algorithm independent of the
hardware/software environment

Analysis of Algorithms 10
 Pseudocode
 In this course, we will Example: find max
mostly use pseudocode element of an array
to describe an algorithm
 Pseudocode is a high- Algorithm arrayMax(A, n)
level description of an Input: array A of n integers
algorithm Output: maximum element of A
 More structured than
currentMax  A[0]
English prose
for i  1 to n  1 do
 Less detailed than a
if A[i]  currentMax then
program
currentMax  A[i]
 Preferred notation for
return currentMax
describing algorithms
 Hides program design
issues
Analysis of Algorithms 11
 Pseudocode Details
Algorithm arrayMax(A, n)
 Control flow Input: array A of n integers
 if … then … [else …] Output: maximum element of A
 while … do …
 repeat … until … currentMax  A[0]
for i  1 to n  1 do
 for … do …
if A[i]  currentMax then
 Indentation replaces braces currentMax  A[i]
return currentMax
 Method declaration
Algorithm method (arg, arg…)
Input …
Output …

Analysis of Algorithms 12
 Pseudocode Details
Algorithm arrayMax(A, n)
 Method call
Input: array A of n integers
var.method (arg [, arg…]) Output: maximum element of A
 Return value currentMax  A[0]
return expression for i  1 to n  1 do
 Expressions if A[i]  currentMax then
currentMax  A[i]
 Assignment
return currentMax
(like  in Java)
 Equality testing
(like  in Java)
n2 superscripts and other
mathematical
formatting allowed

Analysis of Algorithms 13
 Primitive Operations
 For theoretical analysis, we will count primitive
or basic operations, which are simple
computations performed by an algorithm
 Basic operations are:
 Identifiable in pseudocode
 Largely independent from the programming
language
 Exact definition not important (we will see why
later)
 Assumed to take a constant amount of time

Analysis of Algorithms 14
 Primitive Operations
 Examples of primitive operations:
 Evaluating an expression x2+ey
 Assigning a value to a variable cnt  cnt+1
 Indexing into an array A[5]
 Calling a method mySort(A,n)
 Returning from a method return(cnt)

Analysis of Algorithms 15
 Counting Primitive Operations
 By inspecting the pseudocode, we can determine the
maximum number of primitive operations executed by
an algorithm, as a function of the input size

Algorithm arrayMax(A, n) # operations

currentMax  A[0] 2
for i  1 to n  1 do 2+n
if A[i]  currentMax then 2(n  1)
currentMax  A[i] 2(n  1)
{ increment counter i } 2(n  1)
return currentMax 1
Total 7n  1

Analysis of Algorithms 16
 Estimating Running Time

 Algorithm arrayMax executes 7n  1 primitive

operations in the worst case. Define:
a = Time taken by the fastest primitive operation
b = Time taken by the slowest primitive operation
 Let T(n) be worst-case time of arrayMax. Then
a (7n  1)  T(n)  b(7n  1)
 Hence, the running time T(n) is bounded by two
linear functions

Analysis of Algorithms 17
 Growth Rate of Running Time
 Changing the hardware/software environment
 Affects T(n) by a constant factor, but
 Does not alter the growth rate of T(n)
 Thus we focus on the big-picture which is the
growth rate of an algorithm
 The linear growth rate of the running time T(n) is
an intrinsic property of algorithm arrayMax
 algorithm arrayMax grows proportionally with n, with
its true running time being n times a constant factor
that depends on the specific computer

Analysis of Algorithms 18
 Constant Factors
 The growth rate is not affected by
 constant factors or
 lower-order terms
 Examples
 102n + 105 is a linear function
 105n2 + 108n is a quadratic function
 How do we get rid of the constant factors to
focus on the essential part of the running time?

Analysis of Algorithms 19
 Big-Oh Notation Motivation
 Or asymptotic analysis
 The big-Oh notation is used widely to
characterize running times and space bounds
 The big-Oh notation allows us to ignore
constant factors and lower order terms and
focus on the main components of a function
which affect its growth

Analysis of Algorithms 20
 Big-Oh Notation Definition
 Given functions f(n) and 80
g(n), we say that f(n) is 70 3n
O(g(n)) if there are 60 2n+10
positive constants 50
n
c and n0 such that 40
f(n)  cg(n) for n  n0 30

 Example: 2n + 10 is O(n) 20
10
 2n + 10  cn
0
 (c  2) n  10
0 5 10 15 20 25 30
 n  10/(c  2) n
 Pick c  3 and n0  10

Analysis of Algorithms 21
 Big-Oh Example
 Example: the function 100,000
n2 is not O(n) 90,000
n^2
100n
 n2  cn 80,000 10n
 nc 70,000 n
 The above inequality 60,000
cannot be satisfied 50,000
since c must be a
40,000
constant
30,000
20,000
10,000
0
0 100 200 300 400 500
n

Analysis of Algorithms 22
 More Big-Oh Examples
 7n-2
7n-2 is O(n)
need c > 0 and n0  1 such that 7n-2  c•n for n  n0
this is true for c = 7 and n0 = 1

 3n3 + 20n2 + 5
3n3 + 20n2 + 5 is O(n3)
need c > 0 and n0  1 such that 3n3 + 20n2 + 5  c•n3 for n  n0
this is true for c = 4 and n0 = 21

 3 log n + 5
3 log n + 5 is O(log n)
need c > 0 and n0  1 such that 3 log n + 5  c•log n for n  n0
this is true for c = 8 and n0 = 2

Analysis of Algorithms 23
 Big-Oh and Growth Rate
 The big-Oh notation gives an upper bound on the
growth rate of a function
 The statement “f(n) is O(g(n))” means that the growth
rate of f(n) is no more than the growth rate of g(n)
 We can use the big-Oh notation to rank functions
according to their growth rate

f(n) is O(g(n)) g(n) is O(f(n))

g(n) grows more Yes No
f(n) grows more No Yes
Same growth Yes Yes

Analysis of Algorithms 24
 Big-Oh Rules
f n   a0 + a1n + a2 n 2 + ... + ad n d
 If is f(n) a polynomial of degree d, then f(n) is
O(nd), i.e.,
1. Drop lower-order terms
2. Drop constant factors
 Use the smallest possible class of functions
 Say “2n is O(n)” instead of “2n is O(n2)”
 Use the simplest expression of the class
 Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”

Analysis of Algorithms 25
 Asymptotic Algorithm Analysis
 The asymptotic analysis of an algorithm determines the
running time in big-Oh notation
 To perform the asymptotic analysis
 We find the worst-case number of primitive operations
executed as a function of the input size
 We express this function with big-Oh notation
 Example:
 We determine that algorithm arrayMax executes at most
7n  1 primitive operations
 We say that algorithm arrayMax “runs in O(n) time”
 Since constant factors and lower-order terms are
eventually dropped anyhow, we can disregard them
when counting primitive operations

Analysis of Algorithms 26
 Important Functions

 Often appear in algorithm analysis:

 Constant  1
 Logarithmic  log n
 Linear  n
 N-Log-N  n log n
 Quadratic  n2
 Cubic  n3
 Exponential  2n

Analysis of Algorithms 27
 Important Functions Growth Rates
n log(n) n nlog(n) n2 n3 2n
8 3 8 24 64 512 256

16 4 16 64 256 4096 65536

32 5 32 160 1024 32768 4.3x109

64 6 64 384 4096 262144 1.8x1019

128 7 128 896 16384 2097152 3.4x1038

256 8 256 2048 65536 16777218 1.2x1077

Analysis of Algorithms 28
 Growth Rates Illustration
Running Maximum Problem Size (n)
Time in ms
(10-3 of sec) 1000 ms 60000 ms 36*105 m
(1 second) (1 minute) (1 hour)
n 1000 60,000 3,600,000

n2 32 245 1,897

2n 10 16 22

Analysis of Algorithms 29
 Useful Big-Oh Rules
 If is f(n) a polynomial of degree d, then f(n) is
O(nd)

 If d(n) is O(f(n)) and e(n) is O(g(n)) then

 d(n)+e(n) is O(f(n)+g(n))
 d(n)e(n) is O(f(n) g(n))
 If d(n) is O(f(n)) and f(n) is O(g(n)) then d(n) is
O(g(n))
 If p(n) is a polynomial in n then log p(n) is
O(log(n))

Analysis of Algorithms 30
 Computing Prefix Averages
 We further illustrate
asymptotic analysis with 35
two algorithms for prefix X
30
averages A
 The i-th prefix average of
25
an array X is average of the 20
first (i + 1) elements of X:
15
A[i]  X[0] + X[1] + … + X[i])/(i+1)
10
 Computing the array A of 5
prefix averages of another
array X has applications to 0
financial analysis 1 2 3 4 5 6 7

Analysis of Algorithms 31
Prefix Averages (Quadratic)
 The following algorithm computes prefix averages in
quadratic time by applying the definition

Algorithm prefixAverages1(X, n)
Input array X of n integers
Output array A of prefix averages of X
#operations
A  new array of n integers n
for i  0 to n  1 do n
s  X[0] n
for j  1 to i do 1 + 2 + …+ (n  1)
s  s + X[j] 1 + 2 + …+ (n  1)
A[i]  s / (i + 1) n
return A 1
Analysis of Algorithms 32
 Arithmetic Progression

 The running time of 7

prefixAverages1 is 6
O(1 + 2 + …+ n)
5
 The sum of the first n
integers is n(n + 1) / 2 4
 There is a simple visual 3
proof of this fact
2
 Thus, algorithm
prefixAverages1 runs in 1
O(n2) time 0
1 2 3 4 5 6

Analysis of Algorithms 33
 Prefix Averages (Linear)
 The following algorithm computes prefix averages in
linear time by keeping a running sum
Algorithm prefixAverages2(X, n)
Input array X of n integers
Output array A of prefix averages of X
#operations
A  new array of n integers n
s0 1
for i  0 to n  1 do n
s  s + X[i] n
A[i]  s / (i + 1) n
return A 1

 Algorithm prefixAverages2 runs in O(n) time

Analysis of Algorithms 34
 More Examples

Algorithm SumTripleArray(X, n)
Input triple array X[ ][ ][ ] of n by n by n integers
Output sum of elements of X #operations
s0 1
for i  0 to n  1 do n
for j  0 to n  1 do n+n+…+n=n2
for k  0 to n  1 do n2+n2+…+n2 = n3
s  s + X[i][j][k] n2+n2+…+n2 = n3
return s 1

 Algorithm SumTripleArray runs in O(n3) time

Analysis of Algorithms 35
 Relatives of Big-Oh

 big-Omega
 f(n) is (g(n)) if there is a constant c > 0

and an integer constant n0  1 such that

f(n)  c•g(n) for n  n0

 big-Theta
 f(n) is (g(n)) if there are constants c’ > 0 and c’’
> 0 and an integer constant n0  1 such that
c’•g(n)  f(n)  c’’•g(n) for n  n0

Analysis of Algorithms 36
Intuition for Asymptotic Notation

Big-Oh
 f(n) is O(g(n)) if f(n) is asymptotically less than or

equal to g(n)
big-Omega
 f(n) is (g(n)) if f(n) is asymptotically greater than
or equal to g(n)
 Note that f(n) is (g(n)) if and only if g(n) is O(f(n))

big-Theta
 f(n) is (g(n)) if f(n) is asymptotically equal to g(n)

 Note that f(n) is (g(n)) if and only if if g(n) is

O(f(n)) and if f(n) is O(g(n))

Analysis of Algorithms 37
Example Uses of the Relatives of Big-Oh
 5n2 is (n2)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0  1
such that f(n)  c•g(n) for n  n0
let c = 5 and n0 = 1
 5n2 is (n)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0  1
such that f(n)  c•g(n) for n  n0
let c = 1 and n0 = 1
 5n2 is (n2)
f(n) is (g(n)) if it is (n2) and O(n2). We have already seen the former,
for the latter recall that f(n) is O(g(n)) if there is a constant c > 0 and an
integer constant n0  1 such that f(n) < c•g(n) for n  n0
Let c = 5 and n0 = 1

Analysis of Algorithms 38
 Math you need to Review
 Summations
 Logarithms and Exponents
 properties of logarithms:
logb(xy) = logbx + logby
logb (x/y) = logbx - logby
logbxa = alogbx
logba = logxa/logxb
 properties of exponentials:
a(b+c) = aba c
abc = (ab)c
ab /ac = a(b-c)
b = a logab
bc = a c*logab

Analysis of Algorithms 39
 Final Notes
Running time
 Even though in this course we
focus on the asymptotic growth
A
using big-Oh notation,
practitioners do care about
constant factors occasionally
B
 Suppose we have 2 algorithms
 Algorithm A has running time 30000n
 Algorithm B has running time 3n2 10000
 Asymptotically, algorithm A is problem size
better than algorithm B
 However, if the problem size you
deal with is always less than
10000, then the quadratic one is
faster

Analysis of Algorithms 40