Algorithm Analysis and

Asymptotic Notations

Data Structures Department of Computer Science – University of Zakho 1

 What is the algorithm
 An algorithm is a step-by-step procedure for performing some task
(ex: sorting a set of integers) in a finite amount of time.

Input Algorithm Output

 We are concerned with the following properties:

 Correctness
 Efficiency (how fast it is, how many resources it needs)

Data Structures Department of Computer Science – University of Zakho 2

 Running Time
 Running time is a natural measure of Efficiency.
 So what would be the proper way of measuring it?
 Do experiments, and then find the run time.
 If we have two algorithms for a problem, implement them and do
several experiments on various input size.
 Then decide which algorithm is better.

Data Structures Department of Computer Science – University of Zakho 3

Experimental Studies
9000

8000

7000
 Run the program with inputs of
6000
varying size and composition

Time (ms)
5000

 Use a method like std::clock() to get 4000

an accurate measure of the actual 3000
running time 2000

1000

 Plot the results 0

0 50 100
Input Size

Department of Computer Science – University of Zakho 4

 What is the problem of experimental studies?

 The running time is affected by the hardware (Processor, RAM, etc.)

and software (Compiler, programing language, etc.)

Data Structures Department of Computer Science – University of Zakho 5

 Limitations of Experiments
 Need to implement the algorithm
 may be difficult

 Experiments done on a limited set of test inputs

 may not be indicative of running times on other inputs not included in the
experiment.

 Difficult to compare
 same hardware and software environments must be used

Data Structures Department of Computer Science – University of Zakho 6

 Running time
 We need another way to measure to the running time of an
algorithm which:
 Considers all possible inputs.
 Be independent from hardware and software.

Data Structures Department of Computer Science – University of Zakho 7

 Running Time
 The running time of an algorithm typically best case
average case
grows with the input size. worst case
120

100

 Average case time is often difficult to

Running Time
80

determine. 60

40

20

 We focus on the worst case running time. 0

1000 2000 3000 4000

 Easier to analyze Input Size

 Crucial to applications such as games, finance,

and robotics

Data Structures Department of Computer Science – University of Zakho 8

 Theoretical Analysis
• Uses pseudocode, a high-level description of the algorithm
– no implementation necessary

• Takes into account all possible inputs

• Characterizes running time by f(n), a function of the input size n

– allows us to evaluate the speed of an algorithm independent of
hardware/software environment

Data Structures Department of Computer Science – University of Zakho 9

 Pseudocode

 Mixture of natural language and high-level

programming constructs that describe the Algorithm arrayMax(A, n)
main ideas behind an algorithm Input array A of n integers
implementation. Output maximum element of A
currentMax  A[0]
 Preferred notation for
describing algorithms. for i  1 to n  1 do
if A[i]  currentMax then
 Hides program design issues
currentMax  A[i]
return currentMax

Data Structures Department of Computer Science – University of Zakho 10

 Pseudocode Details
 Control flow  Method call
 if … then … [else …] var.method (arg [, arg…])

 while … do …
 Return value
 repeat … until …
return expression
 for … do …
 Indentation replaces braces  Expressions
 or := Assignment (like  in
C++)
 Method declaration  Equality testing (like  in
C++)
Algorithm method (arg [, arg…])
n2 Superscripts and other
Input … mathematical formatting
allowed
Output …

Data Structures Department of Computer Science – University of Zakho 11

 The Random Access Machine (RAM) Model

• Views a computer as:

– a CPU, with
– a potentially unbounded bank of 2
memory cells, each of which can 1
0
hold an arbitrary number or
character
Memory cells are numbered and accessing any cell in memory
takes unit time.

Random Access refers to ability of CPU to access arbitrary memory

cell with one primitive operation.

Data Structures Department of Computer Science – University of Zakho 12

 Primitive Operations
• Basic computations performed by an algorithm
– Identifiable in pseudocode
– Largely independent from the programming language
– Exact definition not important (we’ll see why later)

• Assumed to take a constant amount of time in the RAM

model
• Includes:
– indexing into an array
– evaluating an expression
– calling a method
– assigning a value to a variable
– returning from a method

Data Structures Department of Computer Science – University of Zakho 13

 Counting Primitive Operations
 By inspecting the Pseudocode, we can determine the maximum
number of primitive operations executed by an algorithm, as a
function of the input size.

Algorithm arrayMax(A, n) Operations

currentMax  A[0] 2
for i  1 to n  1 do 1+n
if A[i]  currentMax then 2(n-1)
currentMax  A[i] 2(n-1)
{ increment counter i } 2(n-1)
return currentMax 1
An algorithm to find the maximum number in array.

Data Structures Department of Computer Science – University of Zakho 14

 Counting Primitive Operations


Analysis of Algorithms 15
Data Structures Department of Computer Science – University of Zakho 15
 Estimating Running Time
 Algorithm arrayMax executes 7n  2 primitive operations in the worst
case.

 Define:
a = time taken by the fastest primitive operation
b = time taken by the slowest primitive operation

 Let T(n) be worst-case time of arrayMax. Then

a(7n  2)  T(n)  b(7n  2)

 Hence, the running time T(n) is bounded by two linear functions.

Data Structures Department of Computer Science – University of Zakho 16

 Growth Rate of Running Time
 Changing the hardware/software environment
 affects T(n) by a constant factor, but
 does not alter the growth rate of T(n)

 The linear growth rate of the running time T(n) is an intrinsic property
of algorithm arrayMax.

Data Structures Department of Computer Science – University of Zakho 17

 Mathematical Review

Data Structures Department of Computer Science – University of Zakho 18

 Growth Rates
Constant 1
Logarithmic  logn
Linear n
Quadratic  n2
Cubic  n3
Polynomial  nk (for k ≥ 1)

Exponential  an (a ≥ 1)

 Growth rate is not affected by

– constant factors or
– lower-order terms

 Ex: 102n + 105 is a linear function

 Ex: 105n2 + 108n is a quadratic function
Data Structures Department of Computer Science – University of Zakho 19
 Asymptotic Complexity
• Worst case running time of an algorithm as a function of input size n
for large n.
• Expressed using only the highest-order term in the expression for the
exact running time.
– Instead of exact running time, say O(n2)

• Written using asymptotic notation (O, , , o, )

– Ex: f(n) = O(n2)
– Describes how f(n) grows in comparison to n2
• The notations describe different rate-of-growth relations between
the defining function and the defined set of functions

Data Structures Department of Computer Science – University of Zakho 20

 O-notation
For functions g(n), we define
O(g(n)), big-O of n, as the set:

O(g(n)) = { f(n) :
∃ positive constants c and n0,
such that ∀n ≥ n0
we have 0  f(n)  cg(n) }

Intuitively: Set of all functions whose rate of growth is the Technically, f(n) ∈ O(g(n)).
same as or lower than that of g(n). Older usage, f(n) = O(g(n)).

g(n) is an asymptotic upper bound for f(n)

Data Structures Department of Computer Science – University of Zakho 21

 Examples
O(g(n)) = { f(n) : ∃ positive constants c and n0,
such that ∀n ≥ n0 , we have 0  f(n)  cg(n) }
• O(n)  O(n2)
 f(n)=7n+3  f(n) = n2 + 1
 f(n) = 2n + 10  f(n) = n2 + n
 f(n) = n + 1  f(n) = 10000n2 +10000n + 300
 f(n) = 10000n  f(n) = n1.99
 f(n) = 10000n + 300

• The function n2 is not O(n)

– the inequality n2  cn cannot be satisfied since c is constant

Data Structures Department of Computer Science – University of Zakho 22

 Big-Oh Rules
 Drop lower-order terms
 Ex: if f(n) is a polynomial of degree d, then f(n) is O(nd)

 Drop constant factors, using the simplest expression of the class

 Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”

 Use the smallest possible class of functions

 Say “2n is O(n)” instead of “2n is O(2n)”

Data Structures Department of Computer Science – University of Zakho 23

 Asymptotic Algorithm Analysis
 The asymptotic analysis of an algorithm determines the running time
in big-Oh notation
 To perform the asymptotic analysis
 Find the worst-case number of primitive operations executed as a function of
the input size
 We express this function with big-Oh notation
 Ex:
 arrayMax executes at most 7n  1 primitive operations
 arrayMax “runs in O(n) time”
 Since constant factors and lower-order terms are eventually
dropped anyhow, we can disregard them when counting primitive
operations

Data Structures Department of Computer Science – University of Zakho 24

 Ex: Computing Prefix Averages

 We further illustrate asymptotic analysis

with two algorithms for prefix averages. 35
X
30 A
 The i-th prefix average of an array X is
average of the first (i + 1) elements of X:
25

A[i]  (X[0] + X[1] + … + X[i])/(i+1) 20

15
 Prefix average has applications in 10
economic and statistics
5
0
1 2 3 4 5 6 7

Data Structures Department of Computer Science – University of Zakho 25

 Prefix Averages V1 O(n2) - Quadratic!

The following algorithm computes prefix averages by

applying the definition
Algorithm prefixAverages1(X, n)
Input array X of n integers
Output array A of prefix averages of X rough # operations
A  new array of n integers n
for i  0 to n  1 do n
s  X[0] n
for j  1 to i do 1 + 2 + … + (n-1)
s  s + X[j] 1 + 2 + … + (n-1)
A[i]  s / (i + 1) n
return A 1

Analysis of Algorithms 26
Data Structures Department of Computer Science – University of Zakho 26
 Prefix Averages V2
O(n) - Linear!
The following algorithm computes prefix averages
by keeping a running sum

Algorithm prefixAverages2(X, n)
Input array X of n integers
rough # operations
Output array A of prefix averages of X
A  new array of n integers n
s0 1
for i  0 to n  1 do n
s  s + X[i] n
A[i]  s / (i + 1) n
return A 1

Analysis of Algorithms 27
Data Structures Department of Computer Science – University of Zakho 27
 Ω-notation
For functions g(n), we define
Ω(g(n)), big-Omega of n, as the
set:
Ω(g(n)) = { f(n) :
∃ positive constants c and n0,
such that ∀n ≥ n0
we have 0  cg(n)  f(n)}

Intuitively: Set of all functions whose rate of growth is the

same as or higher than that of g(n).
g(n) is an asymptotic lower bound for f(n)

Analysis of Algorithms 28
Data Structures Department of Computer Science – University of Zakho 28
 𝛩-notation
For functions g(n), we define
𝛩(g(n)), big-Theta of n, as the set:

𝛩(g(n)) = { f(n) :
∃ positive constants c1, c2, and n0,
such that ∀n ≥ n0
we have 0  c1g(n)  f(n)  c2g(n)}

Intuitively: Set of all functions that have the same rate of

growth as g(n).
g(n) is an asymptotically tight bound for
f(n)
Analysis of Algorithms 29
Data Structures Department of Computer Science – University of Zakho 29
 Relationship between O, Ω, 𝛩

Analysis of Algorithms 30
Data Structures Department of Computer Science – University of Zakho 30