Introduction to Asymptotic

Analysis
CSE 2215 - Lecture 2
Instructor : Fahmid Al Rifat, Lecturer, Dept. of CSE , UIU

[email protected]
1
Today’s Goals
• Discuss Runtime of Programs.
• Compute and Classify growth of functions.
• Analyze complexity classes for algorithms.

2
What is an Algorithm?

An algorithm is a sequence of computational steps that solves a

well-specified computational problem.
• An algorithm is said to be correct if, for every input instance,
it halts with the correct output
• An incorrect algorithm might not halt at all on some input
instances, or it might halt with other than the desired output.

4
 What is a Program?

A program is the expression of an algorithm in a

programming language
A set of instructions which the computer will follow to
solve a problem

5
Define a Problem, and Solve It

Problem:
Description of Input-Output relationship
Algorithm:
A sequence of computational steps that transform the input into
the output.
Data Structure:
An organized method of storing and retrieving data.
Our Task:
Given a problem, design a correct and good algorithm that solves
it.

6
Define a Problem, and Solve It

Problem: Input is a sequence of integers stored in an array.

Output the minimum.

INPUT Algorithm

OUTPUT
instance m:= a[1];
for I:=2 to size of input
25, 90, 53, 23, 11, 34 if m > a[i] then 11
m:=a[I];
return m

m, a[i] Data-Structure

7
What do we Analyze?

o Correctness
o Does the input/output relation match algorithm
requirement?
o Amount of work done (complexity)
o Basic operations to do task
o Amount of space used
o Memory used
o Simplicity, clarity
o Verification and implementation.
o Optimality
o Is it impossible to do better?

8
Running Time

Number of primitive steps that are executed

Except for time of executing a function call most statements roughly require the
same amount of time
y=m*x+b
c = 5 / 9 * (t - 32 )
z = f(x) + g(y)
We can be more exact if need to be

9
An Example: Insertion Sort

10
An Example: Insertion Sort

A = {5, 2, 4, 6, 1, 3}

11
An Example: Insertion Sort

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

12
An Example: Insertion Sort

How many times will

InsertionSort(A, n) {
this loop execute?
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

13
 Analyzing Insertion Sort

Statement Cost Times

InsertionSort(A, n) {
for i = 2 to n { c1 n
key = A[i] c2 (n – 1)
j = i – 1; c3 (n – 1)
while (j > 0) and (A[j] > key) { c4 T
A[j+1] = A[j] c5 (T – (n – 1))
j=j–1} c6 (T – (n – 1))
A[j+1] = key c7 (n – 1)
}
}
T = t2 + t3 + … + tn, where ti is the number of while expression
evaluations for the ith for loop iteration
14
Analyzing Insertion Sort

T(n) = c1n + c2(n-1) + c3(n-1) + c4T + c5(T - (n-1)) + c6(T - (n-1)) + c7(n-1)
= c8T + c9n + c10

What can T be?

Best case: the array is sorted (inner loop body never executed)
ti = 1  T = n
T(n) = an + b, a linear function of n
Worst case: the array is reverse sorted (inner loop body executed for all
previous elements)
ti = i  T = n(n + 1)/2 - 1
T(n) = an2 + bn + c, a quadratic function of n
Average case:
???

15
Asymptotic Performance

We care most about asymptotic performance

• How does the algorithm behave as the problem size gets
very large?
o Running time
o Memory/storage requirements
o Bandwidth/power requirements/logic gates/etc.

16
Asymptotic Analysis

Worst case
o Provides an upper bound on running time
o An absolute guarantee of required resources
Average case
o Provides the expected running time
o Very useful, but treat with care: what is “average”?
o Random (equally likely) inputs
o Real-life inputs
Best case
o Provides a lower bound on running time

17
Upper Bound Notation

We say InsertionSort’s run time is O(n2)

Properly we should say run time is in O(n2)
Read O as “Big-O” (you’ll also hear it as “order”)
In general a function
f(n) is O(g(n)) if there exist positive constants c and n0 such
that 0  f(n)  c  g(n) for all n  n0
Formally
O(g(n)) = { f(n):  positive constants c and n0 such that 0 
f(n)  c  g(n)  n  n0 }

18
Upper Bound Notation

time
c.g(n)

f(n)

n0 n

We say g(n) is an asymptotic upper bound for f(n)

19
Insertion Sort is O(n2)

Proof
The run-time is an2 + bn + c
If any of a, b, and c are less than 0, replace the constant with
its absolute value
an2 + bn + c  (a + b + c)n2 + (a + b + c)n + (a + b + c)
 3(a + b + c)n2 for n  1
Let c’ = 3(a + b + c) and let n0 = 1. Then
an2 + bn + c  c’ n2 for n  1
Thus an2 + bn + c = O(n2).
Question
Is InsertionSort O(n3) ?
Is InsertionSort O(n) ?

20
 Lower Bound Notation

We say InsertionSort’s run time is (n)

In general a function
f(n) is (g(n)) if  positive constants c and n0 such that
0  cg(n)  f(n)  n  n0
Proof:
Suppose run time is an + b
Assume a and b are positive
an  an + b

21
Lower Bound Notation

time
f(n)

c.g(n)

n0 n

We say g(n) is an asymptotic lower bound for f(n)

22
Asymptotic Tight Bound

A function f(n) is (g(n)) if  positive constants c1, c2, and n0 such

that

0  c1 g(n)  f(n)  c2 g(n)  n  n0

Theorem
f(n) is (g(n)) iff f(n) is both O(g(n)) and (g(n))
Proof:

23
Asymptotic Tight Bound

c2.g(n) f(n)
time

c1.g(n)

n0 n
We say g(n) is an asymptotic tight bound for f(n)
24
 Practical Complexity

For large input sizes, constant terms are insignificant

Program A with running time TA(n)= 100n
Program B with running time TB(n)= 2n2
TP(n)

TB(n) = 2n2

TA(n) = 100n
5000

Input Size n
50
25
Practical Complexity

250

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

26
Practical Complexity

500

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

27
Practical Complexity

1000

f(n) = n
f(n) = log(n)
f(n) = n log(n)
f(n) = n^2
f(n) = n^3
f(n) = 2^n

0
1 3 5 7 9 11 13 15 17 19

28
Practical Complexity

5000

4000
f(n) = n
f(n) = log(n)
3000
f(n) = n log(n)
f(n) = n^2
2000 f(n) = n^3
f(n) = 2^n

1000

0
1 3 5 7 9 11 13 15 17 19

29
Practical Complexity

f(n) = n
10000000 f(n) = log(n)
f(n) = n log(n)
1000000
f(n) = n^2
f(n) = n^3
100000
f(n) = 2^n
10000

1000

100

10

1
1 4 16 64 256 1024 4096 16384 65536

30
Practical Complexity

Function Descriptor Big-Oh

c Constant O( 1 )
log n Logarithmic O( log n )
n Linear O( n )
n log n n log n O( n log n )
n2 Quadratic O( n2 )
n3 Cubic O( n3 )
nk Polynomial O( nk )
2n Exponential O( 2n )
n! Factorial O( n! )
Other Asymptotic Notations

A function f(n) is o(g(n)) if  positive constants c and n0

such that
f(n) < c g(n)  n  n0
A function f(n) is (g(n)) if  positive constants c and n0
such that
c g(n) < f(n)  n  n0

Intuitively,
■ o( ) is like < ■ ( ) is like > ■ ( ) is like =
■ O( ) is like  ■ ( ) is like 

32
Thank You

33