Computer Science and Engineering| Indian Institute of Technology Kharagpur

cse.iitkgp.ac.in

Algorithms – I (CS21203)

Autumn 2023, IIT Kharagpur

Analysis of Algorithms
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Analyzing Algorithms
• Predict how your algorithm performs in practice
• By analyzing several candidate algorithms for a problem we can
identify efficient ones
• Criteria
• Running time
• Space usage
• Cache I/O
• Main memory I/O
• Lines of codes

Aug 03, 2023 2

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Analyzing Running Time

• Random Access Machine (RAM) model
• Every operation including memory access, arithmetic operations
etc. takes same amount of time.
• Is it precise?
• Not really. But precise model would be tedious – would yield very
little insight into algorithm design and analysis
• However, we should be careful not to abuse it
• We only care about “Order of the cost”, i.e., we omit
• Lower order terms
• Constants
Asymptotic Analysis

Aug 03, 2023 3

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Why Asymptotic Analysis

• Because we only care about how fast a function grows!

• Would you rather have a million rupees one time or one paisa on
day one, doubled every day for a month?

• Actually, the second option can get you more than 1 million (in
around 27 days itself)

• paisa = 1.342 million (for )

Aug 03, 2023 4

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Running Time Analysis of Insertion Sort

• Lets go back to insertion sort and see its running time
• Our expression will evolve from a messy formula that assumes each
line of the code takes a constant amount of time

denotes the
number of times
line 5 gets
executed for that
value of .

Aug 03, 2023 5

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Running Time Analysis of Insertion Sort

• Best case: (Array is already sorted) ->

• -> a linear function of 1
• Worst case: (Array is reverse sorted) ->
• is maximum each time, i.e.,
• -> Thus a quadratic function of

Aug 03, 2023 6

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Best/Average/Worst Case Analysis

• We looked at both `best case’ (input array was already sorted) and
`worst case’ (input array was reverse sorted)

• In this course, we shall usually concentrate on worst-case running

time

• Major reasons
• Gives an upper bound on the running time for any input
• For some algorithms, worst case occurs fairly often, e.g., searching
• Average case is often roughly as bad as the worst case.

Aug 03, 2023 7

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Order of Growth
• In analyzing running time for `insertion sort’, we started with
constants to represent the cost of each statement
• Then we observed that they give more detail than we need and we
discarded them
• We shall go ahead with more simplifying abstraction: Rate/Order
of Growth
• For the function we care when is large enough. When is small, is
small anyway
• The constant factors and lower order terms doesn’t affect the
growth of the function
• One algorithm is more efficient than another if its worst-case
running time has a lower order of growth

Aug 03, 2023 8

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Order of Growth

𝑛
𝑔 2 ( 𝑛 ) =0.1 × 2
2
𝑔 1 ( 𝑛 ) =𝑛
𝑓 2 ( 𝑛 )=9 𝑛
𝑓 1 (𝑛 )=3 𝑛

Omit the constant factors

When is large enough, will be much larger than or

and will have similar growth trend

Aug 03, 2023 9

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Order of Growth

2
𝑔 1 ( 𝑛 ) =𝑛
2
𝑔 2 ( 𝑛 ) =𝑛 − 6 𝑛
𝑓 ( 𝑛 )=6 𝑛

Omit the lower-order

terms

When is large enough, or will still be much larger than

and will have similar growth trend because is much smaller compared to

Aug 03, 2023 10

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Asymptotic Notations
• These notations are used to describe the asymptotic running time
of an algorithm
• However, asymptotic notations can apply to other functions that
have nothing to do whatsoever with algorithms

O (Big-O)
𝑂 ( 𝑔 ( 𝑛 ) )={ 𝑓 ( 𝑛 ) : ∃𝑐> 0 , 𝑛0 > 0 , 𝑠𝑢𝑐h 𝑡h𝑎𝑡 0 ≤ 𝑓 (𝑛)≤ 𝑐𝑔 ( 𝑛 ) ∀ 𝑛 ≥ 𝑛0 }
After some constant • Asymptotic upper bound
𝑓 ( 𝑛 ) ≤ 𝑐𝑔(𝑛) • Note the : can be of the
same order, but can be
smaller as well

( 2
) 2
( 2
) 3 2
𝑛=𝑂 𝑛 , 𝑛 =𝑂 𝑛 ,𝑛 ≠ 𝑂(𝑛 )
Aug 03, 2023 11
CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

O (Big-O) []
𝑂 ( 𝑔 ( 𝑛 ) )={ 𝑓 ( 𝑛 ) : ∃𝑐> 0 , 𝑛0 > 0 , 𝑠𝑢𝑐h 𝑡h𝑎𝑡 0 ≤ 𝑓 (𝑛)≤ 𝑐𝑔 ( 𝑛 ) ∀ 𝑛 ≥ 𝑛0 }
After some constant
𝑓 ( 𝑛 ) ≤ 𝑐𝑔(𝑛)
• How can we show,
• Let
• , so
• Similarly, let
Always larger than
• , so

• ; How can we show, ?

• Let ;
• so

Aug 03, 2023 12

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Ω (Big- Ω) []
Ω ( 𝑔 ( 𝑛 ) ) ={ 𝑓 ( 𝑛 ) : ∃𝑐 >0 ,𝑛 0 >0 , 𝑠𝑢𝑐h 𝑡h𝑎𝑡 0 ≤ 𝑐𝑔( 𝑛) ≤ 𝑓 (𝑛) ∀ 𝑛 ≥ 𝑛 0 }

After some constant

𝑓 ( 𝑛 ) ≥ 𝑐𝑔(𝑛)
• Asymptotic lower bound
• can be of the same order,
but can be larger as well

𝑛 ≠ Ω ( 𝑛2 ) , 𝑛2 =Ω ( 𝑛2 ) , 𝑛3 =Ω(𝑛 2)
Aug 03, 2023 13
CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Θ (Big- theta) []
Θ ( 𝑔 ( 𝑛 ) )={ 𝑓 ( 𝑛 ) :∃𝑐 1 , 𝑐 2 >0 , 𝑛0 >0 , 𝑠 . 𝑡 . 0 ≤ 𝑐 1 𝑔 (𝑛)≤ 𝑓 (𝑛)≤ 𝑐 2 𝑔 ( 𝑛 ) ∀ 𝑛≥ 𝑛0 }

After some constant • Asymptotic tight bound

must be
within this range • Must be of the same order
• means and ,
must be =

𝑛 ≠ Θ ( 𝑛2 ) , 𝑛2=Θ ( 𝑛 2 ) ,𝑛 3 ≠ Θ(𝑛 2)
Aug 03, 2023 14
CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

What This also Means

• O(g(n)): class of functions f(n) that grow no faster than g(n)

• Θ(g(n)): class of functions f(n) that grow at same rate as g(n)

• Ω(g(n)): class of functions f(n) that grow at least as fast as g(n)

Aug 03, 2023 15

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Analogy to Real Numbers

functions Real Numbers

Aug 03, 2023 16

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

o (small-o) [] and ⍵ (small ⍵)

𝑜 ( 𝑔 ( 𝑛 ) )={ 𝑓 ( 𝑛 ) : 𝑓𝑜𝑟 𝑎𝑛𝑦 𝑐>0 , ∃𝑛0 >0 , 𝑠𝑢𝑐h 𝑡h𝑎𝑡 0 ≤ 𝑓 (𝑛)<𝑐𝑔 ( 𝑛 ) ∀ 𝑛≥𝑛 0 }
• Asymptotic non-tight upper bound
• Note the : must be smaller
• Equivalently, if

• By analogy,
𝜔 ( 𝑔 ( 𝑛 ) )={ 𝑓 ( 𝑛 ) : 𝑓𝑜𝑟 𝑎𝑛𝑦𝑐 >0 , ∃𝑛 0>0 , 𝑠𝑢𝑐h 𝑡h𝑎𝑡 0 ≤ 𝑐𝑔(𝑛)< 𝑓 (𝑛)∀ 𝑛≥ 𝑛0 }
• Asymptotic non-tight lower bound
• Note the : must be larger
• Equivalently, if

Aug 03, 2023 17

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Analogy to Real Numbers

functions Real Numbers

Aug 03, 2023 18

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Theorem
• If t1(n) = O(g1(n)) and t2(n) = O(g2(n)), then t1(n) + t2(n) =
O(max{g1(n), g2(n)})
• The algorithm’s overall efficiency will be determined by the
part with a larger order of growth, i.e., its least efficient part.
• For example, 5n2 + 3nlogn = O(n2)

Proof. There exist constants such that

Define and , then

Aug 03, 2023 19

CS21203 / Algorithms - I | Introduction
Computer Science and Engineering| Indian Institute of Technology Kharagpur
cse.iitkgp.ac.in

Thank You!!