Computer Science and Engineering| Indian Institute of Technology Kharagp

cse.iitkgp.ac

Algorithms – I (CS21203)

Autumn 2024, IIT Kharagpur

Analysis of Algorithms
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Analyzing Algorithms
• Predict how your algorithm performs in practice
• By analyzing several candidate algorithms for a problem
we can identify efficient ones
• Criteria
• Running time
• Space usage
• Cache I/O
• Main memory I/O
• Lines of codes

Jul 25, 26, 2024 2

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Analyzing Running Time

• Random Access Machine (RAM) model
• Every operation including memory access, arithmetic
operations etc. takes same amount of time.
• Is it precise?
• Not really. But precise model would be tedious – would
yield very little insight into algorithm design and analysis
• However, we should be careful not to abuse it
• We only care about “Order of the cost”, i.e., we omit
• Lower order terms
• Constants
Asymptotic Analysis

Jul 25, 26, 2024 3

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Why Asymptotic Analysis

• Because we only care about how fast a function grows!

• Would you rather have a million rupees one time or one

paisa on day one, doubled every day for a month?

• Actually, the second option can get you more than 1

million (in around 27 days itself)

• paisa = 1.342 million (for )

Jul 25, 26, 2024 4

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Running Time Analysis of Insertion Sort

• Lets go back to insertion sort and see its running time
• Our expression will evolve from a messy formula that
assumes each line of the code takes a constant amount of
time

denotes the
number of times
line 5 gets
executed for that
value of .

Jul 25, 26, 2024 5

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Running Time Analysis of Insertion Sort

• Best case: (Array is already sorted) ->

• -> a linear function of 1
• Worst case: (Array is reverse sorted) ->
• is maximum each time, i.e.,
• -> Thus a quadratic function of

Jul 25, 26, 2024 6

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Best/Average/Worst Case Analysis

• We looked at both `best case’ (input array was already
sorted) and `worst case’ (input array was reverse sorted)

• In this course, we shall usually concentrate on worst-case

running time

• Major reasons
• Gives an upper bound on the running time for any input
• For some algorithms, worst case occurs fairly often, e.g.,
searching
• Average case is often roughly as bad as the worst case.

Jul 25, 26, 2024 7

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Order of Growth
• In analyzing running time for `insertion sort’, we started
with constants to represent the cost of each statement
• Then we observed that they give more detail than we
need and we discarded them
• We shall go ahead with more simplifying abstraction:
Rate/Order of Growth
• For the function we care when is large enough. When is
small, is small anyway
• The constant factors and lower order terms doesn’t affect
the growth of the function
• One algorithm is more efficient than another if its worst-
case running time has a lower order of growth

Jul 25, 26, 2024 8

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Order of Growth

𝑛
𝑔 2 ( 𝑛 ) =0.1 × 2
2
𝑔 1 ( 𝑛 ) =𝑛
𝑓 2 ( 𝑛 )=9 𝑛
𝑓 1 (𝑛 )=3 𝑛

Omit the constant factors

When is large enough, will be much larger than or

and will have similar growth trend

Jul 25, 26, 2024 9

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Order of Growth

2
𝑔 1 ( 𝑛 ) =𝑛
2
𝑔 2 ( 𝑛 ) =𝑛 − 6 𝑛
𝑓 ( 𝑛 )=6 𝑛

Omit the lower-order

terms

When is large enough, or will still be much larger than

and will have similar growth trend because is much smaller compared to

Jul 25, 26, 2024 10

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Asymptotic Notations
• These notations are used to describe the asymptotic
running time of an algorithm
• However, asymptotic notations can apply to other
functions that have nothing to do whatsoever with
algorithms
O (Big-O)
𝑂 ( 𝑔 ( 𝑛 ) )={ 𝑓 ( 𝑛 ) : ∃𝑐> 0 , 𝑛0 > 0 , 𝑠𝑢𝑐h 𝑡h𝑎𝑡 0 ≤ 𝑓 (𝑛)≤ 𝑐𝑔 ( 𝑛 ) ∀ 𝑛 ≥ 𝑛0 }
After some constant • Asymptotic upper bound
𝑓 ( 𝑛 ) ≤ 𝑐𝑔(𝑛) • Note the : can be of the
same order, but can be
smaller as well

( 2
) 2
( 2
) 3 2
𝑛=𝑂 𝑛 , 𝑛 =𝑂 𝑛 ,𝑛 ≠ 𝑂(𝑛 )
Jul 25, 26, 2024 11
CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

O (Big-O) []
𝑂 ( 𝑔 ( 𝑛 ) )={ 𝑓 ( 𝑛 ) : ∃𝑐> 0 , 𝑛0 > 0 , 𝑠𝑢𝑐h 𝑡h𝑎𝑡 0 ≤ 𝑓 (𝑛)≤ 𝑐𝑔 ( 𝑛 ) ∀ 𝑛 ≥ 𝑛0 }
After some constant
𝑓 ( 𝑛 ) ≤ 𝑐𝑔(𝑛)
• How can we show,
• Let
• , so
• Similarly, let
Always larger than
• , so

• ; How can we show, ?

• Let ;
• so

Jul 25, 26, 2024 12

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Ω (Big- Ω) []
Ω ( 𝑔 ( 𝑛 ) ) ={ 𝑓 ( 𝑛 ) : ∃𝑐 >0 ,𝑛 0 >0 , 𝑠𝑢𝑐h 𝑡h𝑎𝑡 0 ≤ 𝑐𝑔( 𝑛) ≤ 𝑓 (𝑛) ∀ 𝑛 ≥ 𝑛 0 }

After some constant

𝑓 ( 𝑛 ) ≥ 𝑐𝑔(𝑛)
• Asymptotic lower bound
• can be of the same
order, but can be larger
as well

𝑛 ≠ Ω ( 𝑛2 ) , 𝑛2 =Ω ( 𝑛2 ) , 𝑛3 =Ω(𝑛 2)
Jul 25, 26, 2024 13
CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Θ (Big- theta) []
Θ ( 𝑔 ( 𝑛 ) )={ 𝑓 ( 𝑛 ) :∃𝑐 1 , 𝑐 2 >0 , 𝑛0 >0 , 𝑠 . 𝑡 . 0 ≤ 𝑐 1 𝑔 (𝑛)≤ 𝑓 (𝑛)≤ 𝑐 2 𝑔 ( 𝑛 ) ∀ 𝑛≥ 𝑛0 }

After some constant • Asymptotic tight bound

must be
within this range • Must be of the same
order
• means and ,
must be =

𝑛 ≠ Θ ( 𝑛2 ) , 𝑛2=Θ ( 𝑛 2 ) ,𝑛 3 ≠ Θ(𝑛 2)
Jul 25, 26, 2024 14
CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

What This also Means

• O(g(n)): class of functions f(n) that grow no faster than
g(n)

• Θ(g(n)): class of functions f(n) that grow at same rate as

g(n)

• Ω(g(n)): class of functions f(n) that grow at least as fast as

g(n)

Jul 25, 26, 2024 15

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Analogy to Real Numbers

functions Real Numbers

Jul 25, 26, 2024 16

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

o (small-o) [] and ⍵ (small ⍵)

𝑜 ( 𝑔 ( 𝑛 ) )={ 𝑓 ( 𝑛 ) : 𝑓𝑜𝑟 𝑎𝑛𝑦 𝑐>0 , ∃𝑛0 >0 , 𝑠𝑢𝑐h 𝑡h𝑎𝑡 0 ≤ 𝑓 (𝑛)<𝑐𝑔 ( 𝑛 ) ∀ 𝑛≥𝑛 0 }
• Asymptotic non-tight upper
bound
• Note the : must be smaller
• Equivalently, if
• By analogy,
𝜔 ( 𝑔 ( 𝑛 ) )={ 𝑓 ( 𝑛 ) : 𝑓𝑜𝑟 𝑎𝑛𝑦 𝑐>0 , ∃𝑛0 >0 , 𝑠𝑢𝑐h 𝑡h𝑎𝑡 0 ≤ 𝑐𝑔(𝑛)< 𝑓 (𝑛) ∀ 𝑛≥ 𝑛0 }
• Asymptotic non-tight lower
bound
• Note the : must be larger
• Equivalently, if

Jul 25, 26, 2024 17

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Analogy to Real Numbers

functions Real Numbers

Jul 25, 26, 2024 18

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Time Complexity
• Consider an algorithm running on a computer that can
execute ops/sec
• For , what amount of time will be required?

Jul 25, 26, 2024 19

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Subset Sum Selection Problem

• Given a set of integers and a target , determine if has a
subset that sums to exactly.
• and ?
• What about ?

Jul 25, 26, 2024 20

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Complexity of Parts to Whole

• Suppose you have 3 sections of an algorithm for which you
know the complexities are like this.

2
Θ(𝑛 )
2
𝑂(𝑛 )
2
𝑂(𝑛 )
• What can you tell about the complexity of the overall
algorithm?

Jul 25, 26, 2024 21

CS21203 / Algorithms - I | Introduction
 Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Theorem
• If t1(n) = O(g1(n)) and t2(n) = O(g2(n)), then t1(n) +
t2(n) = O(max{g1(n), g2(n)})
• The algorithm’s overall efficiency will be determined
by the part with a larger order of growth, i.e., its
least efficient part.
• For example, 5n2 + 3nlogn = O(n2)
Proof. There exist constants such that

Define and , then

Jul 25, 26, 2024 22

CS21203 / Algorithms - I | Introduction
Computer Science and Engineering| Indian Institute of Technology Kharagp
cse.iitkgp.ac

Thank You!!