Algorithms Analysis & Design

Analysis II
(Normal Code: Asymptotic Analysis
Recursive Code: Steps & Master Method)
 LOGISTICS
Learning Management System (@trysakai)
– Contains ALL course-related stuff (Assignments, Quizzes, Lec’s, Labs…)

– How to Use? [Video by Abdelrahman Khatab] Course Link

– You MUST enroll and keep-track on it

– If you don’t enroll yet,

1. Register your info here (by a valid @CIS email)

2. You’ll receive an enrollment mail @your CIS email

3. Click the link inside the mail

4. Complete your info

 Roadmap

Algorithm Analysis & Design

Think…Design…Analyze…

Analysis Design

Statements, Recursion Divide & Dynamic Greedy Randomiz. Increm.

Conditions Conquer Program. Method Algorithm Improv.
& Loops (2 methods)
 From last time…
T(N) vs O(N) Analysis Guiding Principles
#steps Dominant
 Check worst case
Small Inp. Large Inp.
 At large input size
How to calc. O()?
 Dominant factor w/o constants
 Stmts: order of 1
 Conditions: chose max
 Loops: order of body × # iterations
Order Classes
 Worst: special inp  max time
 Best: special inp  min time
 PRE-TEST
1. Analyze:
fun(N)
{ sum = 0;
if (N == 1) return 0
for(i = 1 ; i*i < N; i++)
sum += i;
return fun(N/2) × fun(N/2)
}
2. Can we reduce the complexity?
 Agenda
• Asymptotic Analysis
– Big O
PART I
– Big Ω
– Big Θ
• Analysis: Recursive Code
– Recursion Recap [refer to this video]
PART II
– How to Analyze Recursive Code?
– Master Method [Prove: last three videos here]
• Summary & Questions
 Learning Outcomes

• Explain the use of big O, big omega & big theta notation to
describe the amount of work done by an algorithm.
• Use recurrence relations to determine the time complexity of
recursive algorithms.
• Solve elementary recurrence relations.
 References
• Chapter 3
• Chapter 4
– Section 4.5 (Master Method)
 Asymptotic Notations
• Asymptotic Analysis: study the time complexity at large input size
• Can be expressed by:
– Upper bound: O()
– Lower bound: Ω()
– Exact bound: ()
• Always look for 
• When it’s difficult to obtain  look for O, Ω
 PRE-TEST [Revisited]
1. Analyze:
fun(N)
{ sum = 0;
if (N == 1) return 0
for(i = 1 ; i*i < N; i++)
sum += i;
return fun(N/2) × fun(N/2)
}
Since loop from i=1 to N, one can say that # iterations is O(N)
However, this is an Upper bound since the condition is checked on i * i.
Exact bound… stay tuned
 Asymptotic Notations
• O-notation

11
Example

10
2 +
2 n
) =
T( n

g(n) = n2
Example

3g(n) = 3n2

10
2 +
2 n
) =
T( n

g(n) = n2
Example

n0=4
Formally:
• Choose c = 3
3g(n) = 3n 2
• Choose n0 = 4
• Then:

10
2 +
2 n
) =
T( n

g(n) = n2
Same example

Formally:
• Choose c = 7
• Choose n0 = 2
• Then:
7g(n) = 7n2

10
2 +

= 2n
)
T(n
re is not a
g(n) = n2 The o ic e
t ” c h
“correc n0
o f c a n d
n0=2
Another example:

g(n) = n2 • Choose c = 1
• Choose n0 = 1
• Then

T(n) = n
 Take-away from examples

• To prove T(n) = O(g(n)), you have to come up with c and n0 so

that the definition is satisfied.

• Ex: Dis/Prove the following: (by definition)

1. 2 × N2 + 10 × N = O(N2)
2. 100 × N + 5 = O(N2)
3. 3N = O(2N)
 Examples
• Sol: 3N = O(2N)
 Agenda
• Asymptotic Analysis
– Big O
– Big Ω
– Big Θ
• Analysis: Recursive Code
– Recursion Recap [refer to this video]
– How to Analyze Recursive Code?
– Master Method [Prove: last three videos here]
• Summary & Questions
 Asymptotic Notations
•  - notation

20
 Asymptotic Notations
• -notation

O&↔Θ

21
 Examples
1. Show that:
1. 0.5n2 - 3n = (n2)
2. 5n3 + 100n ≠ (n2)

3. 0.5n3 + 100n = (n3) = (n2) = (n)

4. 30n+8 = O(n)
Work through these
5. 100n + 5 = O(n2) on your own!
6. 5n2 = (n)
7. n = (logn)
8. log(n) = Θ( 2loglog(n) )
 Examples
• Sol: 0.5n2 - 3n = (n2)
 Examples
• Sol: 5n3 + 100n ≠ (n2)
 PRE-TEST [Revisited]
1. Analyze:
fun(N)
{ sum = 0;
if (N == 1) return 0
for(i = 1 ; i*i < N; i++)
sum += i;
return fun(N/2) × fun(N/2)
}

Calculate EXACT bound of # iterations

(i.e. Big-Θ)?
Try Yourself…#1.1 Chose the correct CHOICES
Fun(N) value for C1, C2 & a. C1 ≤ 2, C2 = 4 & N0 = 1
K = 1000; Sum = 0 N0 (if any) that b. C1 = 1, C2 ≥ 1 & N0 = 1
Switch (choice)
Case 1: satisfy:
c. C1 ≤ 16, C2 ≥ 16 & N0 = 4
Sum = GetSum(A[], K) 4(N/2) = Θ(2N)
break d. No values exist. The
Case 2: relation is not valid
Res = Factorial(K)
For I = 1 to Res
Sum = Sum + Res
break CHOICES
O(K!)
Chose MOST SUITABLE:Θ(1)
Worst Case = ……….. O(K)
Θ(N!)
Best Case = ………..
Ω(1)
Θ(K)
 Agenda
• Asymptotic Analysis
– Big O
PART I
– Big Ω
– Big Θ
• Analysis: Recursive Code
– Recursion Recap [refer to this video]
PART II
– How to Analyze Recursive Code?
– Master Method [Prove: last three videos here]
• Summary & Questions
 Recursion
• Function calls itself for one or more times
• Consists of two parts:
– Recursive part
– Base case (stopping condition)
• General form:
Fun(…)
…. Pre-code (Base case)
Base Case
…. Pre-code (others)
Normal Code Fun(…), Fun(…), …etc Recursive Code
….
Post-code
….
How to Analyze
Recursive Code?
 Two Steps & Three Methods
Fun(…)
…. Pre-code (Base case)
Base Case
…. Pre-code (others)
Normal Code Fun(…), Fun(…), …etc Recursive Code
….
Post-code
….
1. Calculate Running Time T(N) [Recurrence]
T(N) = Time of Normal Code + T(Recursive Code) + … etc
T(Base) = Time of Base Case
2. Solve it to Get the Order ( or O)
T(N) Solve () or O(N)

Iteration Method Master Method Recursion Tree Method

 1. Calculate Recurrence Relation T(N)
Example1: Factorial
Fact(N)
If (N == 0)
Base Case
return 1
else
Normal Code
return N × Fact(N-1) Recursive Code

T(N) = (1) + T(N – 1)

T(0) = (1)
 1. Calculate Recurrence Relation T(N)
Example2: Fibonacci
Fib(N)
If (N == 1 or N == 2)
Base Case
return 1
else
Normal Code
return Fib(N-1) + Fib(N-2) Recursive Code

T(N) = (1) + T(N – 1) + T(N – 2)

T(1) = (1)
 1. Calculate Recurrence Relation T(N)
Example3: Binary Search
BinSrch(A, I, S, E)
If (S > E)
Base Case
return False
M = (S + E) / 2
If (I == A[M]) return true
else If (I > A[M])
Normal Code
return BinSrch(A, I, M+1, E) Recursive Code
else If (I < A[M]) OR
return BinSrch(A, I, S, M-1) Recursive Code

T(N) = (1) + T(N / 2)

T(1) = (1)
 Agenda
• Asymptotic Analysis
– Big O
PART I
– Big Ω
– Big Θ
• Analysis: Recursive Code
– Recursion Recap [refer to this video]
PART II
– How to Analyze Recursive Code?
– Master Method [Prove: last three videos here]
• Summary & Questions
 2. Solve T(N) to Get  (or O)

1. Iteration method
2. Master method
3. Recursive Tree method
 2. Solve T(N) to Get  (or O)
2) Master Method:
Direct apply to the master theory
Suitable ONLY for recurrences of the following form

T(n) = aT(n/b) + f(n) , T(Base) = Θ(1)

a ≥ 1, b > 1

T(N) = 2 T(N/2) + cN2 T

T(N) = T(N/3) + O(1) T
T(N) = 2 T(3N/2) + √N F (b = 2/3 < 1)
T(N) = T(N-1) + T(N-2) + c F (not on the form) 38
 The Master Theorem

• The top node effort is f(n) , the next level has work af(n/b) associated with each node,
the next level a2f(n/b2), and so on.

• The total time taken at all levels is the sum of the terms aif(n/bi).

• What this sum looks like depends on how the growth of f(n) compares to the growth
of the number of leaves.

f(N) vs
[Prove: last three videos here] 40
 The Master Theorem
C 1: f(n)
ASE 1:
CASE f(n) isis O(),
O(), ε>0
ε>0.. Since
Since the
the leaves
leaves grow
grow faster
fasterthan
than f,f, asymptotically
asymptotically all
all
of
ofthe
thework
workisisdone
doneat atthe
theleaves,
leaves,and
andso T(n)isisΘ()
soT(n) Θ()..

41
 The Master Theorem
C
CASE 2: f(n)
ASE 2: f(n) isis Θ()
Θ()..The
Theleaves
leavesgrow
growat
atthe
thesame
samerate
rateas
asf,f,so
sothe
thesame
sameorder
orderof
of
work
work isis done
done at
at every
every level
level of
of the
the tree.
tree. The
The tree has O(lg
tree has O(lg n)
n) levels,
levels, times
times the
the work
work
done
doneon
ononeonelevel,
level,yielding T(n)isisΘ(
yieldingT(n) Θ( lglg n)
n)..

42
 The Master Theorem
C 3: f(n)
ASE 3:
CASE f(n) isis Ω()
Ω() ,, ε>0 AND aa f(n/b)
ε>0.. AND f(n/b) ≤≤ cc f(n)
f(n) ,, 00 << cc << 1.1. Since
Since height
height
grows
grows faster
faster than
than the
the number
number ofof leaves,
leaves, asymptotically,
asymptotically, the
the work
work isis done
done at
at the
the root
root
node
nodeand
andinternal
internallevels, T(n) isisΘ(f(n))
levels,T(n) Θ(f(n))..

43
 The Master Theorem
where a ³ 1, b > 1, and f is
• If T(n) is a recurrence of the form asymptotically positive.

T(n) = aT(n/b) + f(n), T(Base) = O(1) then

 

 n 
log b a
f ( n )  O 
n 
log b a  

 
   0
T (n)   n log b a

log n f (n)   n  
log b a

  c 1
 
 f (n)   
f (n)   n log b a  AND 
 
 David Luebke
44
af (n / b)  cf (n) for large n
09/30/2023
Ex2.
• T(n) = 3T(n/4) + n lg n
a=3, b=4, f(n) = n lg n
nlog a = nlog 3 = n0.793
b 4

Since f(n) = Ω (n0.793 + ), where =0.2, case 3

applies:
Check the condition: a f(n/b) ≤ c f(n) , 0<c<1
af(n/b) = 3(n/4) lg (n/4) ≤ ¾ n lg n = cf(n) for
c=¾

Thus the solution is T(n) = f(n) = (n lg n)

46
Ex3. T(n) = 4T(n/2) + n2
a = 4, b = 2  nlogba = n2; f (n) = n2.
CASE 2: f (n) = Q(n2lg0n), that is, k = 0.
 T(n) = Q(n2lg n).

Ex4. T(n) = 2T(n/2) + n lg n

a = 2, b = 2  nlogba = n; f (n) = n lg n.
Master method does not apply.
In particular, for every constant e > 0,
we have ne = w(lg n).
L2.47
 @Home Examples
Ex. T(n) = 4T(n/2) + n3
a = 4, b = 2  nlogba = n2; f (n) = n3.
CASE 3: f (n) = W(n2 + e) for e = 1
and 4(cn/2)3 £ cn3 (reg. cond.) for c = 1/2.
 T(n) = Q(n3).
Ex. T(n) = 4T(n/2) + n
a = 4, b = 2  nlogba = n2; f (n) = n.
CASE 1: f (n) = O(n2 – e) for e = 1.
 T(n) = Q(n2).
Ex. T(n) = 4T(n/2) + n2/lg n
a = 4, b = 2  nlogba = n2; f (n) = n2/lg n.
Master method does not apply. In particular,
for every constant e > 0, we have ne = w(lg n). L2.48
 @Home Examples
• Analyze: F1(N)
If (N > 1)
For I = 1 to N do
J = N
While J > 1 Do
Print “Hello World!!”
Print “Hello World!!”
J = J - 2
End while
End For
For I = 1 to 8 do
F1(N / 2)
End For
End If
End F1
 @Home Examples
• Solution:
– Non-recursive part = Θ(N2)
– Recurrence T(N) = 8 T(N/2) + Θ(N2)
– Using Master Method Case1:
• 0<≤1
• Final Order Θ(N3)
 Agenda
• Asymptotic Analysis
– Big O
PART I
– Big Ω
– Big Θ
• Analysis: Recursive Code
– Recursion Recap [refer to this video]
PART II
– How to Analyze Recursive Code?
– Master Method
• Summary & Questions
 POST-TEST
1. Analyze: Calculate Recurrence Equation (i.e. T(N))?
fun(N)
Solve it?
{ sum = 0;

if (N == 1) return 0

for(i = 1 ; i*i < N; i++)

sum += i;

return fun(N/2) × fun(N/2)

}
 POST-TEST
2. Can we reduce the complexity?
fun(N)

{ sum = 0;

if (N == 1) return 0

for(i = 1 ; i*i < N; i++)

sum += i;

return fun(N/2) × fun(N/2)

}
 SUMMARY
Asymptotic Analysis
• @large N O() Ω() Θ()
• Upper bound: O()
• Lower bound: Ω()
• Exact bound: ()
• Always look for 
• When difficult  look for O, Ω
Recursive Analysis
O&↔Θ
Fun(…) 1. Calculate Running Time T(N) [Recurrence]
Base Case …. Pre-code (Base case) T(N) = Time of Normal Code + T(Recursive Code) + … etc
…. Pre-code (others) T(Base) = Time of Base Case
2. Solve it to Get the Order ( or O)
Normal Code Fun(…), Fun(…), …etc Recursive
T(N) Solve () or O(N)
….
Post-code
…. Iteration Master Recursion Tree
 SUMMARY
Master Method:
T(n) = aT(n/b) + f(n) , T(Base) = Θ(1)
a ≥ 1, b > 1
 

 n 
log b a
 
f (n)  O n log b a  

 
   0
f(N) vs 
T (n)   n log b a log n  
f ( n)   n 
log b a

  c 1
 
 f (n)   
f (n)   n log b a  AND 
 
 af (n / b)  cf (n) for large n
 QUESTIONS
ASYMPTOTIC ANALYSIS
• Test yourself @this online quiz#1.
– Questions: 3.2, 4, 5, 6, 7
– Answers are available

ANALYSIS OF RECURSION
• Test yourself @this online quiz#2.
– Master Method Part
– Answers are available