Introduction to the Theory of Computation

CSC236
Week 04: Runtime Analysis

Ahmed Ashraf

Department of Mathematical and Computational Sciences,

University of Toronto, Mississauga

September 27-29, 2023

Outline

Motivating example

Runtime Complexity: The Language

Closed form expressions

Factorial
Program - Example of a recursive code

def fact(n):
if n == 0:
return 1
else:
return n * fact(n-1)

=) if
* n
=
0
I

This is !
saying n

n .
(n-1) ! otherwise .
Factorial
Program
def fact(n):
if n == 0:
return 1 -- Tol
else:
return n * fact(n-1)
Analysis ** T(n 1)
-

1. The recursive part:

2. The non-recursive part:
Factorial
Program
def fact(n):
if n == 0:
return 1
else:
return n * fact(n-1)
Analysis
1. The recursive part:
2. The non-recursive part:

Time Complexity
T(0) = c and
T(n) = T(n 1) + d for n > 0
Runtime complexity
For Factorial program
(
c if n = 0
T(n) =
T(n 1) + d if n > 0

Derivation 0
↓
T(n) =

T(n-1
For n<1 T(n) ( T(n-2) d) d
=
+
+

For n<2 , T(n) =

((T(n -
3) +
d) d)+ +
d

T(n-37 +
36
: "

For n<k T(n) =

T(n -k) + kd

Let k =
n
,

T(n) =

TC) +
nd
 T(n) =

c +
nd

The number c and d are not properties of the

algorithm .

that
They are the time steps consumed
by operations are

predefined ,
and not
depend on the input n .

>
Assumption
good enough
We would like to that T(n) is linear for
say now

in n .

That is the
, given algorithm takes linear time (in its
input)
to Cule
-

n!
comy

For this we
develop a
language .
Outline

Motivating example

Runtime Complexity: The Language

Closed form expressions

Three bigs asymptotic
upper bound
function
familiar
time steps
(Big) O counting
Given a function f : N ! N, and a function g : N ! N we
write f 2 O(g) if there exists a positive real number C and a
positive integer n0 such that

f(n)  C · g(n) for all positive integer n n0

f is
eventually bounded above by a scalar multiple of g

~
Cg(n)
.

I~
R ↑
runtime f(n)

time

iof input
Abuse of notation :

Sometimes we write :

: f
=

O(g) to mean f G
Olg>
1 F =
0
E

(9) and h =

0
C
(g)
h
# F
=

2 . n to mean the function F :

N->Nor IR
f(n) n
=

3 .

log(n) to mean the function f :

/orIR-> /Nor I
-

and so on .
fini =

log (n) ,

(or some other

base)
 Examples :

1 . n G 0 (n) because i
for all nx0 .

In
general , f(n) EO(fini) .

2 .
ntO( because n n"
for all nx,

In
general ,
n t O(n") where K = 1 .

3 .
n GO(2") because ns2" for all nx 1

In
general ,
ne0(a") for all a <, 2 .
Three bigs
asymptotic
lower bound
(Big) ⌦
Given a function f : N ! N, and a function g : N ! N we
write f 2 ⌦(g) if there exists a positive real number C and a
positive integer n0 such that

C · g(n)  f(n) for all positive integer n n0

f is
eventually bounded below by a scalar multiple of g

an~
runtime f(nz

~
R
(g(n)

input
-

size of
 Examples :

n( 2(n) because n < x

for all n < 0 .

In
general , f(n) (1(finl) .

2 .
E (n) because n
> n"
for all nx

In
general ,
nE -(n) where K 1.

"

3 .
n - /log(n)) because n < 2 for alt nx 1 .

:
log(n) is an
increasing function

log(n) <
for all nx1
=>
n
.
Three bigs light
bound

(Big) ⇥
Given a function f : N ! N, and a function g : N ! N we
write f 2 ⇥(g) if f 2 O(g) and f 2 ⌦(g). In other words, there
exists positive real numbers C and D, and positive integer n0 such
that

C · g(n)  f(n) and f(n)  D · g(n) (1)

f is
eventually bounded
-
between scalar multiples of g
for all n n0 .

~
Cg(n)
.

~
R W
↑runtime

-
f(n)

Dg(n)

iof input
 Examples :

1 -

Of t
Note n n E O(nY) because

nn +n -
-> n
2
An 1
,

2 I

So we take no=1 and C=1

and
because
ninEr(n2) I na
I with
So n=1 and C
we take
=I
Using the Notations

I These notations can be used to determine the asymptotic

running time of an algorithm
I For iterative algorithms, the strategy is to sum the total work
done by the algorithm, and then determine a ⇥ bound from
the sum.
I For recursive algorithms, the strategy is to use recursion, and
repeated substitution to come up with a closed-form
expression.
Outline

Motivating example

Runtime Complexity: The Language

Closed form expressions

Bounding a Sum
def f(n):
total = 0
for i in range(n):
for j in range(i, n):
total = total + 1
return total

Es= en-pit

En -i =

E
↑

n
Here the runtime tin)
i
n
=

,
Repeated Substitution

Instead of trying to guess a closed-form from nothing, we’ll use a

technique called repeated substitution.
I Step 1: substitute a few times to find a pattern
I Step 2: guess the recurrence after k substitutions
I Step 3: solve for k
I Step 4: substitute for k into Step 2 to find a potential
closed-form
I “Potential”, because our guess might be wrong!
I Step 5: prove, using induction, that the recursive and
potential closed-form are equivalent
The creativity is in Step 2. The rest is just lots of careful algebra.
Exercise: Developing Closed-Form
Let F : N ! N be a function given by
(
1, if n = 0
F (n) = (2)
F (n 1) + 2n, if n 1

Use the five steps of repeated substitution to rediscover and prove

the closed-form for the above function.
Solution :

For ny
, I F(n)
=

F(n 1)
-
+

2n
For n>, 2 ,
F(n) =

( F(n-2) 2(n-1))
+
+ 2n

For n<, 3 F(n) (F(n-3) 2(n 3)) 2(n 1) 2n

= + - + - +

"

For nik , F(n) =

F(n-K) +
2 (n-i)

Let k =
n
,
F(n)
=

F(0) +
2
(i)
!
=

1 + n(n + 1)
So we claim F(n) =
n2+ n + 1

Proof :
(by simple induction)

Becase :
For n
=
0
,
F(0) =
1

and 02 +
0 +
1 = 1 checked v

Inactive hypothesis :
Assume that for
F(k) k =
2
+
k +
1

In
e
step :

We have F(k 1) +
=

F(k) +

2(k 1) +

=>

k2 +
k 1 +
+
2k +
2

=
k +
2k +
1 +
k 1 +
+
1
=

(k 1)=
+

(k 1) +
+
1

Which exactly the predicate for k+1 . This implies by principle

,

of simple mathematical induction ,

F(n) =

n
+

n
+

1 for all nEIN .

B
Exercise: Developing Closed-Form
Let T : N ! N be a function given by
(
1, if n = 0
T (n) = (3)
2T (n 1) + 1, if n 1

Use the five steps of repeated substitution to rediscover and prove

the closed-form for the above function.
Solution :

For nx1
,
T(n) =
2 T (n-17+ 1

For n = 2 T(n) 2
(2T(n-2) 1) + 1
=

+
,

For n>, 3 , T(n) =

2 (Tin -3)+1) +

2 +
1
For n= 4 ,
T(n) =

2(2T( 4) - +

1) +
2 +
2 +

1
-

For n<, k
,
T(n) =

"T(n-k)
2 + 2 +
2 - ... +
2 +

2 +
1
 k then
If n
=

2 2
,

T(n)
=

2 T(O) + + + ... +
2 +
2 +
1

T(n) =

2 +
2" - 1

n +
1

T(n) 2 I
= -

S
We leave this as an exercise for you
to
show this
by induction
Exercise: Developing Closed-Form
Let T : N ! N be a function given by
8
>
> 1 if n = 0,
>
>
<3 if n = 1,
T (n) = n 1 (4)
>
> X
>
:3T (n 1) +
> 2T (n i) otherwise.
i=2

Use the five steps of repeated substitution to rediscover and prove

the closed-form for the above function.
Solution :

For n32 ,
T(n) =

3 T(n-D + 2 /T(13 +
T(2) +
.. .
+

T(n -

2)
For n, 3
,
T(n-17 =
3 Y(n-27 +
2 (T(1) +
T(2) +
.. .
+

T(n-3))
Subtracting the second from the first ,
we
get
T(n) Y(n-1) 3T(n-1) 3 T(n -2) 2 T(n 2)
= - +
-
-

=> T(n) -

4T(n 1 -
+
T(n-2) =

y
Recall that solve this the ansatz
we can
using

T(n) =
Y
(see (ec2)