Basics of Complexity Analysis:

The RAM Model and the

Growth of Functions

Dr. Aimal Tariq Rextin

The Department of Computer Science, CIIT

Adv Algo
 Review :The Fibonacci Sequence

The sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .


0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Adv Algo
 Review :The Fibonacci Sequence

The sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .


0
 if n = 0
Mathematical definition: Fn = 1 if n = 1

Fn−1 + Fn−2 if n > 1


Implementation on a computer:
“pseudo-code”
Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

Adv Algo
 Complexity of Our First Algorithm
We need a mathematical characterization of the performance of the
algorithm

We’ll call it the algorithm’s computational complexity

Adv Algo
 Complexity of Our First Algorithm
We need a mathematical characterization of the performance of the
algorithm

We’ll call it the algorithm’s computational complexity

Let T (n) be the number of basic steps needed to compute

Fibonacci(n)

Adv Algo
 Complexity of Our First Algorithm
We need a mathematical characterization of the performance of the
algorithm

We’ll call it the algorithm’s computational complexity

Let T (n) be the number of basic steps needed to compute

Fibonacci(n)
Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

Adv Algo
 Complexity of Our First Algorithm
We need a mathematical characterization of the performance of the
algorithm

We’ll call it the algorithm’s computational complexity

Let T (n) be the number of basic steps needed to compute

Fibonacci(n)
Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

T (0) = 2; T (1) = 3

Adv Algo
 Complexity of Our First Algorithm
We need a mathematical characterization of the performance of the
algorithm

We’ll call it the algorithm’s computational complexity

Let T (n) be the number of basic steps needed to compute

Fibonacci(n)
Fibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else return Fibonacci(n − 1) + Fibonacci(n − 2)

T (0) = 2; T (1) = 3
T (n) = T (n − 1) + T (n − 2) + 4
Adv Algo
Complexity of Our First Algorithm (2)
But T (n) = T (n − 1) + T (n − 2) + 4 does not give us much
information.
Let us try to derive a simpler equation that although is not very
accurate is more meaningful.

Adv Algo
Complexity of Our First Algorithm (2)
But T (n) = T (n − 1) + T (n − 2) + 4 does not give us much
information.
Let us try to derive a simpler equation that although is not very
accurate is more meaningful.
Let us assume that T (n − 1) ≈ T (n − 2)

Adv Algo
Complexity of Our First Algorithm (2)
But T (n) = T (n − 1) + T (n − 2) + 4 does not give us much
information.
Let us try to derive a simpler equation that although is not very
accurate is more meaningful.
Let us assume that T (n − 1) ≈ T (n − 2)
Hence T (n) = T (n − 1) + T (n − 2) + 4 can now be written as
T (n) = T (n − 1) + T (n − 1) + 4
T (n) = 2T (n − 1) + c, where c is a constant equal to 4

Adv Algo
Complexity of Our First Algorithm (2)
But T (n) = T (n − 1) + T (n − 2) + 4 does not give us much
information.
Let us try to derive a simpler equation that although is not very
accurate is more meaningful.
Let us assume that T (n − 1) ≈ T (n − 2)
Hence T (n) = T (n − 1) + T (n − 2) + 4 can now be written as
T (n) = T (n − 1) + T (n − 1) + 4
T (n) = 2T (n − 1) + c, where c is a constant equal to 4
T (n) = 2T (n−1)+c = 2[T (n−2)+T (n−3)+c]+c = 4T (n−2)+3c

Adv Algo
Complexity of Our First Algorithm (2)
But T (n) = T (n − 1) + T (n − 2) + 4 does not give us much
information.
Let us try to derive a simpler equation that although is not very
accurate is more meaningful.
Let us assume that T (n − 1) ≈ T (n − 2)
Hence T (n) = T (n − 1) + T (n − 2) + 4 can now be written as
T (n) = T (n − 1) + T (n − 1) + 4
T (n) = 2T (n − 1) + c, where c is a constant equal to 4
T (n) = 2T (n−1)+c = 2[T (n−2)+T (n−3)+c]+c = 4T (n−2)+3c
T (n) = 4T (n−2)+3c = 4[T (n−3)+T (n−4)]+3c = 8T (n−3)+7

Adv Algo
Complexity of Our First Algorithm (2)
But T (n) = T (n − 1) + T (n − 2) + 4 does not give us much
information.
Let us try to derive a simpler equation that although is not very
accurate is more meaningful.
Let us assume that T (n − 1) ≈ T (n − 2)
Hence T (n) = T (n − 1) + T (n − 2) + 4 can now be written as
T (n) = T (n − 1) + T (n − 1) + 4
T (n) = 2T (n − 1) + c, where c is a constant equal to 4
T (n) = 2T (n−1)+c = 2[T (n−2)+T (n−3)+c]+c = 4T (n−2)+3c
T (n) = 4T (n−2)+3c = 4[T (n−3)+T (n−4)]+3c = 8T (n−3)+7
We can repeat this .... and the general form of the formula is
T (n) = 2k T (n − k) + (2k − 1)c

Adv Algo
Complexity of Our First Algorithm (2)
But T (n) = T (n − 1) + T (n − 2) + 4 does not give us much
information.
Let us try to derive a simpler equation that although is not very
accurate is more meaningful.
Let us assume that T (n − 1) ≈ T (n − 2)
Hence T (n) = T (n − 1) + T (n − 2) + 4 can now be written as
T (n) = T (n − 1) + T (n − 1) + 4
T (n) = 2T (n − 1) + c, where c is a constant equal to 4
T (n) = 2T (n−1)+c = 2[T (n−2)+T (n−3)+c]+c = 4T (n−2)+3c
T (n) = 4T (n−2)+3c = 4[T (n−3)+T (n−4)]+3c = 8T (n−3)+7
We can repeat this .... and the general form of the formula is
T (n) = 2k T (n − k) + (2k − 1)c We know that T (0) = 2 hence by
replacing n − k = 0 or n = k we get
2n T (0) + (2n − 1)c = 2n ∗ 2 + C

Adv Algo
Complexity of Our First Algorithm (2)
But T (n) = T (n − 1) + T (n − 2) + 4 does not give us much
information.
Let us try to derive a simpler equation that although is not very
accurate is more meaningful.
Let us assume that T (n − 1) ≈ T (n − 2)
Hence T (n) = T (n − 1) + T (n − 2) + 4 can now be written as
T (n) = T (n − 1) + T (n − 1) + 4
T (n) = 2T (n − 1) + c, where c is a constant equal to 4
T (n) = 2T (n−1)+c = 2[T (n−2)+T (n−3)+c]+c = 4T (n−2)+3c
T (n) = 4T (n−2)+3c = 4[T (n−3)+T (n−4)]+3c = 8T (n−3)+7
We can repeat this .... and the general form of the formula is
T (n) = 2k T (n − k) + (2k − 1)c We know that T (0) = 2 hence by
replacing n − k = 0 or n = k we get
2n T (0) + (2n − 1)c = 2n ∗ 2 + C
T (n) ∝ 2n
Can we do better?
Adv Algo
 Running Time
T (n) ∝ 2n i.e Running time is directly proportional an exponential
function
60
Ruby
Scheme
running time (seconds)

Python
40 C
C-wiz
Java
C-gcc
20

0
20 40 60 80 100 120 140 160 180 200
n Adv Algo
 A Better Algorithm
Again, the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

Adv Algo
 A Better Algorithm
Again, the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

Idea: we can build Fn from the ground up!

Adv Algo
 A Better Algorithm
Again, the sequence is 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, . . .

Idea: we can build Fn from the ground up!

SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else pprev = 0
6 prev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f
Adv Algo
 Results

60
Ruby
Scheme
running time (seconds)

Python
40 C
C-wiz
Java
C-gcc
(Python) SmartFibonacci
20

0
20 40 60 80 100 120 140 160 180 200
n

Adv Algo
Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) =

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) = 6 + 6(n − 1)

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) = 6 + 6(n − 1) = 6n

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) = 6 + 6(n − 1) = 6n

The complexity of SmartFibonacci(n) is linear in n

Adv Algo
Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) =

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) = c1 + c2 n

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) = c1 + c2 n hence T (n) ∝ n

Adv Algo
 Complexity of SmartFibonacci
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else prev = 0
6 pprev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

T (n) = c1 + c2 n hence T (n) ∝ n

The complexity of SmartFibonacci(n) is linear in n

Adv Algo
 Results

60
Ruby
Scheme
running time (seconds)

Python
40 C
C-wiz
Java
C-gcc
(Python) SmartFibonacci
20

0
20 40 60 80 100 120 140 160 180 200
n

Adv Algo
 Slow vs. Fast Fibonacci

We informally characterized our two Fibonacci algorithms

Adv Algo
 Slow vs. Fast Fibonacci

We informally characterized our two Fibonacci algorithms

◮ Fibonacci is exponential in n

◮ SmartFibonacci is (almost) linear in n

Adv Algo
 Slow vs. Fast Fibonacci

We informally characterized our two Fibonacci algorithms

◮ Fibonacci is exponential in n

◮ SmartFibonacci is (almost) linear in n

How do we characterize the complexity of algorithms?

◮ in general

Adv Algo
 Slow vs. Fast Fibonacci

We informally characterized our two Fibonacci algorithms

◮ Fibonacci is exponential in n

◮ SmartFibonacci is (almost) linear in n

How do we characterize the complexity of algorithms?

◮ in general

◮ in a way that is specific to the algorithms

◮ but independent of implementation details

Adv Algo
Slow versus Fast Fibonacci Algorithm

60
Ruby
Scheme
running time (seconds)

Python
40 C
C-wiz
Java
C-gcc
(Python) SmartFibonacci
20

0
20 40 60 80 100 120 140 160 180 200
n

Adv Algo
 A Model of the Computer
An informal model of the random-access machine (RAM)

Adv Algo
 A Model of the Computer
An informal model of the random-access machine (RAM)

Basic types in the RAM model

Adv Algo
 A Model of the Computer
An informal model of the random-access machine (RAM)

Basic types in the RAM model

◮ integer and floating-point numbers
◮ limited size of each “word” of data (e.g., 64 bits)

Adv Algo
 A Model of the Computer
An informal model of the random-access machine (RAM)

Basic types in the RAM model

◮ integer and floating-point numbers
◮ limited size of each “word” of data (e.g., 64 bits)

Basic steps in the RAM model

Adv Algo
 A Model of the Computer
An informal model of the random-access machine (RAM)

Basic types in the RAM model

◮ integer and floating-point numbers
◮ limited size of each “word” of data (e.g., 64 bits)

Basic steps in the RAM model

◮ operations involving basic types
◮ load/store: assignment, use of a variable
◮ arithmetic operations: addition, multiplication, division, etc.
◮ branch operations: conditional branch, jump
◮ subroutine call

Adv Algo
 A Model of the Computer
An informal model of the random-access machine (RAM)

Basic types in the RAM model

◮ integer and floating-point numbers
◮ limited size of each “word” of data (e.g., 64 bits)

Basic steps in the RAM model

◮ operations involving basic types
◮ load/store: assignment, use of a variable
◮ arithmetic operations: addition, multiplication, division, etc.
◮ branch operations: conditional branch, jump
◮ subroutine call

A basic step in the RAM model takes a constant time

Adv Algo
 Analysis in the RAM Model
SmartFibonacci(n)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else pprev = 0
6 prev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

Adv Algo
 Analysis in the RAM Model
SmartFibonacci(n) cost times (n > 1)
1 if n == 0
2 return 0
3 elseif n == 1
4 return 1
5 else pprev = 0
6 prev = 1
7 for i = 2 to n
8 f = prev + pprev
9 pprev = prev
10 prev = f
11 return f

Adv Algo
 Analysis in the RAM Model
SmartFibonacci(n) cost times (n > 1)
1 if n == 0 c1 1
2 return 0 c2 0
3 elseif n == 1 c3 1
4 return 1 c4 0
5 else pprev = 0 c5 1
6 prev = 1 c6 1
7 for i = 2 to n c7 n
8 f = prev + pprev c8 n−1
9 pprev = prev c9 n−1
10 prev = f c10 n−1
11 return f c11 1

T (n) = c1 + c3 + c5 + c6 + c11 + nc7 + (n − 1)(c8 + c9 + c10 )

Adv Algo
 Analysis in the RAM Model
SmartFibonacci(n) cost times (n > 1)
1 if n == 0 c1 1
2 return 0 c2 0
3 elseif n == 1 c3 1
4 return 1 c4 0
5 else pprev = 0 c5 1
6 prev = 1 c6 1
7 for i = 2 to n c7 n
8 f = prev + pprev c8 n−1
9 pprev = prev c9 n−1
10 prev = f c10 n−1
11 return f c11 1

T (n) = nC1 + C2 ⇒ T (n) is a linear function of n

Adv Algo
 Input Size
In general we measure the complexity of an algorithm as a function
of the size of the input
◮ size of the input, can be measured as number of elements in an array,
or number of bits
◮ depends what size in a given problem

Adv Algo
 Input Size
In general we measure the complexity of an algorithm as a function
of the size of the input
◮ size of the input, can be measured as number of elements in an array,
or number of bits
◮ depends what size in a given problem
◮ did we do that for SmartFibonacci?

Adv Algo
 Input Size
In general we measure the complexity of an algorithm as a function
of the size of the input
◮ size of the input, can be measured as number of elements in an array,
or number of bits
◮ depends what size in a given problem
◮ did we do that for SmartFibonacci?

Example: given a sequence A = ha1 , a2 , . . . , an i, and a value x,

output true if A contains x

Adv Algo
 Input Size
In general we measure the complexity of an algorithm as a function
of the size of the input
◮ size of the input, can be measured as number of elements in an array,
or number of bits
◮ depends what size in a given problem
◮ did we do that for SmartFibonacci?

Example: given a sequence A = ha1 , a2 , . . . , an i, and a value x,

output true if A contains x

Find(A, x)
1 for i = 1 to length(A)
2 if A[i ] == x
3 return true
4 return false

Adv Algo
 Input Size
In general we measure the complexity of an algorithm as a function
of the size of the input
◮ size of the input, can be measured as number of elements in an array,
or number of bits
◮ depends what size in a given problem
◮ did we do that for SmartFibonacci?

Example: given a sequence A = ha1 , a2 , . . . , an i, and a value x,

output true if A contains x

Find(A, x)
1 for i = 1 to length(A)
2 if A[i ] == x
3 return true
4 return false

T (n) = Cn Adv Algo

 Worst-Case Complexity
In general we measure the complexity of an algorithm in the worst
case

Adv Algo
 Worst-Case Complexity
In general we measure the complexity of an algorithm in the worst
case

Example: given a sequence A = ha1 , a2 , . . . , an i, output true if A

contains two equal values ai = aj (with i 6= j)

Adv Algo
 Worst-Case Complexity
In general we measure the complexity of an algorithm in the worst
case

Example: given a sequence A = ha1 , a2 , . . . , an i, output true if A

contains two equal values ai = aj (with i 6= j)

FindEquals(A)
1 for i = 1 to length(A) − 1
2 for j = i + 1 to length(A)
3 if A[i ] == A[j]
4 return true
5 return false

Adv Algo
 Worst-Case Complexity
In general we measure the complexity of an algorithm in the worst
case

Example: given a sequence A = ha1 , a2 , . . . , an i, output true if A

contains two equal values ai = aj (with i 6= j)

FindEquals(A)
1 for i = 1 to length(A) − 1
2 for j = i + 1 to length(A)
3 if A[i ] == A[j]
4 return true
5 return false

n(n + 1)
T (n) = C ( − 1)
2
Hence T (n) ∝ n2
Adv Algo
 Exercise

What the worst case running time of the following:

Adv Algo
 Exercise

What the worst case running time of the following:

printSomething(n)
1 i =1
2 while i <= n
3 print Hello
4 i =i ∗2
5 return

Where n is always a in the form of 2k where k is an integer.

Adv Algo
 Constant Factors

Does a load/store operation cost more than, say, an arithmetic

operation?

x = 0 vs. y + z

Adv Algo
 Constant Factors

Does a load/store operation cost more than, say, an arithmetic

operation?

x = 0 vs. y + z

We do not care about the specific costs of each basic step

◮ these costs are likely to vary significantly with languages,
implementations, and processors
◮ so, we assume c1 = c2 = c3 = · · · = ci

Adv Algo
 Constant Factors

Does a load/store operation cost more than, say, an arithmetic

operation?

x = 0 vs. y + z

We do not care about the specific costs of each basic step

◮ these costs are likely to vary significantly with languages,
implementations, and processors
◮ so, we assume c1 = c2 = c3 = · · · = ci
◮ we also ignore the specific value ci , and in fact we ignore every
constant cost factor

Adv Algo
 Order of Growth

We care only about the order of growth or rate of growth of T (n)

Adv Algo
 Order of Growth

We care only about the order of growth or rate of growth of T (n)

◮ so we ignore lower-order terms
E.g., in
T (n) = an2 + bn + c
we only consider the n2 term and say that T (n) is a quadratic
function in n

Adv Algo
 Order of Growth

We care only about the order of growth or rate of growth of T (n)

◮ so we ignore lower-order terms
E.g., in
T (n) = an2 + bn + c
we only consider the n2 term and say that T (n) is a quadratic
function in n
We write

T (n) = Θ(n2 )

and say that “T (n) is theta of n-squared”

Adv Algo