Bilgisayar Mühendisliği Bölümü

DATA STRUCTURES AND

ALGORITHMS

Efficiency of Algorithms

GIT – Computer Engineering Department

Recapture

Asymptotic Notations
– O Notation
– Ω Notation
– Θ Notation
– o Notation

GIT – Computer Engineering Department 2

Big Oh Notation (O)
Provides an “upper bound” for the function f

Definition :

T(N) = O (f(N)) if there are positive constants
c and n0 such that
T(N) ≤ c f(N) when N ≥ n0
– T(N) grows no faster than f(N)
– growth rate of T(N) is less than or equal to growth
rate of f(N) for large N
– f(N) is an upper bound on T(N)
• not fully correct !

GIT – Computer Engineering Department 3

Omega Notation (Ω)

Definition :
T(N) = Ω (f(N)) if there are positive constants c and
n0 such that T(N) ≥ c f(N) when N≥ n0

– T(N) grows no slower than f(N)

– growth rate of T(N) is greater than or equal to growth rate
of f(N) for large N
– f(N) is a lower bound on T(N)
• not fully correct !

GIT – Computer Engineering Department 4

Theta Notation (θ)

Definition :
T(N) = θ (h(N)) if and only if
T(N) = O(h(N)) and T(N) = Ω(h(N))

– T(N) grows as fast as h(N)

– growth rate of T(N) and h(N) are equal for large N
– h(N) is a tight bound on T(N)
• not fully correct !

GIT – Computer Engineering Department 5

Little o Notation (o)

Definition :
T(N) = o(p(N)) if
T(N) = O(p(N)) and T(N)≠θ(p(N))

– p(N) grows strictly faster than T(N)

– growth rate of T(N) is less than the growth rate of p(N) for
large N
– p(N) is an upperbound on T(N) (but not tight)
• not fully correct !

GIT – Computer Engineering Department 6

ROAD MAP

Efficiency of Algorithms

Model

General Rules

Recursive Calls

Maximum Subsequence Sum Problem

Binary Search

Experimentally checking analysis

GIT – Computer Engineering Department 7

Efficiency of Algorithms
Question: How can we characterize the performance of an
algorithm ...

Without regard to a specific computer?

Without regard to a specific language?

Over a wide range of inputs?

What is performance?

Execution Time (most important !)

Required memory space

GIT – Computer Engineering Department 8

Efficiency of Algorithms
Desire: Function that describes execution time in
terms of input size

How?
– Emprical  after implementation
– Theoritical  before implementation

If there are several algorithms, we need to evaluate

them without implementation

GIT – Computer Engineering Department 9

MODEL

A formal framework for analysis (simplify the
real computers)

There are many models
– Automata
– Turing Machine
– RAM

GIT – Computer Engineering Department 10

MODEL

We use RAM model (normal computer)
– addition
– multiplication take a unit time
– comparison
– assignment
– fixed size word (32 bit)
– no complicated operation supported
• requires an algorithm (algorithm may not take unit time)

GIT – Computer Engineering Department 11

Efficiency of Algorithms

Example:
int sum (int N)
{
int i, partialsum;  does not count
partialsum = 0; 1
for (i=1; i<=N; i++)  1+(N+1)+N
partialsum+=i*i*i;  3N
return partialsum; 1
}
= 5N + 4

GIT – Computer Engineering Department 12

Efficiency of Algorithms
Basic idea: Use Asymptotic Notations

Ignore constant factor: computer and language
implementation details affect that: go for fundamental rate
of increase with problem size.

Consider fastest growing term: Eventually, for large
problems, it will dominate.

Algorithm complexity is effected by

• compiler
usually effects the constants &
• computer lower order terms

• algorithm use asymptotic notation

GIT – Computer Engineering Department 13

Efficiency of Algorithms

Symbols used in Quantifying Software Performance

T(n) The time that a function takes as a function of the number of inputs, n. We
may not be able to measure or determine this exactly.
f(n) Any function of n. Generaly f(n) will represent a simpler function than T(n),
for example n2 rather than 1.5n2 - 1.5n.
O(f(n)) Order of magnitude. O(f(n)) is the set of functions that grow no faster than
f(n). We say that T(n) = O(f(n)) to indicate that the growth of T(n) is
bounded by the growth of f(n).

GIT – Computer Engineering Department 14

ROAD MAP

Efficiency of Algorithms

Model

General Rules

Recursive Calls

Maximum Subsequence Sum Problem

Binary Search

Experimentally checking analysis

GIT – Computer Engineering Department 15

GENERAL RULES

RULE 1 : For Loops
The running time of a for loop is at most the running time of
the statements in the for loop times the number of iterations

int i, a = 0;
for (i=0; i<n; i++)
{
print i; T(n) = θ(n)
a=a+i;
}
return i;

GIT – Computer Engineering Department 16

GENERAL RULES

RULE 2 : Nested Loops
Analyze nested loops inside out

Example :

for (int i=1;i<=q;i++)

{
for (int j=1;j<=r;j++) θ(q)
k++; θ(r)
}

T(n) = θ(r*q)
GIT – Computer Engineering Department 17
GENERAL RULES

RULE 3 : Consequtive Statements
Add the running times

for … θ(N)
…;
for … θ(N2)
for … θ(N2)
…;

GIT – Computer Engineering Department 18

GENERAL RULES

RULE 4 : If / Else

if (condition) T3(n)
S1; T1(n) T(n)
else
S2; T2(n)

Running time is never more than the running time of

the test plus larger of the running times of S1 and S2

(may overestimate but never underestimates)

T(n) ≤ T3(n) + max (T1(n), T2(n))

GIT – Computer Engineering Department 19

Types of complexition
T worst ( N ) = max { T ( I )} → usually used
|I |= N

Tav( N ) = ∑T ( I ).Pr( I ) → difficult to compute

|I|= N

T best ( N ) = min {T ( I )}
|I |= N

T worst ( N ) ≥ T av ( N ) ≥ T best ( N )
T ( n ) = O (T worst ( n )) = Ω (T best ( n ))

GIT – Computer Engineering Department 20

GENERAL RULES

RULE 4 : If / Else
if (condition) T3(n)
S1; T1(n) T(n)
else
S2; T2(n)

Tw (n) = T3(n) + max (T1(n), T2(n))

Tb (n) = T3(n) + min (T1(n), T2(n))
Tav (n) = p(T)T1(n) + p(F)T2(n) + T3(n)
p(T)  p (condition = True)
p(F)  p (condition = False)

GIT – Computer Engineering Department 21

GENERAL RULES

Example :

if (condition) T3(n) = θ(n)

S1; T1(n) = θ(n2) T(n)
else
S2; T2(n) = θ(n)

Tw (n) = T3 (n) + max (T1 (n) , T2 (n)) = θ(n2)

Tb (n) = T3 (n) + min (T1 (n), T2 (n)) = θ(n)

if p(T) = p(F) = ½
Tav (n) = p(T)T1 (n) + p(F)T2 (n) + T3 (n) = θ(n2)

T(n) = O (n2) = Ω (n)

GIT – Computer Engineering Department 22

ROAD MAP

Efficiency of Algorithms

Model

General Rules

Recursive Calls

Maximum Subsequence Sum Problem

Binary Search

Experimentally checking analysis

GIT – Computer Engineering Department 23

RECURSIVE CALLS
Example 1:
Algorithm for computing factorial

int factorial (int n)

{
if (n<=1)
return 1;
else
return n*factorial(n-1); 1 for multiplication
} + 1 for substraction
+ cost of evaluation of
factorial(n-1)
T(n) = cost of evaluation of factorial of n
T(n) = 4 + T(n-1)
T(1) =2

GIT – Computer Engineering Department 24

RECURSIVE CALLS
T(n) = 4 + T(n-1)
T(n) = 4 + 4+ T(n-2)
T(n) = 4 + 4 + 4 + T(n-3)
.
.
.

T(n) = k*4 + T(n-k) k= n-1 =>

T(n) = (n-1)*4 + T(n-(n-1))
T(n) = (n-1)*4 + T(1)
T(n) = (n-1)*4 + 2
T(n) = θ (n)

GIT – Computer Engineering Department 25

RECURSIVE CALLS
Example 2:
Algorithm for fibonacci series
Fib (int N)
{
if (N<=1)
return 1;
else
return Fib(N-1)+Fib(N-2);
}

T(N) = T(N-1) + T(N-2) + 4 , T(1)=T(0)=2

=> T(N) = O(2n) (by induction)

GIT – Computer Engineering Department 26

ROAD MAP

Efficiency of Algorithms

Model

General Rules

Recursive Calls

Maximum Subsequence Sum Problem

Binary Search

Experimentally checking analysis

GIT – Computer Engineering Department 27

MAXIMUM SUBSEQUENCE SUM PROBLEM

Given (possibly negative) integers A1, A2,…, AN, find

j
max ∑ Ak
1 ≤ i ≤ j ≤ N k =i

-2 11 -4 13 -5 -2
A[2,4]  sum = 20
A[2,5]  sum = 15

There are many algorithms to solve this problem !

GIT – Computer Engineering Department 28

MAXIMUM SUBSEQUENCE SUM PROBLEM

for small N all algorithms are fast

for large N O(N) is the best
GIT – Computer Engineering Department 29
MAXIMUM SUBSEQUENCE SUM PROBLEM

1) Try all possibilities exhaustively

for each i (0 to N-1)

for each j (i to N-1)
compute the sum of subsequence for (i to j)
check if it is maximum

T(N) = O(N3)
GIT – Computer Engineering Department 30
Algorithm 1:

GIT – Computer Engineering Department 31

© Mark Allen Weiss, Data Structures and Algorithm Analysis in C
MAXIMUM SUBSEQUENCE SUM PROBLEM

2) How to compute sum of a subsequence

j j −1

∑A
k =i
k = A j + ∑ Ak
k =i

– We can use previous subsequence sum to calculate

current one in O(1) time

T(N) = O(N2)

GIT – Computer Engineering Department 32

Algorithm 2:

GIT – Computer Engineering Department 33

© Mark Allen Weiss, Data Structures and Algorithm Analysis in C
MAXIMUM SUBSEQUENCE SUM PROBLEM

3) Compeletely different algorithm ?

Divide and Conquer Strategy

Maximum subsequence can be

in L
solved recursively

in R

in the middle, in both sides

largest sum in L ending with
middle element
+
largest sum in R begining with
middle element

GIT – Computer Engineering Department 34

GIT – Computer Engineering Department 35
© Mark Allen Weiss, Data Structures and Algorithm Analysis in C
Algorithm 4:

GIT – Computer Engineering Department 36

© Mark Allen Weiss, Data Structures and Algorithm Analysis in C
ROAD MAP

Efficiency of Algorithms

Model

General Rules

Recursive Calls

Maximum Subsequence Sum Problem

Binary Search

Experimentally checking analysis

GIT – Computer Engineering Department 37

Binary Search

GIT – Computer Engineering Department 38

© Mark Allen Weiss, Data Structures and Algorithm Analysis in C
ROAD MAP

Efficiency of Algorithms

Model

General Rules

Recursive Calls

Maximum Subsequence Sum Problem

Binary Search

Checking your analysis experimentally

GIT – Computer Engineering Department 39

Checking Your Analysis Experimentally

Implement your algorithm, measure the running
time of your implementation
– check with theoretical results

When N doubles (2x)

– linear => 2x
– quadratic => 4x
– cubic => 8x

what about logarithm ?

O(N) O(N logN)
not easy to see the difference !

GIT – Computer Engineering Department 40

Checking Your Analysis Experimentally
Tt (N) = theoretical running time
Te (N) = experimental running time

Let Tt(N) = O(f(N))

compute
Te (N) Te (N)
=
Tt (N) f (N)

– if converges to a constant
f(N) is a tight bound
– if converges to zero
not tight bound (overestimate)
– if diverges
underestimate

GIT – Computer Engineering Department 41

Checking Your Analysis Experimentally

N Tt Te Te /Tt

GIT – Computer Engineering Department 42

Checking Your Analysis Experimentally

GIT – Computer Engineering Department 43

The Process of Algorithm Development

Design
– divide&conquer, greedy, dynamic
programming

Validation
– check whether it is correct

Analysis
– determine the properties of algorithm

Implementation

Testing
– check whether it works for all possible cases

GIT – Computer Engineering Department 44