Chapter two

Fundamental of the
Analysis of Alg.
Efficiency

Computer Science and Engineering Program:DAA

Gedamu Alemu
Fundamental of the Analysis of Alg.
Efficiency

Gedamu A.
[email protected]
Image processing and Computer vision SIG

Some of the sides are exported from different sources to clarify the topic

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Analysis of algorithms

 Issues:
 Correctness
 space efficiency
 time efficiency
 optimality

 Approaches:
 Theoretical analysis
 Empirical analysis

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Space Analysis
 When considering space complexity, algorithms are divided
into those that need extra space to do their work and those that
work in place.

 Space analysis would examine all of the data being stored to

see if there were more efficient ways to store it.

 Example : As a developer, how do you store the real numbers ?

 Suppose we are storing a real number that has only one place of precision
after the decimal point and ranges between -10 and +10.
 How many bytes you need ?

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Space Analysis
 Example : As a developer, how do you store the real numbers ?
 Suppose we are storing a real number that has only one place of precision
after the decimal point and ranges between -10 and +10.
 How many bytes you need ?

 Most computers will use between 4 and 8 bytes of memory.

 If we first multiply the value by 10. We can then store this as an integer
between -100 and +100. This needs only 1 byte, a savings of 3 to 7 bytes.
 A program that stores 1000 of these values can save 3000 to 7000 bytes.

 It makes a big difference when programming mobile or PDAs or when you

have large input .

Computer Science and Engineering Program:DAA

Gedamu Alemu
Theoretical analysis of time efficiency
 Time efficiency is analyzed :
 Basic operation
 Input size
 Basic operation: the operation that contributes the most towards the
running time of the algorithm
input size

T(n) ≈ copC(n)
running time execution time Number of times
basic operation
for basic operation
is executed
or cost
Note: Different basic operations may cost differently!

Computer Science and Engineering Program:DAA

Gedamu Alemu
Why Input Classes are Important?
 Input determines what the path of execution through an
algorithm will be.
 If we are interested in finding the largest value in a list of N
numbers, we can use the following algorithm:

Computer Science and Engineering Program:DAA

Gedamu Alemu
Why Input Classes are Important?
 If the list is in decreasing order,
 There will only be one assignment done before the loop starts.
 If the list is in increasing order,
 There will be N assignments (one before the loop starts and N -1 inside the
loop).
 Our analysis must consider more than one possible set of input,
because if we only look at one set of input, it may be the set that is
solved the fastest (or slowest).

Computer Science and Engineering Program:DAA

Gedamu Alemu
Input size and basic operation examples

Problem Input size measure Basic operation

Searching for key in a list

Number of list’s items, i.e. n Key comparison
of n items

Multiplication of two Matrix dimensions or total Multiplication of two

matrices number of elements numbers

Checking primality of a n’size = number of digits (in

Division
given integer n binary representation)

Visiting a vertex or
Typical graph problem #vertices and/or edges
traversing an edge

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Importance of the analysis

 It gives an idea about how fast the algorithm

T(n) ≈ copC(n)
 If the number of basic operations

C(n) = ½ n (n-1) = ½ n2 – ½ n ≈ ½ n2
How much longer if the algorithm doubles its input
size? 1
( 2n) 2
T (2n) COP C (2n) 2
T (n)
  4
COP C (n) 1 2
 Increasing input size increases the complexity ( n)

2
We tend to omit the constants because they have no effect with large inputs
 Everything is based on estimation

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Empirical analysis of time efficiency

 Select a specific (typical) sample of inputs

 Use physical unit of time (e.g., milliseconds)

or
Count actual number of basic operation’s executions

 Analyze the empirical/experimental data

Computer Science and Engineering Program:DAA

Gedamu Alemu
 case to consider in analysis

Best-case, average-case, worst-case

For some algorithms, efficiency depends on form of input:

 Worst case: Cworst(n) – maximum over inputs of size n

 Best case: Cbest(n) – minimum over inputs of size n

 Average case: Cavg(n) – “average” over inputs of size n

 The toughest to do

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Best-case, average-case, worst-case
 Average case: Cavg(n) – “average” over inputs of size n
 Determine the number of different groups into which all possible input
sets can be divided.
 Determine the probability that the input will come from each of these
groups.
 Determine how long the algorithm will run for each of these groups.

n is the size of the input,

m is the number of groups,
pi is the probability that the input will be from
group i,
ti is the time that the algorithm takes for input
from group i.

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Example: Sequential search

 Worst case n key comparisons

 Best case 1 comparison

 Average case (n+1)/2, assuming K is in A

Computer Science and Engineering Program:DAA
Gedamu Alemu
 Computing the Average Case for the Sequential
search
 Neither the Worst nor the Best case gives the yield to the actual performance of an algorithm
with random input.
 The Average Case does
 Assume that:
 The probability of successful search is equal to p(0≤ p ≤1)
 The probability of the first match occurring in the ith position is the same for every i .
 The probability of a match occurs at ith position is p/n for every i
 In the case of unsuccessful search , the number of comparison is n with probability (1-
p).

p p p p
C avg (n)  [1.  2.  3.  ...  n ]  n.(1  p )
n n n n
p
 [1  2  3  ...n]  n.(1  p )
n
p n(n  1) p (n  1)
 .  n.(1  p )   n(1  p )
n 2 2

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Computing the Average Case for the Sequential
search
 If p =1 (I found the key k)
 The average number of comparisons is (n+1)/2
 If p=0
 The average number of key comparisons is n

 The average Case is more difficult than the Best and Worst cases

p p p p
C avg (n)  [1.  2.  3.  ...  n ]  n.(1  p )
n n n n
p
 [1  2  3  ...n]  n.(1  p )
n
p n(n  1) p (n  1)
 .  n.(1  p )   n(1  p )
n 2 2

Computer Science and Engineering Program:DAA

Gedamu Alemu
 17

Mathematical Background
 Mathematical Background

 Logarithms

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Logarithms

 Which Base ?

Loga n = Loga b Logb n

Loga n = c Logb n

In terms of complexity , we tend to

ignore the constant

Computer Science and Engineering Program:DAA

Gedamu Alemu
Mathematical Background

Computer Science and Engineering Program:DAA

Gedamu Alemu
Mathematical Background

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Mathematical Background
 ..

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Types of formulas for basic operation’s count
 Exact formula
e.g., C(n) = n(n-1)/2

 Formula indicating order of growth with specific multiplicative

constant
e.g., C(n) ≈ 0.5 n2

 Formula indicating order of growth with unknown

multiplicative constant
e.g., C(n) ≈ cn2

Computer Science and Engineering Program:DAA

Gedamu Alemu
 24

Order of growth
 Order of growth
 Of greater concern is the rate of increase in operations for an
algorithm to solve a problem as the size of the problem
increases.

 This is referred to as the rate of growth of the algorithm.

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Order of growth
 The function based on x2
increases slowly at first, but as
the problem size gets larger, it
begins to grow at a rapid rate.

 The functions that are based on

x both grow at a steady rate for
the entire length of the graph.

 The function based on log x

seems to not grow at all, but
this is because it is actually
growing at a very slow rate.

Computer Science and Engineering Program:DAA

Gedamu Alemu
Values of some important functions as n

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Order of growth

Second point to consider :

 Because the faster growing functions increase at such a
significant rate, they quickly dominate the slower-growing
functions.
 This means that if we determine that an algorithm’s
complexity is a combination of two of these classes, we will
frequently ignore all but the fastest growing of these terms.

 Example : if the complexity is

we tend to ignore 30x term

Computer Science and Engineering Program:DAA

Gedamu Alemu
 29

Classification of Growth
 Classification of Growth: Asymptotic order of growth

A way of comparing functions that ignores constant factors and small

input sizes.
 O(g(n)): class of functions f(n) that grow no faster than g(n)
 All functions with smaller or the same order of growth as g(n)
n  O(n 2 ), 100n  5  O(n 2 ), 0.5n(n  1)  O(n 2 ), n 3  O(n 2 )
 Ω(g(n)): class of functions f(n) that grow at least as fast as g(n)
 All functions that are larger or have the same order of growth as g(n)
n 3  (n 2 ), 0.5n(n  1)  (n 2 ), 100n  5  (n 2 ),
 Θ(g(n)): class of functions f(n) that grow at same rate as g(n)
 Set of functions that have the same order of growth as g(n)

an 2  bn  (n 2 )

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Big-oh
• O(g(n)): class of functions t(n) that grow no faster than g(n)
• if there exist some positive constant c and some nonnegative n0
such that
t (n)  cg (n) for all n  n0

Ex : Prove that 100n  5  O(n 2 )

100n  5  100n  n (for all n  5)

 101n  101n 2
c  101 and n0  5

You may come up with different c and n0

Computer Science and Engineering Program:DAA
Gedamu Alemu
 Big-omega

Ω(g(n)): class of functions t(n) that grow at least as fast as g(n)

t (n)  cg (n) for all n  n0

prove that n 3  (n 2 ) ?

n 3  n 2 for all n  0
c  2 and n0  1

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Big-theta

Θ(g(n)): class of functions t(n) that grow at same rate as g(n)

c2 g (n)  t (n)  c1g (n) for all n  n0

You need to get

c1 , c2 , and n0

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Summary

>=
(g(n)), functions that grow at least as fast as g(n)

=
(g(n)), functions that grow at the same rate as g(n)
g(n)

<=
O(g(n)), functions that grow no faster than g(n)

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Theorem
 If t1(n)  O(g1(n)) and t2(n)  O(g2(n)), then
t1(n) + t2(n)  O(max{g1(n), g2(n)}).
 The analogous assertions are true for the -notation and -
notation.
 Implication: The algorithm’s overall efficiency will be
determined by the part with a larger order of growth, i.e., its
least efficient part.
 For example, 5n2 + 3nlogn  O(n2)

Proof. There exist constants c1, c2, n1, n2 such that

t1(n)  c1*g1(n), for all n  n1
t2(n)  c2*g2(n), for all n  n2
Define c3 = c1 + c2 and n3 = max{n1,n2}. Then
t1(n) + t2(n)  c3*max{g1(n), g2(n)}, for all n  n3
Computer Science and Engineering Program:DAA
Gedamu Alemu
 Some properties of asymptotic order of growth
 f(n)  O(f(n))

 f(n)  O(g(n)) iff g(n) (f(n))

 If f (n)  O(g (n)) and g(n)  O(h(n)) , then f(n)  O(h(n))

If f1(n)  O(g1(n)) and f2(n)  O(g2(n)) , then

f1(n) + f2(n)  O(max{g1(n), g2(n)})

Also, 1in (f(i)) =  (1in f(i))

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Orders of growth of some important functions
 All logarithmic functions loga n belong to the same class
(log n) no matter what the logarithm’s base a > 1 is
because log a n  log b n / log b a
 All polynomials of the same degree k belong to the same class:

aknk + ak-1nk-1 + … + a0  (nk)

 Exponential functions an have different orders of growth for

different a’s

Computer Science and Engineering Program:DAA

Gedamu Alemu
Basic asymptotic efficiency classes

1 constant
log n logarithmic

n linear

n log n n-log-n

n2 quadratic

n3 cubic

2n exponential

n! factorial

Computer Science and Engineering Program:DAA

Gedamu Alemu
 39
Example
 Compare the order of growth of 1/2n(n-1) and n2

 Compare order n! and 2n

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Time efficiency of nonrecursive algorithms
General Plan for Analysis

 Decide on parameter n indicating input size

 Identify algorithm’s basic operation

 Determine worst, average, and best cases for input of size n

 Set up a sum for the number of times the basic operation is executed

 Simplify the sum using standard formulas and rules.

Computer Science and Engineering Program:DAA

Gedamu Alemu
 41
Examples
 Example 1: The Largest Element
 O(n)
 Example :Element uniqueness problem
 O(n2)
 Example 3: Matrix multiplication
 O(n3)
 Example 4: Counting binary digits
 log2n

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Example 1: The Largest Element

T(n) = 1in-1 (1) = n-1 = (n) comparisons

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Example 2: Element uniqueness problem

1  u  l  1
i l
n2

T(n) = 0in-2 (i+1jn-1 (1))

 n  1  i  (n  1)  (n  2)  .....  1 
i 0

= 0in-2 (n-i+1) = (n  1)n 1 2

  n  ( n 2 )
= (n 2 ) comparisons 2 2
Computer Science and Engineering Program:DAA
Gedamu Alemu
 Example 3: Matrix multiplication

n 1 n 1 n 1

T(n) =   1 =
i 0 j 0 k 0
( n3 ) multiplications
Ignored addition for simplicity

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Example 4: Counting binary digits

It cannot be investigated the way the previous examples are.

The halving game: Find integer i such that n/ ≤ 1.

Answer: i ≤ log n. So, T(n) = (log n) divisions.
2i
Another solution: Using recurrence relations.

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Plan for Analysis of Recursive Algorithms
 Decide on a parameter indicating an input’s size.

 Identify the algorithm’s basic operation.

 Check whether the number of times the basic op. is executed may
vary on different inputs of the same size. (If it may, the worst,
average, and best cases must be investigated separately.)
 Set up a recurrence relation with an appropriate initial condition
expressing the number of times the basic op. is executed.
 Solve the recurrence (or, at the very least, establish its solution’s
order of growth) by backward substitutions or another method.

Computer Science and Engineering Program:DAA

Gedamu Alemu
 47
Example :
 Example 1 : Factorial: Recursive Algorithm
 Example 2: The Tower of Hanoi Puzzle
 Example 3: Fibonacci Sequence

Computer Science and Engineering Program:DAA

Gedamu Alemu
 48
Example 1 : Factorial: Recursive Algorithm

The stopping condition is n=0

A repetitive algorithm uses recursion whenever the algorithm
appears within the definition itself.

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Example 1: Recursive evaluation of n!
Definition: n ! = 1  2  … (n-1)  n for n ≥ 1 and
0! = 1

Recursive definition of n!: F(n) = F(n-1)  n for n ≥ 1

and
F(0) = 1

Size: N
Basic operation: Multiplication
Recurrence relation: M(n) = M(n-1) (for F(n-1) ) + 1 (for the n * F(n-1))
M(0) = 0

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Solving the recurrence for M(n)

M(n) = M(n-1) + 1, M(0) = 0 (no multiplication when n=0)

M(n) = M(n-1) + 1
= (M(n-2) + 1) + 1 = M(n-2) + 2
= (M(n-3) + 1) + 2 = M(n-3) + 3
…
= M(n-i) + i
= M(0) + n
=n
The method is called backward
substitution.

Computer Science and Engineering Program:DAA

Gedamu Alemu
 51
Example 2: The Tower of Hanoi Puzzle
Towers of Hanoi (aka Tower of Hanoi) is a mathematical puzzle
invented by a French Mathematician Edouard Lucas in 1983.

• Initially the game has few discs arranged in the increasing

order of size in one of the tower.

• The number of discs can vary, but there are only three
towers.
• The goal is to transfer the discs from one tower another tower.
However you can move only one disk at a time and you can
never place a bigger disc over a smaller disk. It is also
understood that you can only take the top most disc from any
given tower.
Computer Science and Engineering Program:DAA
Gedamu Alemu
 Example 2: The Tower of Hanoi Puzzle

Recurrence for number of moves: M(n) = 2M(n-1) + 1= m(n-1)+1+m(n-1)

Computer Science and Engineering Program:DAA

Gedamu Alemu
0 Move Sequence for d = 3

7
Computer Science and Engineering Program:DAA
Gedamu Alemu
 The Tower of Hanoi Puzzle
 Here's how to find the number of moves needed to transfer larger
numbers of disks from post A to post C, when M = the number of
moves needed to transfer n-1 disks from post A to post C:
 for 1 disk it takes 1 move to transfer 1 disk from post A to post C;
 for 2 disks, it will take 3 moves: 2M + 1 = 2(1) + 1 = 3
 for 3 disks, it will take 7 moves: 2M + 1 = 2(3) + 1 = 7
 for 4 disks, it will take 15 moves: 2M + 1 = 2(7) + 1 = 15
 for 5 disks, it will take 31 moves: 2M + 1 = 2(15) + 1 = 31
 for 6 disks... ?

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Explicit Pattern
 Number of Disks Number of Moves
1 1
2 3
3 7
4 15
5 31
6 63

Computer Science and Engineering Program:DAA

Gedamu Alemu
Powers of two help reveal the pattern:

 Number of Disks (n) Number of Moves

1 21 - 1 = 2 - 1 = 1
2 22 - 1 = 4 - 1 = 3
3 23 - 1 = 8 - 1 = 7
4 24 - 1 = 16 - 1 = 15
5 25 - 1 = 32 - 1 = 31
6 26 - 1 = 64 - 1 = 63

Computer Science and Engineering Program:DAA

Gedamu Alemu
 Fascinating fact

So the formula for finding the number of steps it takes to transfer

n disks from post A to post C is:

2 n- 1

Computer Science and Engineering Program:DAA

Gedamu Alemu
Solving recurrence for number of moves
M(n) = 2M(n-1) + 1, M(1) = 1

M(n) = 2M(n-1) + 1
= 2(2M(n-2) + 1) + 1 = 2^2*M(n-2) + 2^1 + 2^0
= 2^2*(2M(n-3) + 1) + 2^1 + 2^0
= 2^3*M(n-3) + 2^2 + 2^1 + 2^0
=…
= 2^(n-1)*M(1) + 2^(n-2) + … + 2^1 + 2^0
= 2^(n-1) + 2^(n-2) + … + 2^1 + 2^0
= 2^n - 1

Computer Science and Engineering Program:DAA

Gedamu Alemu
Fibonacci Sequence
 The Fibonacci numbers:
0, 1, 1, 2, 3, 5, 8, 13, 21, …
What Is the
Important of
 The Fibonacci recurrence:
F(n) = F(n-1) + F(n-2) Fibonacci
F(0) = 0 Sequence?
F(1) = 1
 F(n) = F(n-1) + F(n-2) or F(n) - F(n-1) - F(n-2) = 0
General 2nd order linear homogeneous recurrence with
constant coefficients:
aX(n) + bX(n-1) + cX(n-2) = 0

Computer Science and Engineering Program:DAA

Gedamu Alemu