Algorithm Analysis Concept

1
 Algorithm analysis refers to the process of
determining how much computing time and
storage that algorithms will require.

 In other words, it’s a process of predicting

the resource requirement of algorithms in
a given environment.

2
 In order to solve a problem, there are many
possible algorithms.

 One has to be able to choose the best

algorithm for the problem at hand using some
scientific method.

 To classify some data structures and

algorithms as good:
 we need precise ways of analyzing them in
terms of resource requirement.

3
 The main resources are:
• Running Time
• Memory Usage
• Communication Bandwidth

Note:
Running time is the most important since
computational time is the most precious
resource in most problem domains.

4
There are two approaches to measure the
efficiency of algorithms:
1. Empirical
 based on the total running time of the

program.
 Uses actual system clock time.

Example:
t1
for(int i=0; i<=10; i++)
cout<<i;
t2
Running time taken by the above algorithm
(TotalTime) = t2-t1;
5
 It is difficult to determine efficiency of
algorithms using this approach, because clock-
time can vary based on many factors.
For example:
a) Processor speed of the computer
1.78GHz 2.12GHz
10s <10s
b) Current processor load
 Only the work 10s
 With printing 15s
 With printing & browsing the internet >15s

6
c) Specific data for a particular run of the
program
 Input size
 Input properties
t1
for(int i=0; i<=n; i++)
cout<<i;
t2
T=t2-t1;
For n=100, T>=0.5s
n=1000, T>0.5s

7
d) Operating System
 Multitasking Vs Single tasking
 Internal structure

8
2. Theoretical

 Determining the quantity of resources

required
using mathematical concept.

 Analyze an algorithm according to the

number of
basic operations (time units) required,
rather than according to an absolute
amount of time involved.

9
 We use theoretical approach to determine
the efficiency of algorithm because:

• The number of operation will not vary under

different conditions.

• It helps us to have a meaningful measure that

permits comparison of algorithms independent
of operating platform.

• It helps to determine the complexity of

algorithm.

10
 Complexity Analysis is the systematic study
of the cost of computation, measured
either
in:

 Time units
 Operations performed, or
 The amount of storage space required.

11
 Two important ways to characterize the
effectiveness of an algorithm are its Space
Complexity and Time Complexity.

Time Complexity: Determine the approximate

amount of time (number of operations) required to
solve a problem of size n.
The limiting behavior of time complexity as size
increases is called the Asymptotic Time
Complexity.
Space Complexity: Determine the approximate
memory required to solve a problem of size n.
The limiting behavior of space complexity as size
increases is called the Asymptotic Space
Complexity.
12
 Asymptotic Complexity of an algorithm
determines the size of problems that can be solved
by the algorithm.
 Factors affecting the running time of a program:
◦ CPU type (80286, 80386, 80486, Pentium I---IV)
◦ Memory used
◦ Computer used
◦ Programming Language
 C (fastest), C++ (faster), Java (fast)
 C is relatively faster than Java, because C is relatively
nearer to Machine language, so, Java takes relatively
larger amount of time for interpreting/translation to
machine code.
 Algorithm used
 Input size
Note: Important factors for this course are Input size and
Algorithm used.

13
 Complexity analysis involves two distinct phases:
• Algorithm Analysis: Analysis of the algorithm or
data structure to produce a function T(n) that
describes the algorithm in terms of the operations
performed in order to measure the complexity of the
algorithm.
Example: Suppose we have hardware capable of
executing 106 instructions per second. How long would
it take to execute an algorithm whose complexity
function is T(n)=2n2 on an input size of n=108?
Solution: T(n)= 2n2=2(108)2 = 2*1016
Running time=T(108)/106=2*1016/106= seconds. 2*1010

• Order of Magnitude Analysis: Analysis of the

function T (n) to determine the general complexity
category to which it belongs.

14
 There is no generally accepted set of rules
for algorithm analysis.
 However, an exact count of operations is
commonly used.
 To count the number of operations we can
use the following Analysis Rule.

Analysis Rules:
1. Assume an arbitrary time unit.
2. Execution of one of the following operations
takes time 1 unit:
 Assignment Operation
Example: i=0;
 Single Input/Output Operation
Example: cin>>a;
cout<<“hello”;

15
  Single Boolean Operations
Example: i>=10
 Single Arithmetic Operations
Example: a+b;
 Function Return
Example: return sum;
3. Running time of a selection statement (if,
switch) is the time for the condition
evaluation plus the maximum of the running
times for the individual clauses in the
selection.

16
Example: int x;
int sum=0;
if(a>b)
{
sum= a+b;
cout<<sum;
}
else
{
cout<<b;
}
T(n) = 1 +1+max(3,1)
=5

17
4. Loop statements:
• The running time for the statements inside
the loop * number of iterations + time for
setup(1) + time for checking (number of
iteration + 1) + time for update (number of
iteration)
• The total running time of statements inside a
group of nested loops is the running time of
the statements * the product of the sizes of
all the loops.
• For nested loops, analyze inside out.
• Always assume that the loop executes the
maximum number of iterations possible.
(Why?)
 Because we are interested in the worst case
complexity.

18
5. Function call:
• 1 for setup + the time for any parameter
calculations + the time required for the
execution of the function body.

Examples:
1)
int k=0,n;
cout<<“Enter an integer”;
cin>>n
for(int i=0;i<n; i++)
k++;
T(n)= 3+1+n+1+n+n=3n+5

19
2)
int i=0;
while(i<n)
{
cout<<i;
i++;
}
int j=1;
while(j<=10)
{
cout<<j;
j++;
}
T(n)=1+n+1+n+n+1+11+2(10)
= 3n+34
20
3)
int k=0;
for(int i=1 ; i<=n; i++)
for( int j=1; j<=n; j++)
k++;

T(n)=1+1+(n+1)+n+n(1+(n+1)+n+n)
= 2n+3+n(3n+2)
= 2n+3+3n2+2n
= 3n2+4n+3

21
4). int sum=0;
for(i=1;i<=n;i++))
sum=sum+i;
T(n)=1+1+(n+1)+n+(1+1)n
=3+4n=O(n)

5). int counter(){

int a=0;
cout<<”Enter a number”;
cin>>n;
for(i=0;i<n;i++)
a=a+1;
return 0; }
T(n)=1+1+1+(1+n+1+n)+2n+1
=4n+6=O(n)

22
6). void func( ){
int x=0; int i=0; int j=1;
cout<<”Enter a number”;
cin>>n;
while(i<n){
i=i+1;
}
while(j<n){
j=j+1;
}
}

T(n)=1+1+1+1+1+n+1+2n+n+2(n-1)
= 6+4n+2n-2
=4+6n=O(n)

23
7). int sum(int n){
int s=0;
for(int i=1;i<=n;i++)
s=s+(i*i*i*i);
return s;
}
T(n)=1+(1+n+1+n+5n)+1
=7n+4=O(n)
8). int sum=0;
for(i=0;i<n;i++)
for(j=0;j<n;j++)
sum++;
T(n)=1+1+(n+1)+n+n*(1+
(n+1)+n+n)
=3+2n+n2+2n+2n2
=3+2n+3n2+2n
=3n2+4n+3=O(n2)

24
 In the above examples we have seen that
analyzing Loop statements is so complex.

 It can be simplified by using some formal

approach in which case we can ignore
initializations, loop controls, and updates.

Simple Loops: Formally

 For loop can be translated to a summation.

25
 The index and bounds of the summation are
the same as the index and bounds of the for
loop.
 Suppose we count the number of additions

that are done. There is 1 addition per iteration

of the loop, hence n additions in total.

N
f
or(
in
ti=1
;i<
=N;i
++){

}
s
um =
su
m +
i
; 

1N

i1

26
Nested Loops: Formally
 Nested for loops translate into multiple

summations, one for each For loop.

for (int i = 1; i <= N; i++) {

for (int j = 1; j <= M; j++) { N M N

}
sum= sum+i+j; 2 2M 2MN
i1 j1 i1
}

27
 Consecutive Statements: Formally
 Add the running times of the separate
blocks of your code.

for (int i = 1; i <= N; i++) {

sum = sum+i;
}  
N  N N 

for (int i = 1; i <= N; i++) { 

  1     2
 i 1   i 1 j 1 
  N  2 N 2

for (int j = 1; j <= N; j++) {

sum = sum+i+j;
}
}
28
 Conditionals: (Formally take maximum)
Example:

if (test ==1) {
for (inti =1; i <=N; i++) { N N N 
sum=sum+i; 1, 2 
max
}}  
 i1 i1 j1 
elsefor (inti =1; i <=N; i++) {
for (int j =1; j <=N; j++) { max 2

N, 2N 2N 2

sum=sum+i+j;
}}

29
 Categories of Algorithm
Analysis
 Algorithms may be examined under different
situations to correctly determine their
efficiency for accurate comparison.
Best Case Analysis:
 Assumes the input data are arranged in the

most advantageous order for the algorithm.


Takes the smallest possible set of inputs.
 Causes execution of the fewest number of

statements.

30
 Computes the lower bound of T(n), where
T(n) is the complexity function.

Examples:
For sorting algorithm
 If the list is already sorted (data are arranged in
the required order).
For searching algorithm
 If the desired item is located at first accessed
position.

31
Worst Case Analysis:
 Assumes the input data are arranged in the most
disadvantageous order for the algorithm.
 Takes the worst possible set of inputs.
 Causes execution of the largest number of
statements.
 Computes the upper bound of T(n) where T(n) is
the complexity function.
◦ Example: While sorting, if the list is in opposite
order. While searching, if the desired item is
located at the last position or is missing.

Examples:
For sorting algorithms
 If the list is in opposite order.
For searching algorithms
 If the desired item is located at the last
position or is missing.

32
Worst Case Analysis:
 Worst case analysis is the most common analysis
because:
 It provides the upper bound for all input (even for bad
ones).
 Average case analysis is often difficult to determine
and define.
 If situations are in their best case, no need to develop
algorithms because data arrangements are in the
best situation.
 Best case analysis can not be used to estimate
complexity.
 We are interested in the worst case time since it
provides a bound for all input-this is called the “Big-
Oh” estimate.

33
Average Case Analysis:
 Determine the average of the running time overall
permutation of input data.
 Takes an average set of inputs.
 It also assumes random input size.
 It causes average number of executions.
 Computes the optimal bound of T(n) where T(n)
is the complexity function.
 Sometimes average cases are as bad as worst
cases and as good as best cases.

Examples:
For sorting algorithms
While sorting, considering any arrangement (order of
input data).
For searching algorithms
While searching, if the desired item is located at any
location or is missing.

34
 The study of algorithms includes:
◦ How to Design algorithms (Describing
algorithms)
◦ How to Analyze algorithms (In terms of
time and memory space)
◦ How to validate algorithms (for any input)
◦ How to express algorithms (Using
programming language)
◦ How to test a program (debugging and
maintaining)

35
 Order of Magnitude
 Refers to the rate at which the storage or

time
grows as a function of problem size.

 It is expressed in terms of its relationship

to
some known functions.

 This type of analysis is called

Asymptotic
analysis.
36
 Asymptotic Notations
 Asymptotic Analysis is concerned with how

the running time of an algorithm increases

with the size of the input in the limit, as
the size of the input increases without
bound!
 Asymptotic Analysis makes use of O (Big-

Oh) ,  (Big-Omega),  (Theta), o

(little-o),  (little-omega) - notations in
performance analysis and characterizing
the complexity of an algorithm.
 Note: The complexity of an algorithm is a

numerical function of the size of the

problem (instance or input size). 37
 Types of Asymptotic Notations

1. Big-Oh Notation

 Definition: We say f(n)=O(g(n)), if there

are positive constants no and c, such that
to the right of no, the value of f(n) always
lies on or below c.g(n).
 As n increases f(n) grows no faster than
g(n).
 It’s only concerned with what happens for
very large values of n.
 Describes the worst case analysis.
 Gives an upper bound for a function to
within a constant factor.
38
 O-Notations are used to represent the
amount of time an algorithm takes on the
worst possible set of inputs, “Worst-Case”.
39
 Question-1

f(n)=10n+5 and g(n)=n. Show that f(n) is

O(g(n)). To show that f(n) is O(g(n)), we
must show that there exist constants c and
k such that f(n)<=c.g(n) for all n>=k.
10n+5<=c.n  for all n>=k
let c=15, then show that 10n+5<=15n
5<=5n or 1<=n
So, f(n)=10n+5<=15.g(n) for all n>=1
(c=15, k=1), there exist two constants that satisfy
the above constraints.

40
 Question-2

f(n)=3n2+4n+1. Show that f(n)=O(n2).

4n<=4n2 for all n>=1 and 1<=n2
for all n>=1
3n2+4n+1<=3n2+4n2+n2 for all
n>=1
<=8n2 for all n>=1
So, we have shown that f(n)<=8n2 for all
n>=1.
Therefore, f(n) is O(n2), (c=8, k=1), there exist
two constants that satisfy the constraints.

41
2. Big-Omega ()-Notation (Lower bound)

 Definition: We write f(n)= (g(n)) if there are

positive constants no and c such that to the right
of no the value of f(n) always lies on or above
c.g(n).
 As n increases f(n) grows no slower than g(n).
 Describes the best case analysis.
 Used to represent the amount of time the
algorithm takes on the smallest possible set of
inputs-“Best case”.

Example:
Find g(n) such that f(n) = (g(n)) for f(n)=3n+5
g(n) = √n, c=1, k=1.
f(n)=3n+5=(√n)
42
Big-Omega ()-Notation (Lower bound)

43
3. Theta Notation (-Notation) (Optimal
bound)

 Definition: We say f(n)= (g(n)) if there exist positive

constants no, c1 and c2 such that to the right of no, the
value of f(n) always lies between c1.g(n) and c2.g(n)
inclusive, i.e., c1.g(n)<=f(n)<=c2.g(n), for all n>=no.
 As n increases f(n) grows as fast as g(n).
 Describes the average case analysis.
 To represent the amount of time the algorithm takes on
an average set of inputs- “Average case”.
Example: Find g(n) such that f(n) = Θ(g(n)) for
f(n)=2n2+3

 n2 ≤ 2n2 ≤ 3n2 c1=1, c2=3 and no=1
f(n) = Θ(g(n)).
44
Theta Notation (-Notation) (Optimal bound)

45
4. Little-oh (small-oh) Notation
 Definition: We say f(n)=o(g(n)), if there are
positive constants no and c such that to the right of
no, the value of f(n) lies below c.g(n).
 As n increases, g(n) grows strictly faster than f(n).
 Describes the worst case analysis.
 Denotes an upper bound that is not
asymptotically tight.
 Big O-Notation denotes an upper bound that may or
may not be asymptotically tight.

Example:
Find g(n) such that f(n) = o(g(n)) for f(n) = n2

n2<2n2, for all n>1,  k=1, c=2, g(n)=n2

n2< n3, g(n) = n3, f(n)=o(n3)
n2< n4 , g(n) =n4 , f(n)=o(n4)

46
5. Little-Omega () notation

 Definition: We write f(n)=(g(n)), if there are

positive constants no and c such that to the right of
no, the value of f(n) always lies above c.g(n).
 As n increases f(n) grows strictly faster than g(n).
 Describes the best case analysis.
 Denotes a lower bound that is not
asymptotically tight.
 Big -Notation denotes a lower bound that may or
may not be asymptotically tight.
Example: Find g(n) such that f(n)=(g(n)) for
f(n)=n2+3

g(n)=n, Since n2 > n, c=1, k=2.

g(n)=√n, Since n2 > √n, c=1, k=2, can also
be solution.
47
 Rules to estimate Big Oh of a given
function
 Ignore Pick the highest order.
 the coefficient.
Example:
1. T(n)=3n + 5  O(n)
2. T(n)=3n2+4n+2  O(n2)

 Some known functions encountered when

analyzing algorithms. (Complexity category
for Big-Oh). See next slide 

48
Rule 1:
If T1(n)=O(f(n)) and T2(n)=O(g(n)), then
a) T1(n)+T2(n)=max(O(f(n)),O(g(n))),
b) T1(n)*T2(n)=O(f(n)*g(n))
Rule 2:
If T(n) is a polynomial of degree k, then T(n)=(nk).
Rule 3:
logk n=O(n) for any constant k. This tells us that
logarithms grow very slowly.

 We can always determine the relative growth rates

of two functions f(n) and g(n) by computing lim n-
>infinity f(n)/g(n). The limit can have four possible
values.

49
 ◦ The limit is 0: This means that f(n)=o(g(n)).
◦ The limit is c≠0: This means that f(n)=(g(n)).
◦ The limit is infinity: This means that g(n)=o(f(n)).
◦ The limit oscillates: This means that there is no
relation between f(n) and g(n).
Example:
 n3 grows faster than n2, so we can say that
n2=O(n3) or n3=(n2).
 f(n)=n2 and g(n)=2n2 grow at the same
rate, so both f(n)=O(g(n)) and f(n)=(g(n))
are true.
 If f(n)=2n2, f(n)=O(n4), f(n)=O(n3), and
f(n)=O(n2) are all correct, but the last
option is the best answer.

50
T(n) Complexity Big-O
Category
functions F(n)
c, c is constant 1 C=O(1)

10logn + 5 logn T(n)=O(logn)

√n +2 √n T(n)=O(√n)
5n+3 n T(n)=O(n)
3nlogn+5n+2 nlogn T(n)=O(nlogn)

10n2 +nlogn+1 n2 T(n)=O(n2)

5n3 + 2n2 + 5 n3 T(n)=O(n3)
2n+n5+n+1 2n T(n)=O(2n)

7n!+2n+n2+1 n! T(n)=O(n!)

8nn+2n +n2 +3 nn T(n)=O(nn)

51
52
 Arrangement of Function Name
common functions
by growth rate. c Constant
List of typical
growth rates. log N Logarithmic
log2 N Log-squared
N Linear
N log N Log-Linear
N2 Quadratic
N3 Cubic
2N Exponential
53
 The order of the body statements of a given
algorithm is very important in determining
Big-Oh of the algorithm.

Example: Find Big-Oh of the following

algorithm.
1. for( int i=1;i<=n; i++)
sum=sum + I;
T(n)=2*n=2n=O(n).
2. for(int i=1; i<=n; i++)
for(int j=1; j<=n; j++)
k++;
T(n)=1*n*n=n2 = O(n2).
54
55