Lecture 2

Program Efficiency
&
Complexity Analysis
Algorithm
 An algorithm is a definite procedure for solving
a problem in finite number of steps
 Algorithm is a well defined computational
procedure that takes some value(s) as input,
and produces some value(s) as output.
 Algorithm is finite number of computational
statements that transform input into the output
Good Algorithms?
 Run in less time
 Consume less memory

But computational resources (time

complexity) is usually more important
Measuring Efficiency
 The efficiency of an algorithm is a measure of the
amount of resources consumed in solving a problem of
size n.
 The resource we are most interested in is time
 We can use the same techniques to analyze the consumption of
other resources, such as memory space.
 It would seem that the most obvious way to measure
the efficiency of an algorithm is to run it and measure
how much processor time is needed
 But is it correct???
Factors
 Hardware
 Operating System
 Compiler
 Size of input
 Nature of Input
 Algorithm

Which should be improved?

Running Time of an Algorithm
 Depends upon
 Input Size
 Nature of Input
 Generally time grows with size of input, so
running time of an algorithm is usually
measured as function of input size
 Running time is measured in terms of number
of steps/primitive operations performed
 Independent from machine, OS
Finding running time of an
Algorithm / Analyzing an Algorithm
 Running time is measured as number of
steps/primitive operations performed
 Steps means elementary operation like

+, *,<, =, A[i] etc.

 We will measure number of steps taken in
term of size of input
Simple Example (1)
// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A

int Sum(int A[], int N)

{
int s=0;
for (int i=0; i< N; i++)
s = s + A[i];
return s;
}
How should we analyse this?
Simple Example (2)
// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A

int Sum(int A[], int N){

int s=0; 1
for (int i=0; i< N; i++)
2 3 4
s = s + A[i];
5 6 7 1,2,8: Once
return s; 3,4,5,6,7: Once per each iteration
}
8 of for loop, N iteration
Total: 5N + 3
The complexity function of the
algorithm is : f(N) = 5N +3
Simple Example (3)
Growth of 5n+3
Estimated running time for different values of N:

N = 10 => 53 steps
N = 100 => 503 steps
N = 1,000 => 5003 steps
N = 1,000,000 => 5,000,003 steps

As N grows, the number of steps grow in linear

proportion to N for this function “Sum”
What Dominates in Previous
Example?
What about the +3 and 5 in 5N+3?
 As N gets large, the +3 becomes insignificant
 5 is inaccurate, as different operations require
varying
amounts of time and also does not have any significant
importance

What is fundamental is that the time is linear in N.

Asymptotic Complexity: As N gets large, concentrate on
the highest order term:
 Drop lower order terms such as +3
 Drop the constant coefficient of the highest order
term i.e. N
Asymptotic Complexity
The 5N+3 time bound is said to "grow
asymptotically" like N
 This gives us an approximation of the
complexity of the algorithm
 Ignores lots of (machine dependent)
details, concentrate on the bigger picture
 Example (2): Comparing

Functions 4000
Which function
3500
is better?
3000
10 n2 Vs n3
2500

10 n^2
2000
n^3

1500

1000

500

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Comparing Functions
 As inputs get larger, any algorithm of a
smaller order will be more efficient than an
algorithm of a larger order
0.05 N2 = O(N2)
Time (steps)

3N = O(N)

Input (size)
N = 60
Big-Oh Notation
 Even though it is correct to say “7n - 3 is
O(n3)”, a better statement that “7n - 3 is O(n)”,
that is, one should make the approximation as
tight as possible
 Simple Rule:
Drop lower order terms and constant
factors
7n-3 is O(n)
8n2log n + 5n2 + n is O(n2log n)
Performance Classification
f(n) Classification
1 Constant: run time is fixed, and does not depend upon n. Most instructions are
executed once, or only a few times, regardless of the amount of information being
processed
log n Logarithmic: when n increases, so does run time, but much slower. Common in
programs which solve large problems by transforming them into smaller problems.

n Linear: run time varies directly with n. Typically, a small amount of processing is
done on each element.
n log n When n doubles, run time slightly more than doubles. Common in programs which
break a problem down into smaller sub-problems, solves them independently, then
combines solutions
n2 Quadratic: when n doubles, runtime increases fourfold. Practical only for small
problems; typically the program processes all pairs of input (e.g. in a double nested
loop).
n3 Cubic: when n doubles, runtime increases eightfold

2n Exponential: when n doubles, run time squares. This is often the result of a natural,
“brute force” solution.
Size does matter[1]
What happens if we double the input size N?

N log2N 5N N log2N N2 2N
8 3 40 24 64 256
16 4 80 64 256 65536
32 5 160 160 1024 ~109
64 6 320 384 4096 ~1019
128 7 640 896 16384 ~1038
256 8 1280 2048 65536 ~1076
Size does matter[2]
 Suppose a program has run time O(n!) and the
run time for
n = 10 is 1 second

For n = 12, the run time is 2 minutes

For n = 14, the run time is 6 hours
For n = 16, the run time is 2 months
For n = 18, the run time is 50 years
For n = 20, the run time is 200 centuries
Standard Analysis Techniques
 Constant time statements
 Analyzing Loops
 Analyzing Nested Loops
 Analyzing Sequence of Statements
 Analyzing Conditional Statements
Constant time statements
 Simplest case: O(1) time statements
 Assignment statements of simple data types
int x = y;
 Arithmetic operations:
x = 5 * y + 4 - z;
 Array referencing:
A[j] = 5;
 Array assignment:
 j, A[j] = 5;
 Most conditional tests:
if (x < 12) ...
Analyzing Loops[1]
 Any loop has two parts:
 How many iterations are performed?
 How many steps per iteration?

int sum = 0,j;

for (j=0; j < N; j++)
sum = sum +j;
 Loop executes N times (0..N-1)
 4 = O(1) steps per iteration
 Total time is N * O(1) = O(N*1) = O(N)
Analyzing Nested Loops[1]
 Treat just like a single loop and evaluate each level of
nesting as needed:
int j,k;
for (j=0; j<N; j++)
for (k=N; k>0; k--)
sum += k+j;
 Start with outer loop:
 How many iterations? N
 How much time per iteration? Need to evaluate inner loop
 Inner loop uses O(N) time
 Total time is N * O(N) = O(N*N) = O(N2)
Analyzing Nested Loops[2]
 What if the number of iterations of one loop
depends on the counter of the other?
int j,k;
for (j=0; j < N; j++)
for (k=0; k < j; k++)
sum += k+j;
 Analyze inner and outer loop together:
 Number of iterations of the inner loop is:
 0 + 1 + 2 + ... + (N-1) = O(N2)
Analyzing Sequence of Statements
 For a sequence of statements, compute their
complexity functions individually and add them
up
for (j=0; j < N; j++)
for (k =0; k < j; k++) O(N2)
sum = sum + j*k;
for (l=0; l < N; l++)
sum = sum -l;
cout<<“Sum=”<<sum; O(N)
O(1)
Total cost is O(N2) + O(N) +O(1) = O(N2)
SUM RULE
Analyzing Conditional Statements
What about conditional statements such as

if (condition)
statement1;
else
statement2;
where statement1 runs in O(N) time and statement2 runs in O(N2)
time?

We use "worst case" complexity: among all inputs of size N, that is the
maximum running time?
The analysis for the example above is O(N2)
Best Case
 Best case is defined as which input of size
n is cheapest among all inputs of size n.
 “The best case for my algorithm is n=1
because that is the fastest.” WRONG!
Misunderstanding
 Complexity Examples
What does the following algorithm compute?
procedure who_knows(a1, a2, …, an: integers)
m := 0
for i := 1 to n-1
for j := i + 1 to n
if |ai – aj| > m then m := |ai – aj|
{m is the maximum difference between any two
numbers in the input sequence}
Comparisons: n-1 + n-2 + n-3 + … + 1
= (n – 1)n/2 = 0.5n2 – 0.5n
Time complexity is O(n2).
 Complexity Examples
Another algorithm solving the same problem:
procedure max_diff(a1, a2, …, an: integers)
min := a1
max := a1
for i := 2 to n
if ai < min then min := ai
else if ai > max then max := ai
m := max - min
Comparisons: 2n - 2

Time complexity is O(n).