Program Efficiency and Complexity

Lec-2 & 3

1/31
Questions that will be answered
• What is a “good” or "efficient" program?
• How to measure the efficiency of a program?
• How to analyse a simple program?
• How to compare different programs?
• What is the big-O notation?
• What is the impact of input on program performance?
• What are the standard program analysis techniques?
• Do we need fast machines or fast algorithms?

2/31
Which is better?
The running time of a program.

Program easy to understand?

Program easy to code and debug?
Program making efficient use of resources?
Program running as fast as possible?

3/31
Measuring Efficiency
Ways of measuring efficiency:

• Run the program and see how long it takes

• Run the program and see how much memory it uses

•Lots of variables to control:

• What is the input data?

• What is the hardware platform?
• What is the programming language/compiler?
• Just because one program is faster than another right now, means it
will always be faster?

4/31
Measuring Efficiency

Want to achieve platform-independence

• Use an abstract machine that uses steps of time and units

of memory, instead of seconds or bytes

• - each elementary operation takes 1 step

• - each elementary instance occupies 1 unit of memory

5/31
A Simple Example

// Input: int A[N], array of N integers

// Output: Sum of all numbers in array A

int Sum(int A[], int N) {

int s=0;
for (int i=0; i< N; i++)
s = s + A[i];
return s;
}

How should we analyse this?

6/31
A Simple Example
Analysis of Sum
1.) Describe the size of the input in terms of one ore more
parameters:
- Input to Sum is an array of N ints, so size is N.

2.) Then, count how many steps are used for an input of that size:
- A step is an elementary operation such as
+, <, =, A[i]

7/31
A Simple Example
Analysis of Sum (2)
// Input: int A[N], array of N integers
// Output: Sum of all numbers in array A

int Sum(int A[], int N {

int s=0; 1
for (int i=0; i< N; i++)
2 3 4
s = s + A[i];
5 6 7 1,2,8: Once
return s; 3,4,5,6,7: Once per each iteration
}
8 of for loop, N iteration
Total: 5N + 3
The complexity function of the
algorithm is : f(N) = 5N +3

8/31
Analysis: A Simple Example
How 5N+3 Grows
Estimated running time for different values of N:

N = 10 => 53 steps
N = 100 => 503 steps
N = 1,000 => 5003 steps
N = 1,000,000 => 5,000,003 steps

As N grows, the number of steps grow in linear proportion to

N for this Sum function.

9/31
Analysis: A Simple Example
What Dominates?
What about the 5 in 5N+3? What about the +3?
• As N gets large, the +3 becomes insignificant
•5 is inaccurate, as different operations require varying
amounts of time

What is fundamental is that the time is linear in N.

Asymptotic Complexity: As N gets large, concentrate on the

highest order term:
• Drop lower order terms such as +3
• Drop the constant coefficient of the highest order term i.e. N

10/31
Analysis: A Simple Example
Asymptotic Complexity

• The 5N+3 time bound is said to "grow asymptotically" like N

• This gives us an approximation of the complexity of the

algorithm

• Ignores lots of (machine dependent) details, concentrate

on the bigger picture

11/31
Comparing Functions

Definition: If f(N) and g(N) are two complexity functions, we say

f(N) = O(g(N))

(read "f(N) as order g(N)", or "f(N) is big-O of g(N)")

if there are constants c and N0 such that for N  N0, T(N)  cN
f(N) c g(N)
for all sufficiently large N.

12/31
Comparing Functions
100n2 Vs 5n3, which one is better?

250000

200000

150000

100000

50000

0
1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3
1 2 3 4 5 6 7 8 9
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4
100n2 10 40 90 16 25 36 49 64 81 10 12 14 16 19 22 25 28 32 36 40 44 48 52 57 62 67 72 78 84 90 96 1E 1E1E
5n3 5 40 13 32 62 10 17 25 36 50 66 86 10 13 16 20 24 29 34 40 46 53 60 69 78 87 98 1E1E 1E 1E2E 2E2E

13/31
Comparing Functions
100n2 Vs 5n3, which one is better?

Differenec of functions

20000
10000
0
-10000 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33
-20000
-30000
-40000
-50000
-60000
-70000
-80000
-90000

14/31
Comparing Functions
Why is this useful?
As inputs get larger, any algorithm of a smaller order will
be more efficient than an algorithm of a larger order

0.05 N2 = O(N2)
Time (steps)

3N = O(N)

Input (size)
N = 60

15/31
Comparing Functions
What is the relationship between the following?

What is the polynomial representation for the following?

f1(n)

f2(n)

f3(n)
F(n)

f4(n)

Input size (n)

16/31
Big-O Notation

• Think of f(N) = O(g(N)) as

" f(N) grows at most like g(N)" or
" f grows no faster than g"
(ignoring constant factors, and for large N)

Important:
• Big-O is not a function!
• Never read = as "equals"
• Examples:
5N + 3 = O(N)
37N5 + 7N2 - 2N + 1 = O(N5)

17/31
Big-O Notation
350000
100n2
300000
5n3
250000
100n2 + 5n3
200000
5n4
150000

100000

50000

0
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33

18/31
Size does matter
Common Orders of Growth
Let N be the input size, and b and k be constants

O(k) = O(1) Constant Time

Increasing Complexity
O(logbN) = O(log N) Logarithmic Time
O(N) Linear Time
O(N log N)
O(N2) Quadratic Time
O(N3) Cubic Time
...
O(kN) Exponential Time

19/31
Size does matter
What happens if we double the input size N?

N log2N 5N N log2N N2 2N
8 3 40 24 64 256
16 4 80 64 256 65536
32 5 160 160 1024 ~109
64 6 320 384 4096 ~1019
128 7 640 896 16384 ~1038
256 8 1280 2048 65536 ~1076

If one execution takes 10-6 seconds,

2256 will take about 1061 centuries
20/31
Size does matter
Big Numbers
Suppose a program has run time O(n!) and the run time
for n = 10 is 1 second

For n = 12, the run time is 2 minutes

For n = 14, the run time is 6 hours
For n = 16, the run time is 2 months
For n = 18, the run time is 50 years
For n = 20, the run time is 200 centuries

21/31
Standard Analysis Techniques
Constant time statements
Simplest case: O(1) time statements
• Assignment statements of simple data types
int x = y;
• Arithmetic operations:
x = 5 * y + 4 - z;
• Array referencing:
A[j] = 5;
• Array assignment:
 j, A[j] = 5;
• Most conditional tests:
if (x < 12) ...

22/31
Standard Analysis Techniques
Analyzing Loops
Any loop has two parts:
1. How many iterations are performed?
2. How many steps per iteration?

int sum = 0,j;

for (j=0; j < N; j++)
sum = sum +j;

- Loop executes N times (0..N-1)

- 4 = O(1) steps per iteration
- Total time is N * O(1) = O(N*1) = O(N)

23/31
Standard Analysis Techniques
Analyzing Loops (2)
What about this for-loop?

int sum =0, j;

for (j=0; j < 100; j++)
sum = sum +j;

- Loop executes 100 times

- 4 = O(1) steps per iteration
- Total time is 100 * O(1) = O(100 * 1) = O(100) = O(1)

24/31
Standard Analysis Techniques
Analyzing Loops (3)
What about while-loops?
Determine how many times the loop will be executed:

bool done = false;

int result = 1, n;
scanf("%d", &n);
while (!done){
result = result *n;
n--;
if (n <= 1) done = true;
}
Loop terminates when done == true, which happens
after N iterations. Total time: O(N)

25/31
Standard Analysis Techniques
Nested Loops
Treat just like a single loop and evaluate each level of
nesting as needed:

int j,k;
for (j=0; j<N; j++)
for (k=N; k>0; k--)
sum += k+j;

Start with outer loop:

- How many iterations? N
- How much time per iteration? Need to evaluate inner loop
Inner loop uses O(N) time
Total time is N * O(N) = O(N*N) = O(N2)
PRODUCT RULE
26/31
Standard Analysis Techniques
Nested Loops (2)
What if the number of iterations of one loop depends on the
counter of the other?

int j,k;
for (j=0; j < N; j++)
for (k=0; k < j; k++)
sum += k+j;

Analyze inner and outer loop together:

- Number of iterations of the inner loop is:
0 + 1 + 2 + ... + (N-1) = O(N2) How?

27/31
Standard Analysis Techniques
Digression
When doing Big-O analysis, we sometimes have to compute
a series like:
1 + 2 + 3 + ... + (N-1) + N

What is the complexity of this?

Remember Gauss:

28/31
Standard Analysis Techniques
Sequence of Statements
For a sequence of statements, compute their complexity functions
individually and add them up

for (j=0; j < N; j++)

for (k =0; k < j; k++) O(N2)
sum = sum + j*k;
for (l=0; l < N; l++)
O(N)
sum = sum -l;
printf("sum is now %f", sum); O(1)

Total cost is O(N2) + O(N) +O(1) = O(N2)

SUM RULE
29/31
Standard Analysis Techniques
Conditional Statements
What about conditional statements such as

if (condition)
statement1;
else
statement2;
where statement1 runs in O(N) time and statement2 runs
in O(N2) time?

We use "worst case" complexity: among all inputs of size N, what

is the maximum running time?

The analysis for the example above is O(N2)

30/31
Fast machine Vs Fast Algorithm
Get a 10 times fast computer, that can do a job in 10 3
seconds for which the older machine took 104 seconds
.

Comparing the performance of algorithms with time

complexities T(n)s of n, n2 and 2n (technically not an
algorithm) for different problems on both the
machines.

Question: Is it worth buying a 10 times fast machine?

31/31