1b-Program Efficiency & Complexity Analysis
1b-Program Efficiency & Complexity Analysis
COMPLEXITY ANALYSIS
1
BINARY TREE
Input: 23, 53, 12, 14, 47, 36, 45, 37, 25, 9, 20
Ordered: 9, 12, 14, 20, 23, 25, 36, 37, 45, 47, 53
25
14 45
9 20 36 47
12 23 37 53
GOOD ALGORITHMS?
3
MEASURING EFFICIENCY
The efficiency of an algorithm is a measure of the number of
resources consumed in solving a problem of size n.
The resource we are most interested in is time
We can use the same techniques to analyze the consumption of other
resources, such as memory space.
Is it correct ?
4
FACTORS
Hardware
Operating System
Compiler
Size of input
Nature of Input
Algorithm
5
Which should be improved?
RUNNING TIME OF AN
ALGORITHM
Depends upon
Input Size
Nature of Input
7
SIMPLE EXAMPLE (1)
N = 10 => 53 steps
N = 100 => 503 steps
N = 1,000 => 5003 steps
N = 1,000,000 => 5,000,003 steps
10
WHAT DOMINATES IN PREVIOUS
EXAMPLE?
What about the +3 and 5 in 5N+3?
As N gets large, the +3 becomes insignificant
5 is inaccurate, as different operations require varying amounts of time and
does not have any significant importance
11
ASYMPTOTIC COMPLEXITY
12
COMPARING FUNCTIONS
4000
Which function is better?
10 n2 Vs n3 3500
3000
2500
10 n^2
2000
n^3
1500
1000
500
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
13
SIZE DOES MATTER
N log2N 5N N log2N N2 2N
8 3 40 24 64 256
16 4 80 64 256 65536
32 5 160 160 1024 ~109
64 6 320 384 4096 ~1019
128 7 640 896 16384 ~1038
256 8 1280 2048 65536 ~1076
14
SIZE DOES MATTER
Suppose a program has run time O(n!) and the run time for
n = 10 is 1 second
16
BIG OH NOTATION [1]
f(N) = O(g(N))
17
BIG OH NOTATION [2]
18
COMPARING FUNCTIONS
0.05 N2 = O(N2)
Time (steps)
3N = O(N)
Input (size)
N = 60
19
BIG-OH NOTATION
Simple Rule:
Drop lower order terms and constant factors
7n-3 is O(n)
8n2log n + 5n2 + n is O(n2log n)
20
BIG OMEGA NOTATION
If we wanted to say “running time is at least…” we use Ω
If f(n) and g(n) are two complexity functions, then we can say:
f(n) is Ω(g(n)) if there exist positive numbers c and n0 such that 0<=f(n)>=cΩ (n) for all n>=n0
21
BIG THETA NOTATION
If we wish to express tight bounds we use the theta notation, Θ
22
WHAT DOES THIS ALL MEAN?
If f(n) = Θ(g(n)) we say that f(n) and g(n) grow at the same
rate, asymptotically
23
WHICH NOTATION DO WE USE?
To express the efficiency of our algorithms which of the
three notations should we use?
Why?
log n Logarithmic: when n increases, so does run time, but much slower. Common in programs which solve
large problems by transforming them into smaller problems. Exp : binary Search
n Linear: run time varies directly with n. Typically, a small amount of processing is done on each
element. Exp: Linear Search
n log n When n doubles, run time slightly more than doubles. Common in programs which break a problem
down into smaller sub-problems, solves them independently, then combines solutions. Exp: Merge
n2 Quadratic: when n doubles, runtime increases fourfold. Practical only for small problems; typically
the program processes all pairs of input (e.g. in a double nested loop). Exp: Insertion Search
2n Exponential: when n doubles, run time squares. This is often the result of a natural, “brute force”
solution. Exp: Brute Force.
Note: logn, n, nlogn, n2>> less Input>>Polynomial
n3, 2n>>high input>> non polynomial
25
COMPLEXITY CLASSES
Time (steps)
26
STANDARD ANALYSIS
TECHNIQUES
Analyzing Loops
Arithmetic operations:
x = 5 * y + 4 - z;
Array referencing:
A[3] = 5;
Array assignment:
j, A[j] = 5;
28
Most conditional tests:
if (x < 12) ...
ANALYZING LOOPS[1]
Any loop has two parts:
How many iterations are performed?
How many steps per iteration?
int j,k;
for (j=0; j<N; j++)
for (k=N; k>0; k--)
sum += k+j;
int j,k;
for (j=0; j < N; j++)
for (k=0; k < j; k++)
sum += k+j;
Gauss figured out that the sum of the first n numbers is always:
34
SEQUENCE OF STATEMENTS
For a sequence of statements, compute their complexity
functions individually and add them up
35
CONDITIONAL STATEMENTS
What about conditional statements such as
if (condition)
statement1;
else
statement2;
Output:
3. O(N + M) time
Explanation: The first loop is O(N) and the second loop is O(M).
Since we don’t know which is bigger, we say this is O(N + M). This
can also be written as O(max(N, M)).
EXAMPLE-2
2. What is the time complexity of following code:
int a = 0;
for (i = 0; i < N; i++) { Options:
for (j = N; j > i; j--) { 1.O(N)
a = a + i + j; 2.O(N*log(N))
} 3.O(N * Sqrt(N))
} 4.O(N*N)
Output:
2. O(N*N)
Explanation:
The above code runs total no of times
= N + (N – 1) + (N – 2) + … 1 + 0
= N * (N + 1) / 2
= 1/2 * N2 + 1/2 * N
O(N2) times.
EXAMPLE-3
3. What is the time complexity of following code:
int a = 0, i = N;
while (i > 0) { Options:
a += i; 1.O(N)
i /= 2; 2.O(Sqrt(N))
}
3.O(N / 2)
4.O(log N)
Output:
4. O(log N)
EXAMPLE-4
4. What is the time complexity of following code:
int i, j, k = 0; Options:
for (i = n / 2; i <= n; i++) { 1.O(n)
for (j = 2; j <= n; j = j * 2) { 2.O(nLogn)
k = k + n / 2; 3.O(n^2)
}
4.O(n^2Logn)
}
Output:
4. O(nLogn)
Explanation: If you notice, j keeps doubling till it is less than or
equal to n. Number of times, we can double a number till it is less
than n would be log(n).
Let’s take the examples here.
for n = 16, j = 2, 4, 8, 16
for n = 32, j = 2, 4, 8, 16, 32
So, j would run for O(log n) steps.
i runs for n/2 steps.
So, total steps = O(n/ 2 * log (n)) = O(n*logn)
EXAMPLE-5
5. What does it mean when we say that an algorithm X
is asymptotically more efficient than Y?
Options:
1.X will always be a better choice for small inputs
2.X will always be a better choice for large inputs
3.Y will always be a better choice for small inputs
4.X will always be a better choice for all inputs
Output:
2. X will always be a better choice for large inputs
Explanation: In asymptotic analysis we consider growth of
algorithm in terms of input size. An algorithm X is said to be
asymptotically better than Y if X takes smaller time than y for all
input sizes n larger than a value n0 where n0 > 0.
DERIVING A RECURRENCE
EQUATION
So far, all algorithms that we have been analyzing have been non
recursive
However, if N ≥ 2, then running time T(N) is the cost of each step taken plus time
required to compute power(x,n-1). (i.e. T(N) = 2+T(N-1) for N ≥ 2)
42
How do we solve this? One way is to use the iteration method.
ITERATION METHOD
This is sometimes known as “Back Substituting”.
00304
Row 2
4 x 5 matrix
00570
4 rows
00000 5 columns
0 2Column
6 0 04 20 elements
6 nonzero elements
Examples:
• Diagonal
Only elements along diagonal may be nonzero
n x n matrix ratio is n/n2 = 1/n
• Tridiagonal
• Only elements on 3 central diagonals may be
nonzero
• Ratio is (3n-2)/n2 = 3/n – 2/n2
SPARSE MATRICES
• Lower triangular (?)
• Only elements on or below diagonal may be
nonzero
• Ratio is n(n+1)/(2n2) ~ 0.5
00570
A[4][5]={0, 0, 3, 0, 4,…….,0, 2, 6, 0, 0}
00000
02600
Array representation
1D Array of triples of type term
int row, col, value
row 0 0 1 1 3 3 1 2 2 2 3 4
column 2 4 2 3 1 2 3 0 1 3 1 0
value 3 4 5 7 2 6 2 3 5 6 7 4
A[18] = {1, 3, 2, 2, 0, 3, 2, 1, 5,…..}
RUNTIME
PERFORMANCE
Matrix Transpose
500 x 500 matrix with 1994 nonzero
elements
Run time measured on a 300MHz
Pentium II PC
2D array 210 ms
SparseMatrix 6 ms
PERFORMANCE
Matrix Addition
500 x 500 matrices with 1994 and 999
nonzero elements
2D array 880 ms
SparseMatrix 18 ms
SUMMARY
Algorithmscan be classified according to their
complexity => O-Notation
only relevant for large input sizes
58
THANK YOU
59