CA729 DESIGN AND
ANALYSIS OF ALGORITHMS
The Course
CA729 DESIGN AND ANALYSIS OF ALGORITHMS
Objective:
A rigorous introduction to the design and analysis
of algorithms
To learn about Time Complexity and various
algorithmic design methodologies
Textbook: Introduction to Algorithms, Cormen,
Leiserson, Rivest, Stein
Third edition
Instructor:
Algorithms as a
technology
Suppose computers were infinitely fast and
computer memory was free.
Would you have any reason to study algorithms?
The answer is yes
You would still like to demonstrate that your solution
method terminates and does so with the correct answer.
If computers were infinitely fast, any correct method
for solves the problem.
Implementation to be within the bounds of good software
engineering practice (i.e., well designed and documented)
Among many available algorithms, people use whichever
method was the easiest to implement.
Algorithms as a
technology
Of course, computers may be fast, but
they are not infinitely fast.
Memory may be cheap, but it is not free.
Computing time is therefore a bounded
resource, and so is space in memory.
These resources should be used
cleverly.
Algorithms that are efficient in terms of
time or space will help you do so.
Algorithms as a
technology
Algorithms devised to solve the same problem often differ
dramatically in their Efficiency.
Consider two algorithms
Insertion sort
c1 n 2
Merge sort
c2 n log n
C1 and C2 are not depends on n
Insertion sort usually has a smaller constant factor than
merge sort, so that c1 < c2
Constant factors can be far less significant in the running
time.
Insertion sort is usually faster than merge sort for small
input sizes.
For large n, merge sort runs faster.
Algorithms as a
technology
Consider two Computers
Computer A
Insertion Sort
One billion instructions
per second
100 times faster
C1 n2
c1 = 2
2n2
Computer B
Merge Sort
Ten million instructions
per second
C2 n log n
c2 = 50
50 n log n
Algorithms as a
technology
By using an algorithm whose running time grows more slowly,
even with computer B runs 20 times faster than computer A!
The advantage of merge sort is even more pronounced when
we sort ten million numbers:
Where insertion sort takes approximately 2.3 days, merge
sort takes under 20 minutes.
In general, as the problem size increases, so does the relative
advantage of merge sort increases.
The example above shows that algorithms, like computer
hardware, are a technology.
Total system performance depends on choosing efficient
algorithms as much as on choosing fast hardware.
Just as rapid advances are being made in other computer
technologies, they are being made in algorithms as well.
Insertion Sort
Sorting a hand of cards using insertion sort
Insertion Sort
Pseudocode
Insertion Sort
Example
Insertion Sort
At beginning of each iteration,
the subarray A[1j-1] constitutes sorted array
the subarray A[j+1n] corresponds to unsorted array
Loop Invariant:
At the start of each iteration elements A[1j-1] are the elements originally in positions from 1 to j-1
But, in sorted order
Correctness of Algorithm:
Loop invariant is used
Initialization:
J=2,
A[1j-1] = A[1]
Only one element
Maintenance:
For any j value, body of the for loop moves A[j-1], A[j-2], A[j-3],.
until find a proper position fie A[j].
Then inserts A[j] in correct position
Incrementing j for next iteration also preserves the loop invariant
Termination:
one position right
When for loop terminates?
Termination condition j > A.length = n
When j = n+1, we have the subarray A[1j-1] in sorted order which is equal to A[1]
Analysis of Algorithms
Analysis is performed with respect to a
computational model
We will usually use a generic uniprocessor
random-access machine (RAM)
All memory equally expensive to access
No concurrent operations
All reasonable instructions take unit time
Except, of course, function calls
Constant word size
Unless we are explicitly manipulating bits
Input Size
Time and space complexity
This is generally a function of the input
size
E.g., sorting, multiplication
How we characterize input size depends:
Sorting: number of input items
Multiplication: total number of bits
Graph algorithms: number of nodes & edges
Etc
Running Time
Number of primitive steps that are
executed
Except for time of executing a function
call
Most statements roughly require the
same amount of time
y=m*x+b
c = 5 / 9 * (t - 32 )
z = f(x) + g(y)
We can be more exact if need be
Analysis
Worst case
Provides an upper bound on running
time
An absolute guarantee
Average case
Provides the expected running time
Very useful, but treat with care: what is
average?
Random (equally likely) inputs
Real-life inputs
Analysis
Why it matters
Growth of functions
An Example: Insertion Sort
30
10
40
20
i = j = key =
A[j] =
A[j+1] =
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
30
10
40
20
i=2 j=1
A[j] = 30
key = 10
A[j+1] = 10
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
30
30
40
20
i=2 j=1
A[j] = 30
key = 10
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
30
30
40
20
i=2 j=1
A[j] = 30
key = 10
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
30
30
40
20
i=2 j=0
A[j] =
key = 10
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
30
30
40
20
i=2 j=0
A[j] =
key = 10
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
20
i=2 j=0
A[j] =
key = 10
A[j+1] = 10
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
20
i=3 j=0
A[j] =
key = 10
A[j+1] = 10
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
20
i=3 j=0
A[j] =
key = 40
A[j+1] = 10
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
20
i=3 j=0
A[j] =
key = 40
A[j+1] = 10
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
20
i=3 j=2
A[j] = 30
key = 40
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
20
i=3 j=2
A[j] = 30
key = 40
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
20
i=3 j=2
A[j] = 30
key = 40
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
20
i=4 j=2
A[j] = 30
key = 40
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
20
i=4 j=2
A[j] = 30
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
20
i=4 j=2
A[j] = 30
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
20
i=4 j=3
A[j] = 40
key = 20
A[j+1] = 20
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
20
i=4 j=3
A[j] = 40
key = 20
A[j+1] = 20
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
40
i=4 j=3
A[j] = 40
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
40
i=4 j=3
A[j] = 40
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
40
i=4 j=3
A[j] = 40
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
40
i=4 j=2
A[j] = 30
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
40
40
i=4 j=2
A[j] = 30
key = 20
A[j+1] = 40
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
30
40
i=4 j=2
A[j] = 30
key = 20
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
30
40
i=4 j=2
A[j] = 30
key = 20
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
30
40
i=4 j=1
A[j] = 10
key = 20
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
30
30
40
i=4 j=1
A[j] = 10
key = 20
A[j+1] = 30
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
20
30
40
i=4 j=1
A[j] = 10
key = 20
A[j+1] = 20
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
An Example: Insertion Sort
10
20
30
40
i=4 j=1
A[j] = 10
key = 20
A[j+1] = 20
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Done!
Insertion Sort
What is the precondition
for this loop?
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Insertion Sort
InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
How many times will
}
this loop execute?
}
Insertion Sort
Statement Effort
InsertionSort(A, n) {
for i = 2 to n { c1n
key = A[i] c2(n-1)
j = i - 1; c3(n-1)
while (j > 0) and (A[j] > key) { c4T
A[j+1] = A[j] c5(T-(n-1))
j = j - 1
c6(T-(n-1))
} 0
A[j+1] = key
c7(n-1)
} 0
}
T = t2 + t3 + + tn where ti is number of while expression evaluations for
the ith for loop iteration
Analyzing Insertion Sort
T(n) = c1n + c2(n-1) + c3(n-1) + c4T + c5(T - (n-1)) + c6(T - (n-1))
+ c7(n-1)
= c8T + c9n + c10
What can T be?
Best case -- inner loop body never executed
ti = 1 T(n) is a linear function
Worst case -- inner loop body executed for all
previous elements
ti = i T(n) is a quadratic function
Average case
???
Analysis
Simplifications
Ignore actual and abstract statement
costs
Order of growth is the interesting
measure:
Highest-order term is what counts
Remember, we are doing asymptotic analysis
As the input size grows larger it is the high order
term that dominates
Upper Bound Notation
We say InsertionSorts run time is O(n2)
Properly we should say run time is in O(n2)
Read O as Big-O (youll also hear it as
order)
In general a function
f(n) is O(g(n)) if there exist positive constants
c and n0 such that f(n) c g(n) for all n n0
Formally
O(g(n)) = { f(n): positive constants c and n0
such that f(n) c g(n) n n0
David Luebke
51
Insertion Sort Is O(n )
2
Proof
Suppose runtime is an2 + bn + c
If any of a, b, and c are less than 0 replace the
constant with its absolute value
an2 + bn + c (a + b + c)n2 + (a + b + c)n + (a + b
+ c)
3(a + b + c)n2 for n 1
Let c = 3(a + b + c) and let n0 = 1
Question
Is InsertionSort O(n3)?
Is InsertionSort O(n)?
