Design and Analysis of

Algorithms
Presented By:

Israr Ali
 Properties

• An instance of a problem consists of all inputs needed to compute a

solution to the problem.

• An algorithm is said to be correct if, for every input instance, it halts

with the correct output.

• A correct algorithm solves the given computational problem. An

incorrect algorithm might not halt at all on some input instance, or it
might halt with other than the desired answer.
 Algorithm choice

• There are a large number of good sorting algorithms.

• Which algorithm is best for a given application
• • It depends on:
• The number of items to be sorted.
• The extent to which the items are already sorted.
• Possible restrictions on item values.
• The kind of storage device to be used.
• An algorithm is said to be correct if, for every input instance, it gives
the correct output.
 Algorithm efficiency

• Algorithms devised to solve the same problem often differ in their

efficiency.
• These differences can be much more significant than differences due
to hardware and software.
Example:
• Insertion sort algorithm takes time = c1n2 to sort n numbers, where
c1 is a constant.
• Merge sort algorithm takes time = c2 n lg n to sort n numbers, where
c2 is a constant.
 Algorithm efficiency

• c1 < c2, but constants less significant in the running time than the
input size.
• lg n < n, but insertion sort is usually faster than merge sort for small
input sizes.
• Once the input size n becomes large enough, merge sort will become
faster than insertion sort.
 Algorithm efficiency

Example:
Computer A executes one billion instruction per second (1 GHz) and
executes insertion sort (c1n2).
Computer B executes one million instruction per second (1 MHz) and
executes merge sort (c2 n lg n).
Assume c1=2 and c2=50.
Assume we need to sort 1 million numbers.
 Algorithm efficiency

• Computer A takes= 2*(107)2 instructions = 2000 s

109 instructions/second
• Computer B takes= 50*107* lg 107 instructions = 100 s
107 instructions/second
 Algorithm efficiency

• Imagine that we need to sort 10 million numbers;

• Insertion sort takes 2.3 days.
• Merge sort takes 20 minutes.

• We analyze algorithms to improve them and choose the best one for
a given problem.
 Algorithm Questions

• How to devise algorithms?

• there are various design techniques.
• example: divide and conquer.

• How to analyze algorithms?

• the task of determining how much computing time and storage and
algorithm requires.
• it needs mathematical skills.
• best case, average case, and worst case.
 Algorithm Questions

• How to validate algorithms?

• validation: showing that the algorithm computes the correct answer
for all legal inputs
• after validation, verification process takes place.
 Insertion sort

• solves the sorting problem

 a1 , a2 ,..., an 
• Input: A sequence of n numbers
 a1' , a2' ,..., a'n 
• Output: A permutation
a1'  a'2 ...  a'n
of the input sequence such that .
Insertion sort
Insertion Sort: Example

40 2 1 43 3 65 0 -1 58 3 42 4

2 40 1 43 3 65 0 -1 58 3 42 4

1 2 40 43 3 65 0 -1 58 3 42 4
Insertion Sort: Example

1 2 40 43 3 65 0 -1 58 3 42 4

1 2 3 40 43 65 0 -1 58 3 42 4

1 2 3 40 43 65 0 -1 58 3 42 4
Insertion Sort: Example

1 2 3 40 43 65 0 -1 58 3 42 4

0 1 2 3 40 43 65 -1 58 3 42 4

-1
0 1
0 2
1 3
2 40
3 40
43 43
65 65 58 3 42 4
Insertion Sort: Example

-1
0 1
0 2
1 3
2 40
3 40
43 43
65 58 65 3 42 4

-1
0 1
0 2
1 3
2 40
3 43
3 40
65 43
43 58 58
65 65 42 4

-1
0 1
0 2
1 3
2 40
3 43
3 40
65 42 43
43 65 58 65 4

-1
0 1
0 2
1 3
2 40
3 43
3 43
65
4 40
42 42
65 43
43 58 58
65 65
 Insertion sort – pseudo code

• InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j=j-1
}
A[j+1] = key
}
}
An Example: Insertion Sort

30 10 40 20 i =  j =  key = 
A[j] =  A[j+1] = 
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
 An Example: Insertion Sort

30 10 40 20 i=2 j=1 key = 10

A[j] = 30 A[j+1] = 10
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

30 30 40 20 i=2 j=1 key = 10

A[j] = 30 A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

30 30 40 20 i=2 j=1 key = 10

A[j] = 30 A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

30 30 40 20 i=2 j=0 key = 10

A[j] =  A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

30 30 40 20 i=2 j=0 key = 10

A[j] =  A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 20 i=2 j=0 key = 10

A[j] =  A[j+1] = 10
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 20 i=3 j=0 key = 10

A[j] =  A[j+1] = 10
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 20 i=3 j=0 key = 40

A[j] =  A[j+1] = 10
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 20 i=3 j=0 key = 40

A[j] =  A[j+1] = 10
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 20 i=3 j=2 key = 40

A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 20 i=3 j=2 key = 40

A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 20 i=3 j=2 key = 40

A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 20 i=4 j=2 key = 40

A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 20 i=4 j=2 key = 20

A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 20 i=4 j=2 key = 20

A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 20 i=4 j=3 key = 20

A[j] = 40 A[j+1] = 20
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 20 i=4 j=3 key = 20

A[j] = 40 A[j+1] = 20
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 40 i=4 j=3 key = 20

A[j] = 40 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 40 i=4 j=3 key = 20

A[j] = 40 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 40 i=4 j=3 key = 20

A[j] = 40 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 40 i=4 j=2 key = 20

A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 40 40 i=4 j=2 key = 20

A[j] = 30 A[j+1] = 40
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 30 40 i=4 j=2 key = 20

A[j] = 30 A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 30 40 i=4 j=2 key = 20

A[j] = 30 A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 30 40 i=4 j=1 key = 20

A[j] = 10 A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 30 30 40 i=4 j=1 key = 20

A[j] = 10 A[j+1] = 30
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 20 30 40 i=4 j=1 key = 20

A[j] = 10 A[j+1] = 20
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 An Example: Insertion Sort

10 20 30 40 i=4 j=1 key = 20

A[j] = 10 A[j+1] = 20
1 2 3 4

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}
Done!
David Luebke
 Loop invariants and Correctness

• Initialization: It is true prior to the first iteration of the loop.

• Maintenance: If it is true before an iteration of the loop, it remains
true before the next iteration.
• Termination: When the loop terminates, the invariant gives us a
useful property that helps show that the algorithm is correct.
 Insertion Sort

What is the precondition

InsertionSort(A, n) { for this loop?
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
}
}

David Luebke
 Insertion Sort

InsertionSort(A, n) {
for i = 2 to n {
key = A[i]
j = i - 1;
while (j > 0) and (A[j] > key) {
A[j+1] = A[j]
j = j - 1
}
A[j+1] = key
} How many times will
} this loop execute?

David Luebke
 Loop Invariants

• How to say that this algorithm is correct?

• an invariant of a loop is a property that holds before (and after) each
repetition
• We must show three things about a loop invariant:
• Initialization: It is true prior to the first iteration of the loop.
• Maintenance: If it is true before an iteration of the loop, it remains true
before the next iteration.
• Termination: When the loop terminates, the invariant gives us a useful
property that helps show that the algorithm is correct.
 Correctness of Insertion Sort

Invariant
• At the start of each iteration of the for loop of lines 1–8, the sub-array A[1 .. j-1] consists of
the elements originally in A[1 .. j-1] , but in sorted order.
• Initialization
• Maintenance:
• the loop moves the A[1 .. j-2] .. Elements to the right until it finds a suitable
place for A[j]. The sub-array A[1 .. j] then consists of the elements originally in
A[1 .. j] , but in sorted order
• Termination
• In the end we have j = n +1, substitute this in the “invariant” wording
 Analysis Insertion Sort

David Luebke
To compute T (n), the running time of INSERTION-SORT on an input of n
values, we sum the products of the cost and times columns, obtaining
• best-case running time is

We can express this running time as (an + b) for constants a and b that
depend on the statement costs c ; it is thus a linear function of n
i
• If the array is in reverse sorted order that is, in decreasing order the worst case results
We can express this worst-case running time as (an2 + bn + c) for
constants a, b, and c that again depend on the statement costs c ; it is
i

thus a quadratic function of n.

 Analyzing Insertion Sort

• What can T be?

• Best case -- inner loop body never executed
• ti = 1  T(n) is a linear function
• Worst case -- inner loop body executed for all previous
elements
• ti = i  T(n) is a quadratic function
• Average case
• ???

David Luebke
 Designing algorithms

• Insertion Sort – Incremental Approach

• The divide-and-conquer approach
• Recursive in nature
• Three parts
• Divide the problem into a number of sub-problems that are smaller instances
of the same problem.
• Conquer the sub-problems by solving them recursively. If the sub-problem
sizes are small enough, however, just solve the sub-problems in a
straightforward manner.
• Combine the solutions to the sub-problems into the solution for the original
problem.
 Merge Sort

• Divide: Divide the n-element sequence to be sorted into two

subsequences of n=2 elements each.
• Conquer: Sort the two subsequences recursively using merge sort.
• Combine: Merge the two sorted subsequences to produce the sorted
answer.
Mergesort

•Mergesort (divide-and-conquer)
• Divide array into two halves.

A L G O R I T H M S

A L G O R I T H M S divide
 Mergesort

•Mergesort (divide-and-conquer)
• Divide array into two halves.
• Recursively sort each half.

A L G O R I T H M S

A L G O R I T H M S divide

A G L O R H I M S T sort
 Mergesort

•Mergesort (divide-and-conquer)
• Divide array into two halves.
• Recursively sort each half.
• Merge two halves to make sorted whole.

A L G O R I T H M S

A L G O R I T H M S divide

A G L O R H I M S T sort

A G H I L M O R S T merge
 Merging

•Merge.
• Keep track of smallest element in each sorted half.
• Insert smallest of two elements into auxiliary array.
• Repeat until done.

smallest smallest

A G L O R H I M S T

A auxiliary array
 Merging

•Merge.
• Keep track of smallest element in each sorted half.
• Insert smallest of two elements into auxiliary array.
• Repeat until done.

smallest smallest

A G L O R H I M S T

A G auxiliary array
 Merging

•Merge.
• Keep track of smallest element in each sorted half.
• Insert smallest of two elements into auxiliary array.
• Repeat until done.

smallest smallest

A G L O R H I M S T

A G H auxiliary array
 Merging

•Merge.
• Keep track of smallest element in each sorted half.
• Insert smallest of two elements into auxiliary array.
• Repeat until done.

smallest smallest

A G L O R H I M S T

A G H I auxiliary array
 Merging

•Merge.
• Keep track of smallest element in each sorted half.
• Insert smallest of two elements into auxiliary array.
• Repeat until done.

smallest smallest

A G L O R H I M S T

A G H I L auxiliary array
 Merging

•Merge.
• Keep track of smallest element in each sorted half.
• Insert smallest of two elements into auxiliary array.
• Repeat until done.

smallest smallest

A G L O R H I M S T

A G H I L M auxiliary array
 Merging

•Merge.
• Keep track of smallest element in each sorted half.
• Insert smallest of two elements into auxiliary array.
• Repeat until done.

smallest smallest

A G L O R H I M S T

A G H I L M O auxiliary array
 Merging

•Merge.
• Keep track of smallest element in each sorted half.
• Insert smallest of two elements into auxiliary array.
• Repeat until done.

smallest smallest

A G L O R H I M S T

A G H I L M O R auxiliary array
 Merging

•Merge.
• Keep track of smallest element in each sorted half.
• Insert smallest of two elements into auxiliary array.
• Repeat until done.
first half
exhausted smallest

A G L O R H I M S T

A G H I L M O R S auxiliary array
 Merging

•Merge.
• Keep track of smallest element in each sorted half.
• Insert smallest of two elements into auxiliary array.
• Repeat until done.
first half
exhausted smallest

A G L O R H I M S T

A G H I L M O R S T auxiliary array
Merge Sort - Example
Merge sort – Merge procedure
 Invariants of Merge Procedure

At the start of each iteration of the for loop of lines 12–17, the sub-array A[p..k-1]
contains the k -p smallest elements of L[1 .. N1+1] and R[1 .. N2+1], in sorted order.
Moreover, L[i] and R[j] are the smallest elements of their arrays that have not been
copied back into A.
• Initialization: Prior to the first iteration of the loop, we have k = p, so
that the sub-array A[p..k-1] is empty.
• Maintenance
• Termination
Merge Sort procedure
 Analyzing divide-and-conquer algorithms
• recurrence equation or recurrence, which describes the overall
running time on a problem of size n in terms of the running time on
smaller inputs.
• Suppose that our division of the problem yields ‘a’ sub-problems,
each of which is ‘1/b’ the size of the original.
• It takes time T (n/b) to solve one sub-problem of size n=b, and so it
takes time ‘a T (n/b)’ to solve ‘a’ of them.
• D(n) : time to divide, C(n) : time to combine
 Analysis of merge sort

• Divide: The divide step just computes the middle of the sub-array,
which takes constant time. Thus, D(n) = Θ(1).
• Conquer: We recursively solve two sub-problems, each of size n=2,
which contributes 2T(n/2) to the running time.
• Combine: The combine step also takes constant time. Thus, C(n) =
Θ(1).

• The solution to this recurrence equation is T(n) = Θ(n lg n)