MTH 524

Algorithms

Lecture 2
 Analysis of Algorithms
Problem : Sorting

Input –
•  A sequence of numbers, say, a1, a2, …, an.
Output –
•  A rearrangement/permutation of those
numbers a′1, a′2, …, a′n, such that a′1 ≤ a′2
≤ … ≤ a′n.
 Sorting
Solution : Insertion Sort
Algorithm –
Insertion-Sort(A,n) // Sorts A[1], …, A[n]
for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Solution : Insertion Sort
key
j

sorted
sorted
Insertion-Sort(A,n) // Sorts A[1], …, A[n]
for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :
key = 1

A 9 1 5 10 4 8

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :
key = 1

A 9 9 5 10 4 8

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :
key = 5

A 1 9 5 10 4 8

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :
key = 5

A 1 9 9 10 4 8

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :
key = 10

A 1 5 9 10 4 8

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :

A 1 5 9 10 4 8

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :
key = 4

A 1 5 9 10 4 8

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :
key = 4

A 1 5 9 10 10 8

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :
key = 4

A 1 5 9 9 10 8

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :
key = 4

A 1 5 5 9 10 8

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :
key = 8

A 1 4 5 9 10 8

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :
key = 8

A 1 4 5 9 10 10

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :
key = 8

A 1 4 5 9 9 10

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Example :

A 1 4 5 8 9 10

sorted

Insertion-Sort(A,n) // Sorts A[1], …, A[n]

for j ← 2 to n
do key ← A[j]
i←j–1
while i > 0 and A[i] > key
do A[i+1] ← A[i]
i ← i-1
A[i+1] ← key
 Insertion Sort
Running Time :

•  Depends on input !!!

–  It is fast on an already sorted sequence !
–  What about the input that is reverse sorted ?
•  Depends on input size !!!
–  It is easier to sort short sequences than
longer ones.
•  Want upper bounds !!!
–  Guarantee to users.
 Analysis of Algorithms
Types of analyses :

•  Worst case
–  T(n) = max time on any input of size of n.
•  Average case
–  T(n) = expected time over all input of size n.
–  Need assumption on the probability
distribution of inputs.
•  Best case
–  This is cheating. Not very useful.
 Insertion Sort
What is insertion sort’s worst-case
running time?

•  Depends on computer !!!

–  Relative time (on same machine).
–  Absolute time (on different machines).

•  Asymptotic Analysis :
–  Ignore machine dependent constants.
–  Look at growth of T(n) as n→∞.
 Asymptotic Notation
Θ-Notation :
•  Mathematical Definition :
–  Θ(g(n)) =
{f(n): there exist positive constants c1, c2
and a positive integer n0 such that
0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0}.
•  Engineering Common Sense :
–  Drop lower order terms; ignore leading
constants.
–  Example : 5n3 + 73n2 + 5003n + 234 = Θ(n3)
 Asymptotic Performance

•  A Θ(n2) algorithm Θ(n ) 3

always beats a Θ(n3)

algorithm, when n Θ(n )2

gets large enough.

T(n)
•  An asymptotic
slower algorithm is
not always useless. n n 0

•  Asymptotic analysis is a useful tool to

help structure our thinking.
 Insertion Sort
•  Worst case :
–  Input is reverse sorted.
–  T(n)
= ∑j Θ(j)
[Arithmetic Series]
= Θ(n2) .
•  Is insertion sort fast?
–  For small n, its okay.
–  For large n, not at all.
 Merge Sort

Pseudocode –
Merge-Sort(A,n) // Sorts A[1], …, A[n]
1. If n = 1, done.
2. Recursively sort A[1], …, A[⎡n/2⎤]
and A[⎡n/2⎤+1], …, A[n].
3. Merge the two sorted lists.
Merge –
A key subroutine.
 Merge Sort
Merging two sorted arrays –

3 4 17 19

2 8 15 16
 Merge Sort
Merging two sorted arrays –

3 4 17 19

8 15 16

2
 Merge Sort
Merging two sorted arrays –

4 17 19

8 15 16

2 3
 Merge Sort
Merging two sorted arrays –

17 19

8 15 16

2 3 4
 Merge Sort
Merging two sorted arrays –

17 19

15 16

2 3 4 8
 Merge Sort
Merging two sorted arrays –

17 19

16

2 3 4 8 15
 Merge Sort
Merging two sorted arrays –

17 19

2 3 4 8 15 16
 Merge Sort
Merging two sorted arrays –

19

2 3 4 8 15 16 17
 Merge Sort
Merging two sorted arrays –

2 3 4 8 15 16 17 19
 Merge Sort
•  Analysis :
T(n)
Merge-Sort(A,n) // Sorts A[1], …, A[n]
1. If n = 1, done.
Θ(1) 2. Recursively sort A[1], …, A[⎡n/2⎤]
2 T(n/2) and A[⎡n/2⎤+1], …, A[n].
Θ(n) 3. Merge the two sorted lists.

•  Recurrence :
Θ(1) if n = 1 ;
T(n) =
2 T(n/2) + Θ(n) if n > 1 .