Sorting

EECS 2011

www.eecs.yorku.ca/course_archive/2017-18/W/2011MF/

1/95
Overview

Review
Lower bound
Linear sorting

2/95
Sorting algorithms

Question
http://www.eecs.yorku.ca/course_archive/2017-18/W/
2011MF/sorting.html

3/95
Sorting algorithms

Answer
1 Merge sort

2 Heap sort
3 Selection sort
4 Quick sort
5 Bubble sort
6 Insertion sort

4/95
Big O, Big Omega and Big Theta
To capture the running time of algorithms, we use the following
notation. Let f , g ∈ N → N.

f ∈ O(g ) if
∃k > 0 : ∃m ∈ N : ∀n ≥ m : f (n) ≤ k g (n).
f is bounded from above by g

f ∈ Ω(g ) if
∃k > 0 : ∃m ∈ N : ∀n ≥ m : k g (n) ≤ f (n).
f is bounded from below by g

f ∈ Θ(g ) if
∃k` > 0 : ∃ku > 0 : ∃m ∈ N : ∀n ≥ m : k` g (n) ≤ f (n) ≤ ku g (n).
f is bounded from above and below by g
5/95
Sorting algorithms

selection sort Θ(n2 )

insertion sort Θ(n2 )
bubble sort Θ(n2 )
merge sort Θ(n log n)
quick sort Θ(n2 )
heap sort Θ(n log n)

6/95
Overview

Review
Lower bound
Linear sorting

7/95
How fast can we sort?

Question
How fast can we sort?

8/95
How fast can we sort?

Question
How fast can we sort?

Answer
We can sort n elements in Θ(n log n).

8/95
How fast can we sort?

Question
How fast can we sort?

Answer
We can sort n elements in Θ(n log n).

Question
Can we sort any faster?

8/95
How fast can we sort?

Question
How fast can we sort?

Answer
We can sort n elements in Θ(n log n).

Question
Can we sort any faster?

Question
Does there exist an algorithm that can sort n elements any faster?

8/95
Comparison sorts

Definition
A sorting algorithm is a comparison sort if the sorting is based on
comparisons between the elements (and not on the values of the
elements).

9/95
Comparison sorts

insertionSort(a) {
for (i = 1; i < a.length; i = i + 1) {
key = a[i];
j = i;
while (j > 0 && a[j - 1] > key) {
a[j] = a[j - 1];
j = j - 1;
}
a[j] = key;
}
}

Question
Is insertion sort a comparison sort?

10/95
Comparison sorts

insertionSort(a) {
for (i = 1; i < a.length; i = i + 1) {
key = a[i];
j = i;
while (j > 0 && a[j - 1] > key) {
a[j] = a[j - 1];
j = j - 1;
}
a[j] = key;
}
}

Question
Is insertion sort a comparison sort?

Question
Yes.
10/95
Comparison sorts
mergeSort(a, l, u) {
if (l + 1 < u) {
m = (l + u) / 2;
mergeSort(a, l, m);
mergeSort(a, m, u);
merge(a, l, m, u);
}
}
merge(a, l, m, u) {
i = l; j = m;
for (k = l; k < u; k = k + 1) {
if (i < m && (j >= u || a[i] <= a[j])) {
b[k] = a[i]; i = i + 1;
} else {
b[k] = a[j]; j = j + 1;
}
}
for (k = l, k < u; k = k + 1) {
a[k] = b[k];
} 11/95
Comparison sorts

Question
Is merge sort a comparison sort?

12/95
Comparison sorts

Question
Is merge sort a comparison sort?

Question
Yes.

12/95
Comparison sorts

Question
Is merge sort a comparison sort?

Question
Yes.

We will see examples of sorting algorithms that are not comparison

sorts later in this lecture.

12/95
Lower bound

Theorem
Any comparison sort must make Ω(n log n) comparisons in the
worst case.

13/95
Lower bound

Theorem
Any comparison sort must make Ω(n log n) comparisons in the
worst case.

Corollary
The worst case running time of any comparison sort is Ω(n log n).

13/95
Lower bound

Theorem
Any comparison sort must make Ω(n log n) comparisons in the
worst case.

Corollary
The worst case running time of any comparison sort is Ω(n log n).

Corollary
Merge sort and heap sort are asymptotically optimal comparison
sorts.

13/95
Proof of theorem
Without loss of generality, we may assume that all elements to be
sorted are different.
Question
Why can we assume that?

14/95
Proof of theorem
Without loss of generality, we may assume that all elements to be
sorted are different.
Question
Why can we assume that?

Answer
Assume that some elements are the same.

1, 6, 1, 6, 6

Add fractions to make them all different (in linear time).

1, 6, 1 12 , 6 13 , 6 23

Sort them.
1, 1 12 , 6, 6 13 , 6 23
Drop fractions (in linear time).

1, 1, 6, 6, 6
14/95
Comparisons

Given two elements ai and aj , we can compare then using

ai < aj
ai ≤ aj
ai = aj
ai ≥ aj
ai > aj

15/95
Comparisons

Question
If all elements are different, then it suffices to compare elements
using the comparator ≤. Why?

16/95
Comparisons

Question
If all elements are different, then it suffices to compare elements
using the comparator ≤. Why?

Answer
The other comparators <, ≥ and > can all be expressed in terms
of ≤ since

ai < aj iff ai ≤ aj
iff aj ≥ ai
iff aj > ai

16/95
Comparisons

Question
If all elements are different, then it suffices to compare elements
using the comparator ≤. Why?

Answer
The other comparators <, ≥ and > can all be expressed in terms
of ≤ since

ai < aj iff ai ≤ aj
iff aj ≥ ai
iff aj > ai

Corollary
If all elements are different, then each comparison sort algorithm
can be rewritten so it only uses ≤.
16/95
Decision tree

Definition
Given the number of elements to be sorted, the decision tree for a
sorting algorithm is a binary tree containing the comparisons
performed by the sorting algorithm.

17/95
Decision tree

insertionSort(a) {
for (i = 1; i < a.length; i = i + 1) {
key = a[i];
j = i;
while (j > 0 && key <= a[j - 1]) { // use <=
a[j] = a[j - 1];
j = j - 1;
}
a[j] = key;
}
}

18/95
Decision tree
The decision tree for insertion sort for three elements can be
depicted as follows.

a1 ≤ a0

a2 ≤ a0 a2 ≤ a1

a2 ≤ a1 h1, 0, 2i a2 ≤ a0 h0, 1, 2i

h2, 1, 0i h1, 2, 0i h2, 0, 1i h0, 2, 1i

19/95
Decision tree

selectionSort(a) {
for (i = 0; i < a.length; i = i + 1) {
min = i;
for (j = i + 1; j < a.length; j = j + 1) {
if (a[j] <= a[min]) { // use <=
min = j;
}
}
temp = a[i];
a[i] = a[min];
a[min] = temp;
}
}

20/95
Decision tree

Question
Draw the decision tree for selection sort for three elements.

21/95
Decision tree

Answer
a1 ≤ a0

a2 ≤ a1 a2 ≤ a0

a0 ≤ a1 a2 ≤ a0 a1 ≤ a0 a1 ≤ a2

h2, 1, 0i h1, 2, 0i h1, 0, 2i h2, 0, 1i h0, 1, 2i h0, 2, 1i

22/95
Height of decision tree

Observation
The worst case running time of a sorting algorithm of n elements

the maximal number of comparisons of a sorting algorithm of n

elements
=
the height of the decision tree for the sorting algorithm of n
elements.

23/95
Decision tree

Question
Given a decision tree for a sorting algorithm of n elements, what
are the leaves?

24/95
Decision tree

Question
Given a decision tree for a sorting algorithm of n elements, what
are the leaves?

Answer
Permutations of 0, 1, . . . , n − 1.

24/95
Decision tree

Question
Given a decision tree for a sorting algorithm of n elements, what
are the leaves?

Answer
Permutations of 0, 1, . . . , n − 1.

Question
How many permutations of 0, 1, . . . , n − 1 are there?

24/95
Decision tree

Question
Given a decision tree for a sorting algorithm of n elements, what
are the leaves?

Answer
Permutations of 0, 1, . . . , n − 1.

Question
How many permutations of 0, 1, . . . , n − 1 are there?

Answer
n × (n − 1) × · · · 2 × 1 = n!.

24/95
Properties of decision trees

Property (Proposition 8.7 of the textbook)

A binary tree of height h has at most 2h leaves.

25/95
Properties of decision trees

Property (Proposition 8.7 of the textbook)

A binary tree of height h has at most 2h leaves.

Property
A decision tree for a sorting algorithm of n elements has at least n!
leaves.

25/95
Properties of decision trees

Property (Proposition 8.7 of the textbook)

A binary tree of height h has at most 2h leaves.

Property
A decision tree for a sorting algorithm of n elements has at least n!
leaves.

Question
Why does a decision tree for a sorting algorithm of n elements
have at least n! leaves?

25/95
Properties of decision trees

Property (Proposition 8.7 of the textbook)

A binary tree of height h has at most 2h leaves.

Property
A decision tree for a sorting algorithm of n elements has at least n!
leaves.

Question
Why does a decision tree for a sorting algorithm of n elements
have at least n! leaves?

Answer
Because the n elements can be ordered in any order and, hence,
any permutation is a possible outcome.
25/95
Properties of decision trees

Property
A binary tree of height h has at most 2h leaves.

Property
A decision tree for a sorting algorithm of n elements has at least n!
leaves.

Conclusion
2h ≥ n! and, hence,
 n

h ≥ log(n!) ≥ log ( n2 ) 2 = n
2 log( n2 ) ∈ Ω(n log n).

26/95
Lower bound

Observation
The worst case running time of a sorting algorithm of n elements

the maximal number of comparisons of a sorting algorithm of n

elements
=
the height of the decision tree for the sorting algorithm of n
elements
≥
n
2 log( n2 ) ∈ Ω(n log n).

27/95
Lower bound

Theorem
Any comparison sort must make Ω(n log n) comparisons in the
worst case.

Corollary
The worst case running time of any comparison sort is Ω(n log n).

Corollary
Merge sort and heap sort are asymptotically optimal comparison
sorts.

28/95
Overview

Review
Lower bound
Linear sorting

29/95
But first...

... a break.

30/95
Overview

Review
Lower bound
Linear sorting
Bucket sort
Counting sort
Radix sort

31/95
Linear time sorting

Theorem
The worst case running time of any comparison sort is Ω(n log n).

Question
Can we do better if use information other than comparison of
elements?

32/95
Linear time sorting

Theorem
The worst case running time of any comparison sort is Ω(n log n).

Question
Can we do better if use information other than comparison of
elements?

Answer
Yes.

32/95
Overview

Review
Lower bound
Linear sorting
Bucket sort
Counting sort
Radix sort

33/95
Bucket sort

Assumption
All elements come from the interval [0, N − 1] for some N ≥ 2.

Main idea
1. Create N buckets.
2. Place each element in “its” bucket.
3. Concatenate the buckets.

34/95
Bucket sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
1. Create five buckets.

0 1 2 3 4

35/95
Bucket sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
2. Place each element in “its” bucket.

0 1 2 3 4

36/95
Bucket sort

Elements to be sorted: 4, 4, 1, 0, 2
2. Place each element in “its” bucket.

0 1 2 3 4

37/95
Bucket sort

Elements to be sorted: 4, 1, 0, 2
2. Place each element in “its” bucket.

0 1 2 3 4

2 4

38/95
Bucket sort

Elements to be sorted: 1, 0, 2
2. Place each element in “its” bucket.

0 1 2 3 4

2 4

39/95
Bucket sort

Elements to be sorted: 0, 2
2. Place each element in “its” bucket.

0 1 2 3 4

1 2 4

40/95
Bucket sort

Elements to be sorted: 2
2. Place each element in “its” bucket.

0 1 2 3 4

0 1 2 4

41/95
Bucket sort

Elements to be sorted:
2. Place each element in “its” bucket.

0 1 2 3 4

0 1 2 4

2 4

42/95
Bucket sort

3. Concatenate the buckets.

0 1 2 3 4

0 1 2 4

2 4

Sorted elements:

43/95
Bucket sort

3. Concatenate the buckets.

0 1 2 3 4

1 2 4

2 4

Sorted elements: 0

44/95
Bucket sort

3. Concatenate the buckets.

0 1 2 3 4

2 4

2 4

Sorted elements: 0, 1

45/95
Bucket sort

3. Concatenate the buckets.

0 1 2 3 4

2 4

Sorted elements: 0, 1, 2

46/95
Bucket sort

3. Concatenate the buckets.

0 1 2 3 4

Sorted elements: 0, 1, 2, 2

47/95
Bucket sort

3. Concatenate the buckets.

0 1 2 3 4

Sorted elements: 0, 1, 2, 2, 4

48/95
Bucket sort

3. Concatenate the buckets.

0 1 2 3 4

Sorted elements: 0, 1, 2, 2, 4, 4

49/95
Bucket sort

Question
How can we represent the buckets?

50/95
Bucket sort

Question
How can we represent the buckets?

Answer
As an array of lists.

50/95
Bucket sort

bucketSort(a, N) {
for (i = 0; i < N; i = i + 1) {
b[i] = empty list;
}
for (i = 0; i < a.length; i = i + 1) {
b[a[i]].add(a[i]);
}
j = 0;
for (i = 0; i < N; i = i + 1) {
while (!b[i].isEmpty()) {
a[j] = b[i].remove();
j++;
}
}
}

51/95
Bucket sort

Question
Express the worst case running time of bucket sort in terms of n
and N.

52/95
Bucket sort

Question
Express the worst case running time of bucket sort in terms of n
and N.

Answer
O(n + N).

52/95
Bucket sort

Question
Express the worst case running time of bucket sort in terms of n
and N.

Answer
O(n + N).

Note
If N ∈ O(n) then the worst case running time of bucket sort is
O(n).

52/95
Overview

Review
Lower bound
Linear sorting
Bucket sort
Counting sort
Radix sort

53/95
Counting sort

Assumption
All elements come from the interval [0, N − 1] for some N ≥ 2.

Main idea
1. Create a frequency table with N entries.
2. Keep track of the frequency of each element.
3. Compute the number of elements smaller than or equal to a
given element.
4. Place each element in “right” place.

54/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
1. Create a frequency table with five entries.

0 1 2 3 4

0 0 0 0 0

55/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
2. Keep track of the frequency of each element.

0 1 2 3 4

0 0 0 0 0

56/95
Counting sort

Elements to be sorted: 4, 4, 1, 0, 2
2. Keep track of the frequency of each element.

0 1 2 3 4

0 0 1 0 0

57/95
Counting sort

Elements to be sorted: 4, 1, 0, 2
2. Keep track of the frequency of each element.

0 1 2 3 4

0 0 1 0 1

58/95
Counting sort

Elements to be sorted: 1, 0, 2
2. Keep track of the frequency of each element.

0 1 2 3 4

0 0 1 0 2

59/95
Counting sort

Elements to be sorted: 0, 2
2. Keep track of the frequency of each element.

0 1 2 3 4

0 1 1 0 2

60/95
Counting sort

Elements to be sorted: 2
2. Keep track of the frequency of each element.

0 1 2 3 4

1 1 1 0 2

61/95
Counting sort

Elements to be sorted:
2. Keep track of the frequency of each element.

0 1 2 3 4

1 1 2 0 2

62/95
Counting sort

3. Compute the number of elements smaller than or equal to a

given element.

0 1 2 3 4

1 1 2 0 2

0 0 0 0 0

63/95
Counting sort

3. Compute the number of elements smaller than or equal to a

given element.

0 1 2 3 4

1 1 2 0 2

1 0 0 0 0

64/95
Counting sort

3. Compute the number of elements smaller than or equal to a

given element.

0 1 2 3 4

1 1 2 0 2

1 2 0 0 0

65/95
Counting sort

3. Compute the number of elements smaller than or equal to a

given element.

0 1 2 3 4

1 1 2 0 2

1 2 4 0 0

66/95
Counting sort

3. Compute the number of elements smaller than or equal to a

given element.

0 1 2 3 4

1 1 2 0 2

1 2 4 4 0

67/95
Counting sort

3. Compute the number of elements smaller than or equal to a

given element.

0 1 2 3 4

1 1 2 0 2

1 2 4 4 6

68/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

1 2 4 4 6

Sorted elements:

69/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

1 2 4 4 6

Sorted elements:

70/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

1 2 4 4 6

Sorted elements:

71/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

1 2 4 4 6

Sorted elements:

72/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

1 2 3 4 6

Sorted elements:

73/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

1 2 3 4 6

Sorted elements:

74/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

1 2 3 4 6

Sorted elements:

75/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

1 2 3 4 6

Sorted elements:

0 2

76/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 2 3 4 6

Sorted elements:

0 2

77/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 2 3 4 6

Sorted elements:

0 2

78/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 2 3 4 6

Sorted elements:

0 2

79/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 2 3 4 6

Sorted elements:

0 1 2

80/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 3 4 6

Sorted elements:

0 1 2

81/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 3 4 6

Sorted elements:

0 1 2

82/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 3 4 6

Sorted elements:

0 1 2

83/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 3 4 6

Sorted elements:

0 1 2 4

84/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 3 4 5

Sorted elements:

0 1 2 4

85/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 3 4 5

Sorted elements:

0 1 2 4

86/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 3 4 5

Sorted elements:

0 1 2 4

87/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 3 4 5

Sorted elements:

0 1 2 4 4

88/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 3 4 4

Sorted elements:

0 1 2 4 4

89/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 3 4 4

Sorted elements:

0 1 2 4 4

90/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 3 4 4

Sorted elements:

0 1 2 4 4

91/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 3 4 4

Sorted elements:

0 1 2 2 4 4

92/95
Counting sort

Elements to be sorted: 2, 4, 4, 1, 0, 2
4. Place each element in “right” place.

0 1 2 3 4

1 1 2 0 2

0 1 2 4 4

Sorted elements:

0 1 2 2 4 4

93/95
Counting sort
countingSort(a, N) {
for (i = 0; i < N; i = i + 1) {
f[i] = 0;;
}
for (i = 0; i < a.length; i = i + 1) {
f[a[i]] = f[a[i]] + 1;
}
for (i = 1; i < N; i = i + 1) {
f[i] = f[i-1] + f[i];
}
for (i = a.length - 1; i >= 0; i = i - 1) {
b[f[a[i]] = a[i];
f[a[i]] = f[a[i]] - 1;
}
for (i = 0; i < a.length; i++) {
a[i] = b[i];
}
94/95
Counting sort

Question
Express the worst case running time of bucket sort in terms of n
and N.

95/95
Counting sort

Question
Express the worst case running time of bucket sort in terms of n
and N.

Answer
O(n + N).

95/95
Counting sort

Question
Express the worst case running time of bucket sort in terms of n
and N.

Answer
O(n + N).

Note
If N ∈ O(n) then the worst case running time of bucket sort is
O(n).

95/95
 Radix Sort
Input:
• An array of N numbers.
• Each number contains d digits.
• Each digit between [0…k-1]
Output:
• Sorted numbers.

Each digit (column) can be sorted (e.g., using

Counting Sort).

Which digit to start from?

 RadixSort
344 125 125
125 134 224
333 Sort by which 143 Sort by which 225
134 digit first? 224 digit Second? 325
224 225 134
334 The most 243 The next most 333
143 significant. 344 significant. 334
225 333 143
325 334 243
243 325 344
All meaning in first sort lost.
 Radix Sort
1. Start from the least significant digit, sort
2. Sort by the next least significant digit
3. Are the last 2 columns sorted?
4. Generalize: after j iterations, the last j columns
are sorted

5. Loop invariant: Before iteration i, the keys

have been correctly stable-sorted with
respect to the i-1 least-significant digits.
 Radix sort
Radix-Sort(A,d)
• for i←1 to d
• do use a stable sort to sort A on digit i

Analysis:
Given n d-digit numbers where each digit takes on
up to k values, Radix-Sort sorts these numbers
correctly in Θ(d(n+k)) time.
 Radix sort – example (binary)

Sorting a sequence of 4-bit integers

1001 0010 1001 1001 0001

0010 1110 1101 0001 0010

1101 1001 0001 0010 1001

0001 1101 0010 1101 1101

1110 0001 1110 1110 1110