Faculty of Computers and

Information Technology

CS/IT 341

ALGORITHMS ANALYSIS AND DESIGN

Lecture 5
Types of Sorting Algorithms
• Non-recursive/Incremental comparison sorting
• Selection sort
• Bubble sort
• Insertion sort
• Recursive comparison sorting
• Merge sort
• Quick sort
• Heap sort
• Non-comparison linear sorting
• Count sort
• Radix sort
2
• Bucket sort
Types of Sorting Algorithms
• Non-recursive/Incremental comparison sorting
• Selection sort
• Bubble sort
• Insertion sort
• Recursive comparison sorting
• Merge sort
• Quick sort
• Heap sort
• Non-comparison linear sorting
• Count sort
• Radix sort
3
• Bucket sort
Insertion Sort
• Idea: like sorting a hand of playing cards
• Start with empty left hand and cards face down on the table.
• Remove one card at a time from the table, and insert it into the
correct position in the left hand
• compare it with each card already in the hand, from right to left
• The cards held in the left hand are sorted
• these cards were originally the top cards of the pile on the table

4
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 2.78 7.42 0.56 1.12 1.17 0.32 6.21 4.42 3.14 7.71

Iteration 0: step 0.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 2.78 7.42 0.56 1.12 1.17 0.32 6.21 4.42 3.14 7.71

Iteration 1: step 0.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 2.78 0.56
7.42 7.42
0.56 1.12 1.17 0.32 6.21 4.42 3.14 7.71

Iteration 2: step 0.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 2.78 2.78
0.56 0.56 7.42 1.12 1.17 0.32 6.21 4.42 3.14 7.71

Iteration 2: step 1.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 2.78 2.78
0.56 0.56 7.42 1.12 1.17 0.32 6.21 4.42 3.14 7.71

Iteration 2: step 2.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 2.78 1.12
7.42 7.42
1.12 1.17 0.32 6.21 4.42 3.14 7.71

10

Iteration 3: step 0.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12
2.78 2.78
1.12 7.42 1.17 0.32 6.21 4.42 3.14 7.71

11

Iteration 3: step 1.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12
2.78 2.78
1.12 7.42 1.17 0.32 6.21 4.42 3.14 7.71

12

Iteration 3: step 2.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 2.78 1.17
7.42 7.42
1.17 0.32 6.21 4.42 3.14 7.71

13

Iteration 4: step 0.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 1.17
2.78 2.78
1.17 7.42 0.32 6.21 4.42 3.14 7.71

14

Iteration 4: step 1.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 1.17 2.78 7.42 0.32 6.21 4.42 3.14 7.71

15

Iteration 4: step 2.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 1.17 2.78 0.32
7.42 7.42
0.32 6.21 4.42 3.14 7.71

16

Iteration 5: step 0.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 1.17 0.32
2.78 2.78
0.32 7.42 6.21 4.42 3.14 7.71

17

Iteration 5: step 1.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 1.12 0.32
1.17 1.17
0.32 2.78 7.42 6.21 4.42 3.14 7.71

18

Iteration 5: step 2.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 0.32
1.12 1.12
0.32 1.17 2.78 7.42 6.21 4.42 3.14 7.71

19

Iteration 5: step 3.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 0.56
0.32 0.32 1.12 1.17 2.78 7.42 6.21 4.42 3.14 7.71

20

Iteration 5: step 4.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.56 0.56
0.32 0.32 1.12 1.17 2.78 7.42 6.21 4.42 3.14 7.71

21

Iteration 5: step 5.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 6.21
7.42 7.42
6.21 4.42 3.14 7.71

22

Iteration 6: step 0.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 6.21
7.42 7.42
6.21 4.42 3.14 7.71

23

Iteration 6: step 1.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 6.21 4.42
7.42 7.42
4.42 3.14 7.71

24

Iteration 7: step 0.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 4.42
6.21 6.21
4.42 7.42 3.14 7.71

25

Iteration 7: step 1.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 4.42
6.21 6.21
4.42 7.42 3.14 7.71

26

Iteration 7: step 2.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 4.42 6.21 3.14
7.42 7.42
3.14 7.71

27

Iteration 8: step 0.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 4.42 3.14
6.21 6.21
3.14 7.42 7.71

28

Iteration 8: step 1.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 3.14
4.42 4.42
3.14 6.21 7.42 7.71

29

Iteration 8: step 2.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 3.14
4.42 4.42
3.14 6.21 7.42 7.71

30

Iteration 8: step 3.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 3.14 4.42 6.21 7.42 7.71

31

Iteration 9: step 0.
Insertion Sort
InsertionSort(A, n)
1. for i = 1 to n-1
2. key = A[i]
3. j = i – 1
4. while (j >= 0) and (A[j] > key)
5. A[j+1] = A[j]
6. j = j – 1
7. A[j+1] = key

Array index 0 1 2 3 4 5 6 7 8 9
Value 0.32 0.56 1.12 1.17 2.78 3.14 4.42 6.21 7.42 7.71

32

Iteration 10: DONE!.

 Insertion Sort Analysis
InsertionSort(A, n) { cost times
for i = 1 to n-1 { c1 n-1
key = A[i] c2 n-1
j = i - 1; c3 n-1
c4 
n 1
while (j>=0) and (A[j] > key){ t
i 1 i
A[j+1] = A[j] c5 
n 1
t
i 1 i
j = j – 1 c6 
n 1
t
i 1 i
}
A[j+1] = key c7 n-1
}
} ti: # of times the while statement is executed at iteration i 33
n 1 n 1 n 1
T (n)  c1 (n  1)  c 2 (n  1)  c3 (n  1)  c 4  t i  c5  t i  c6  t i  c7 (n  1)
i 1 i 1 i 1
 Best Case Analysis
“while j >= 0 and A[j] > key”
• The array is already sorted in ascending order
• Inner loop will not be executed
• The number of moves: 2*(n-1)
• The number of key comparisons: (n-1)

T(n) = c1(n-1) + c2(n -1) + c3(n -1) + c4(n -1) + c7(n-1)

= (c1 + c2 + c3 + c4 + c7)n - (c1 + c2 + c3 + c4 + c7)

= an + b = (n)

34
n 1 n 1 n 1
T (n)  c1 (n  1)  c 2 (n  1)  c3 (n  1)  c 4  t i  c5  t i  c6  t i  c7 (n  1)
i 1 i 1 i 1
 Worst Case Analysis
“while j >= 0 and A[j] > key”
• The array is in reverse sorted order
• The number of moves: 2*(n-1)+(1+2+...+n-1)= 2*(n-1)+ n*(n-1)/2
• The number of key comparisons: (1+2+...+n-1)= n*(n-1)/2
• Always A[j] > key in while loop test
• Have to compare key with all elements to the left of the j-th position
 compare with i-1 elements  tj = i n 1
n(n  1)

i 1
t i 
2
 n(n  1) 
T (n)  c1 (n  1)  c 2 (n  1)  c3 (n  1)  (c 4  c5  c6 )   c7 (n  1)
 2 
 an  bn  c
2
a quadratic function of n
T(n) = (n2) order of growth in n2
35
n 1 n 1 n 1
T (n)  c1 (n  1)  c 2 (n  1)  c3 (n  1)  c 4  t i  c5  t i  c6  t i  c7 (n  1)
i 1 i 1 i 1
Insertion Sort Summary
• Running time not only depends on the size of the array but also
the contents of the array.

• Average-case:  O(n2)
• We have to look at all possible initial data organizations.

• Advantages
• Good running time for “almost sorted” arrays (n)

• Disadvantages
• (n2) running time in worst and average case
•  n2/2 comparisons and exchanges 36
Types of Sorting Algorithms
• Non-recursive/Incremental comparison sorting
• Selection sort
• Bubble sort
• Insertion sort
• Recursive comparison sorting
• Merge sort
• Quick sort
• Heap sort
• Non-comparison linear sorting
• Count sort
• Radix sort
37
• Bucket sort
Recursive Comparison Sorting
There are many ways to design algorithms:

Insertion/Bubble/Selection sort uses an incremental approach

• Having sorted to array A[i..j]
• We insert the single element A[j+1] into its proper place, to get a sorted
array A[i..j+1]

An alternative design approach is “Divide and Conquer”

• Divide the problems into a number of sub-problems
• Conquer the sub-problems by solving them recursively. If sub-problem
sizes are small enough, just solve them in a straight forward manner.
• Combine the solutions to the sub-problems into the solution for the
original problem. 38
Heap Sort
A comparison-based recursive sorting algorithm, that is
implemented using the Heap data structure.

It is similar to the selection sort where we first find the minimum

element and place the minimum element at the beginning. Repeat
the same process for the remaining elements.

39
Heap Data Structure
Uses Heap data structure to manage data during algorithm execution

A (Max/Min)-Heap is a complete, left-justified binary tree in which

no node has a value greater/lesser than the value in its parent
• A[parent[i]]  A[i] (max-heap)
• A[parent[i]]  A[i] (min-heap)

Largest/Smallest element is stored at the root.

Subtree rooted at a node contains values no larger/smaller than that

at the node.

Applications
• Priority Queue Implementation
• Dijkstra's Algorithm (finding the Shortest Path in a Graph) 40
• Heap Sort
Heap Property (1)

41
Heap Property (2)
20
Larger than
its parent
15 8

4 10 7 9
Not a Heap

25

Not left-justified 10 12

9 5 11 7
42

1 6 3 8
Maintaining the Max-Heap Property
Suppose a node is smaller than a child, violating heap property. To
maintain heap property, exchange the node with the larger child,
moving the node down the tree until it is not smaller than its children

Alg: MAX-HEAPIFY(A, i, n)
1. l ← LEFT(i)
2. r ← RIGHT(i)
3. if l ≤ n and A[l] > A[i]
4. then largest ←l
5. else largest ←i
6. if r ≤ n and A[r] > A[largest]
7. then largest ←r Complexity of MAX-HEAPIFY()?
8. if largest  i
9. then exchange A[i] ↔ A[largest] O(log n) where log n is the
43
10. MAX-HEAPIFY(A, largest, n) height of the tree
Array Representation of Heaps
A heap can be stored as an array A.
• Root of tree is A[1] 1 2 3 4 5 6 7 8 9 10
26 24 20 18 17 19 13 12 14 11
• Left child of A[i] = A[2i]
Max-heap as an array
• Right child of A[i] = A[2i + 1]
• Parent of A[i] = A[i/2] 26
• Heapsize[A] ≤ length[A]
24 20
The elements in the subarray
A[(n/2+1) .. n] are leaves
18 17 19 13

Max-heap as a binary tree

12 14 11 44

Node Insertion: Insert in last row from left to right

Node Deletion: Delete from last row from right to left
Building a Heap
A given array containing n elements can be converted into a max-
heap by applying MAX-HEAPIFY on all interior nodes
• The elements in the subarray A[1 … n/2] are interior nodes
• The elements in the subarray A[(n/2+1) .. n] are leaves

Alg: BUILD-MAX-HEAP(A)
1. n = length[A]
2. for i ← n/2 downto 1
O(n)
3. do MAX-HEAPIFY(A, i, n) O(log n)
1

4
Complexity of BUILD-MAX-HEAP? O(n log n)
2 3

1 3 45
4 5 6 7
A: 4 1 3 2 16 9 10 14 8 7 2 16 9 10
8 9 10

14 8 7
 Example: A: 4 1 3 2 16 9 10 14 8 7

i=5 i=4 i=3

1 1 1

4 4 4
2 3 2 3 2 3

1 3 1 3 1 3
4 5 6 7 4 5 6 7 4 5 6 7

8
2 9 10
16 9 10 8
2 9 10
16 9 10 8
14 9 10
16 9 10
14 8 7 14 8 7 2 8 7

i=2 i=1
1 1 1

4 4 16
2 3 2 3 2 3

1 10 16 10 14 10
4 5 6 7 4 5 6 7 4 5 6 7

8
14 9 10
16 9 3 8
14 9 10
7 9 3 8
8 9 10
7 9 3
2 8 7 2 8 1 2 4 1
46
Heap Sort
A given array which has been heapified does not mean it is sorted

25

22 17

14 21 9 11

Notice that the largest number is at the root. It can be swapped

with the rightmost leaf at the deepest level (its original position)
and heap size decreased.

The new root has lost heap property. Heapify the new root and
repeat this process until only one node remain. 47
Example: A=[7, 4, 3, 1, 2]

MAX-HEAPIFY(A, 1, 4) MAX-HEAPIFY(A, 1, 3) MAX-HEAPIFY(A, 1, 2)

MAX-HEAPIFY(A, 1, 1) 48
Heap Sort Pseudocode
HEAPSORT(A)
1. BUILD-MAX-HEAP(A) O(n log n)
2. for i ← length[A] downto 2 O(n)
3. do exchange A[1] ↔ A[i] n-1 times
4. MAX-HEAPIFY(A, 1, i - 1) O(log n)

What is the running time of HEAPSORT()? O(n log n)

Why Heap Sort?

• Heap Sort is always O(n log n) and is in-place
• Quicksort is usually O(n log n) but in the worst case slows to O(n2)
• Quicksort is generally faster, but Heapsort is better in time-critical 49
applications
How Fast Can We Sort?
• Selection Sort, Bubble Sort, Insertion Sort  O(n2)
• Merge Sort, Quick Sort, Heap Sort  O(n log n)

• What is common to all these algorithms?

• Make comparisons between input elements

• Lower bound for comparison based sorting

• Theorem: To sort n elements, comparison sorts must make (n log n)
comparisons in the worst case.

• Proof: Draw decision tree and calculate asymptotic height of any

decision tree for sorting n elements
50
Lower Bound For Comparison Sorting
Consider sorting three numbers a1, a2, a3.
There are 3! = 6 possible combinations:
(a1, a2, a3), (a1, a3, a2) , (a3, a2, a1) (a3, a1, a2), (a2, a1, a3) , (a2, a3, a1)

• What’s the minimum # of leaves? n!

51
• What’s the maximum # of leaves of a binary tree of height h? 2h
• minimum # of leaves ≤ maximum # of leaves
Lower Bound For Comparison Sorting
• So we have…
n!  2h
• Taking logarithms:
lg (n!)  h
n n
n n
n!    h  lg 
• Stirling’s approximation tells us: e e
 n lg n  n lg e

 n lg n 

• Thus the minimum height of a decision tree is (n log n) therefore

the time to comparison sort n elements is (n log n)

52
• Corollary: Quick Sort, Heap Sort and Merge Sort are asymptotically
optimal comparison sorts. Can we do any better than (n log n)?
Types of Sorting Algorithms
• Non-recursive/Incremental comparison sorting
• Selection sort
• Bubble sort
• Insertion sort
• Recursive comparison sorting
• Merge sort
• Quick sort
• Heap sort
• Non-comparison linear sorting
• Count sort
• Radix sort
53
• Bucket sort
 Counting Sort
• Assumptions:
• n integers which are in the range [0,k]
• k is in the order of n, that is, k=O(n).

• Idea:
• For each element x, find the number of elements ≤ x
• Place x into its correct position in the output array

54
https://www.cs.usfca.edu/~galles/visualization/CountingSort.html
Counting Sort
1 2 3 4 5 6 7 8

A 2 5 3 0 2 3 0 3 Input Array
0 1 2 3 4 5

C 2 0 2 3 0 1 Auxiliary Array
0 1 2 3 4 5

C 2 2 4 7 7 8

1 2 3 4 5 6 7 8

B 0 0 2 2 3 3 3 5 Output Array

0 1 2 3 4 5
55
C 0
2 2 2
1 3 7
4 4
6 7 7
5 8
Counting Sort
Counting-Sort(A, B, k)
for i = 1 to k
do C[i] = 0 (k)
for j = 1 to length[A]
do C[A[j]] = C[A[j]] + 1 (n)
// C[i] now contains the number of elements = i
for i = 2 to k
do C[i] = C[i] + C[i-1] (k)
// C[i] now contains the number of elements ≤ i
for j = length[A] downto 1
do B[C[A[j]]] = A[j]
C[A[j]] = C[A[j]] - 1 (n)

56
Running time is (n+k) (or (n) if k = O(n))!
Counting Sort
• Is counting sort stable
• The input order is maintained among items with equal keys

• Cool! Why don’t we always use counting sort?

• Because it depends on range k of elements

• Could we use counting sort to sort 32 bit integers? Why or why not?
• Answer: no, k too large (232 = 4,294,967,296)

57
Radix Sort
• Represents keys as d-digit numbers in some base-k
• key = x1x2...xd where 0≤xi≤k-1

• Example: key=15
• key10 = 15, d=2, k=10 where 0≤xi≤9
• key2 = 1111, d=4, k=2 where 0≤xi≤1

• Assumptions: d=O(1) and k =O(n)

• Sorting looks at one column at a time
• For a d digit number, sort the least significant digit first
• Continue sorting on the next least significant digit, until all
digits have been sorted
• Requires only d passes through the list
58
https://www.cs.usfca.edu/~galles/visualization/RadixSort.html
Radix Sort
RadixSort(A, d)
for i=1 to d
d times
StableSort(A) on digit i (n+k)

Running time is (dn+dk) (or (n) if k = O(n) and d = O(1))

59

(stable sort: preserves order of identical elements)

Radix Sort
• Problem: sort 1 million 64-bit numbers
• Treat as four-digit radix 216 numbers
• Can sort in just four passes with radix sort!

• So why would we ever use anything but radix sort?

60
Bucket Sort
• Assumption:
• Input numbers are uniformly distributed in [0,1).
• Suppose input size is n.

• Idea:
• Divide [0,1) into n equal-sized buckets (k= (n))
• Distribute the n input values into the buckets
• Sort each bucket (insertion sort as default).
• Go through the buckets in order, listing elements in each one

• Input: A[1 . . n], where 0 ≤ A[i] < 1 for all i

61
• Output: elements A[i] sorted
 Bucket Sort
Step 1: Distribute into Buckets
.78 0

.17 1 .12 .17

.39 2 .21 .23 .26
.26 3 .39
.72 4
Step 2: Sort within each Bucket
.94 5
.21 6 .68
.12 7 .72 .78
.23 8
.68 9 .94 62
Step 3: Concatenate all Buckets

.12 .17 .21 .23 .26 .39 .68 .72 .78 .94 /
Bucket Sort
BUCKET-SORT(A, n)
for i ← 0 to n-1
insert A[i] into list B[⌊n A[i]⌋] O(n)
for i ← 0 to n-1
sort list B[i] with insertion sort O(ni2)
concatenate the lists B[0], B[1], . . ., B[n - 1] in order O(n)

Where ni is the size of bucket B[i].

Thus T(n) = (n) + σ𝒏−𝟏 𝟐
𝒊=𝟎 𝑶(𝒏𝒊 )
= (n) + 𝑶(n) = (n)
63
Summary: Sorting Algorithms
Insertion Sort: Suitable only for small n ≤ 50 or nearly sorted inputs
Merge Sort: Guaranteed to be fast even in its worst case; stable
Quick Sort: Most useful general-purpose sorting for very little memory
requirement and fastest average time. (Choose the median of
three elements as pivot in practice
Heap Sort: Requiring minimum memory and guaranteed to run fast;
average and maximum time both roughly twice the average
time of quick sort. Useful for time-critical applications

Counting Sort: Very useful when the keys have small range; stable;
memory space for counters and for 2n records.
Radix Sort: Appropriate for keys either rather short or with an
lexicographic collating sequence. 64

Bucket Sort: Assuming keys to have uniform distribution.

The End

Questions?

65