Set7 Sorting in Linear Time
Set7 Sorting in Linear Time
1
Comparison Sort Algorithms
• Insertion sort, heapsort, merge sort, and Quicksort are comparison sort
algorithms.
• Comparison sorts are sorting algorithms that are based only on comparison
between the input elements.
• Heapsort and merge sort have 𝜪(𝒏 𝐥𝐠 𝒏) while quicksort and BST sort has on
average 𝜪(𝒏 𝒍𝒈 𝒏) .
• Can they do better?
• Answer: No!!
• We will show a lower bound 𝛀(𝒏 𝐥𝐠 𝒏) for the sorting problem.
• Any comparison-based sorting algorithm must do at least 𝒏 𝐥𝐠 𝒏 amount
of work in the worst-case
2
Lower Bounds for Sorting
3
Decision Tree Model
Example:
• The decision tree for insertion sort
• Input: Array of three elements
4
Decision Tree
𝑎 :𝑎 A = [𝑎 , 𝑎 , 𝑎 ]
𝑎 :𝑎 𝑎 :𝑎
[𝑎 , 𝑎 , 𝑎 ] 𝑎 :𝑎 [𝑎 , 𝑎 , 𝑎 ] 𝑎 :𝑎
[𝑎 , 𝑎 , 𝑎 ] [𝑎 , 𝑎 , 𝑎 ] [𝑎 , 𝑎 , 𝑎 ] [𝑎 , 𝑎 , 𝑎 ]
5
Decision Tree
7: 10 sort A = [7,10,5]
Using Insertion sort
10: 5 7: 5
[𝑎 , 𝑎 , 𝑎 ] 7: 5 [𝑎 , 𝑎 , 𝑎 ] 10: 5
[𝑎 , 𝑎 , 𝑎 ] [𝑎 , 𝑎 , 𝑎 ] [𝑎 , 𝑎 , 𝑎 ] [𝑎 , 𝑎 , 𝑎 ]
6
Decision Tree
7: 10 sort A = [7,10,5]
Using Insertion sort
10: 5 7: 5
[𝑎 , 𝑎 , 𝑎 ] 7: 5 [𝑎 , 𝑎 , 𝑎 ] 10: 5
[𝑎 , 𝑎 , 𝑎 ] [5,7,10] [𝑎 , 𝑎 , 𝑎 ] [𝑎 , 𝑎 , 𝑎 ]
7
Decision Tree
Trees
– taken at chapter 3
9
Lower Corollary:
Heapsort and merge sort are
Bounds asymptotically optimal comparison sorts.
Using
Decision Proof:
10
Lower bounds for Algorithms
• Lower bounds are useful in that they would tell us what is the
absolute best we can achieve using ANY algorithm.
11
Tightness vs. Optimality
• Tightness refers to whether the lower bound Ω and upper bound 𝑂 of an algorithm’s
(worst/average/expected) running time are equal.
• If they are equal, we say the analysis is tight and we represent it with 𝚯
• This indicates how good your proof/analysis is – you want to make these bounds equal so
that we do not have any gaps.
• E.g. the worst-case running time of insertion sort is Ω 𝑛 and Ο 𝑛 . This implies that
the running time is Θ 𝑛
• Optimality refers to whether the upper bound 𝑂 of the algorithm is equal to the lower bound Ω
of the problem.
• If they are equal we say the algorithm is asymptotically optimal.
• This indicates whether your algorithm can be further improved or not. If it is optimal, you
cannot improve it further.
• E.g. merge sort is asymptotically optimal because its worst-case running time is
Ο 𝑛 lg 𝑛 which is equal to the sorting lower bound Ω 𝑛 lg 𝑛
12
Sorting in Linear Time
13
Counting Sort
14
Counting Sort
• Basic Idea:
• Count the number of elements less than x
• Place x accordingly.
15
Initial Call:
Example
1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
16
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 1 2 3 4 5
0 0 0 0 0 0
17
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 1 2 3 4 5
2 0 2 3 0 1
18
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
2 2 4 7 7 8
19
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
Step4: find the correct position of each element of 𝑨 starting from the
end of the array, using info from 𝑪 and place it in its correct position at 𝑩
0 1 2 3 4 5
2 2 4 7 7 8
1 2 3 4 5 6 7 8
20
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
Step4: find the correct position of each element of 𝑨 starting from the
end of the array, using info from 𝑪 and place it in its correct position at 𝑩
0 1 2 3 4 5
2 2 4 7 7 8
1 2 3 4 5 6 7 8
21
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 1 2 3 4 5
2 2 4 6 7 8
1 2 3 4 5 6 7 8
22
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
2 2 4 6 7 8
1 2 3 4 5 6 7 8
0 3
23
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 1 2 3 4 5
1 2 4 6 7 8
1 2 3 4 5 6 7 8
0 3
24
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
1 2 4 6 7 8
1 2 3 4 5 6 7 8
0 3 3
25
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 1 2 3 4 5
1 2 4 5 7 8
1 2 3 4 5 6 7 8
0 3 3
26
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
1 2 4 5 7 8
1 2 3 4 5 6 7 8
0 2 3 3
27
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 1 2 3 4 5
1 2 3 5 7 8
1 2 3 4 5 6 7 8
0 2 3 3
28
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
1 2 3 5 7 8
1 2 3 4 5 6 7 8
0 0 2 3 3
29
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 1 2 3 4 5
0 2 3 5 7 8
1 2 3 4 5 6 7 8
0 0 2 3 3
30
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 2 3 5 7 8
1 2 3 4 5 6 7 8
0 0 2 3 3 3
31
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 1 2 3 4 5
0 2 3 4 7 8
1 2 3 4 5 6 7 8
0 0 2 3 3 3
32
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 2 3 4 7 8
1 2 3 4 5 6 7 8
0 0 2 3 3 3 5
33
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 1 2 3 4 5
0 2 3 4 7 7
1 2 3 4 5 6 7 8
0 0 2 3 3 3 5
34
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 2 3 4 7 7
1 2 3 4 5 6 7 8
0 0 2 2 3 3 3 5
35
Initial Call:
Example 1 2 3 4 5 6 7 8
2 5 3 0 2 3 0 3
0 1 2 3 4 5
0 2 2 4 7 7
1 2 3 4 5 6 7 8
0 0 2 2 3 3 3 5
36
Analyzing Counting Sort
Ο(1)
Ο(𝑘)
Ο(𝑛)
Ο(𝑘)
Ο(𝑛)
37
An Issue with Counting Sort
1 0 0 0 …….. 0 1
38
General Notes
39
Radix Sort
Initial Call:
41
Example
• When
Assuming is a constant
43
Summary of Sorting Algorithms
Comparison-
Quicksort Θ(𝑛 ) Θ(𝑛 lg 𝑛) Y N
based Sorts Randomized Quicksort Expected: Θ 𝑛 lg 𝑛 - Y N
BST-Sort Θ(𝑛 ) Θ(𝑛 lg 𝑛) Y Y
Heapsort 𝑂(𝑛 lg 𝑛) 𝑂(𝑛 lg 𝑛) Y N
Counting Sort Θ(𝑘 + 𝑛) Θ(𝑘 + 𝑛) N Y
Distribution- Radix Sort Θ 𝑑 𝑘+𝑛 Θ 𝑑 𝑘+𝑛 N Y
based Sorts
Bucket Sort Θ 𝑛 Θ 𝑛 N Y
44
Other Sorting Algorithms
• Bubble Sort
• Selection Sort
• Shell Sort
• Bitonic Sort
• Timsort
45