Analysis of Algorithms
CS 477/677
Linear Sorting
Instructor: George Bebis
(Chapter 8)
How Fast Can We Sort?
2
How Fast Can We Sort?
• Insertion sort: O(n2)
• Bubble Sort, Selection Sort: (n2)
• Merge sort: (nlgn)
• Quicksort: (nlgn) - average
• What is common to all these algorithms?
– These algorithms sort by making comparisons between the
input elements
3
Comparison Sorts
• Comparison sorts use comparisons between elements to
gain information about an input sequence a1, a2, …, an
• Perform tests:
ai < aj, ai ≤ aj, ai = aj, ai ≥ aj, or ai > aj
to determine the relative order of ai and aj
• For simplicity, assume that all the elements are distinct
4
Lower-Bound for Sorting
• Theorem: To sort n elements, comparison sorts must
make (nlgn) comparisons in the worst case.
5
Decision Tree Model
• Represents the comparisons made by a sorting algorithm on an input
of a given size.
– Models all possible execution traces
– Control, data movement, other operations are ignored
– Count only the comparisons
node
leaf:
6
Example: Insertion Sort
7
Worst-case number of comparisons?
• Worst-case number of comparisons depends on:
– the length of the longest path from the root to a leaf
(i.e., the height of the decision tree)
8
Lemma
• Any binary tree of height h has at most 2h leaves
Proof: induction on h
Basis: h = 0 tree has one node, which is a leaf
20 = 1
Inductive step: assume true for h-1
– Extend the height of the tree with one more level
– Each leaf becomes parent to two new leaves
No. of leaves at level h = 2 (no. of leaves at level h-1)
= 2 2h-1 4
= 2h 1 3 h-1
2 16 9 10 h
9
What is the least number of leaves in a
Decision Tree Model?
• All permutations on n elements must appear as one of
the leaves in the decision tree: n! permutations
• At least n! leaves
10
Lower Bound for Comparison Sorts
Theorem: Any comparison sort algorithm requires
(nlgn) comparisons in the worst case.
Proof: How many leaves does the tree have?
– At least n! (each of the n! permutations if the input appears as
some leaf) n!
– At most 2h leaves
h
n! ≤ 2h
h ≥ lg(n!) = (nlgn)
leaves
We can beat the (nlgn) running time if we use other operations than
comparing elements with each other! 11
Proof
(note: d is the same as h)
12
Counting Sort
• Assumptions:
– Sort n integers which are in the range [0 ... r]
– r is in the order of n, that is, r=O(n)
• Idea:
– For each element x, find the number of elements x
– Place x into its correct position in the output array
output array
13
Step 1
(i.e., frequencies) 14
Step 2
(i.e., cumulative sums)
15
Algorithm
• Start from the last element of A (i.e., see hw)
• Place A[i] at its correct place in the output array
• Decrease C[A[i]] by one
16
Example
1 2 3 4 5 6 7 8 0 1 2 3 4 5
A 2 5 3 0 2 3 0 3 C 2 2 4 7 7 8
0 1 2 3 4 5
(cumulative sums)
C 2 0 2 3 0 1
(frequencies)
1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
B 3 B 0 3
0 1 2 3 4 5 0 1 2 3 4 5
C 2 2 4 6 7 8 C 1 2 4 6 7 8
1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
B 0 3 3 B 0 2 3 3
0 1 2 3 4 5 0 1 2 3 4 5
C 1 2 4 5 7 8 C 1 2 3 5 7 8
17
Example (cont.)
1 2 3 4 5 6 7 8
A 2 5 3 0 2 3 0 3
1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
B 0 0 2 3 3 B 0 0 2 3 3 3 5
0 1 2 3 4 5 0 1 2 3 4 5
C 0 2 3 5 7 8 C 0 2 3 4 7 7
1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
B 0 0 2 3 3 3 B 0 0 2 2 3 3 3 5
0 1 2 3 4 5
C 0 2 3 4 7 8
18
COUNTING-SORT
1 j n
A
Alg.: COUNTING-SORT(A, B, n, k) 0 k
1. for i ← 0 to r C
2. do C[ i ] ← 0 B
1 n
3. for j ← 1 to n
4. do C[A[ j ]] ← C[A[ j ]] + 1
5. C[i] contains the number of elements equal to i
6. for i ← 1 to r
7. do C[ i ] ← C[ i ] + C[i -1]
8. C[i] contains the number of elements ≤ i
9. for j ← n downto 1
10. do B[C[A[ j ]]] ← A[ j ]
11. C[A[ j ]] ← C[A[ j ]] - 1
19
Analysis of Counting Sort
Alg.: COUNTING-SORT(A, B, n, k)
1. for i ← 0 to r (r)
2. do C[ i ] ← 0
3. for j ← 1 to n
(n)
4. do C[A[ j ]] ← C[A[ j ]] + 1
5. C[i] contains the number of elements equal to i
6. for i ← 1 to r
(r)
7. do C[ i ] ← C[ i ] + C[i -1]
8. C[i] contains the number of elements ≤ i
9. for j ← n downto 1
10. do B[C[A[ j ]]] ← A[ j ] (n)
11. C[A[ j ]] ← C[A[ j ]] - 1
Overall time: (n + r) 20
Analysis of Counting Sort
• Overall time: (n + r)
• In practice we use COUNTING sort when r = O(n)
running time is (n)
• Counting sort is stable
• Counting sort is not in place sort
21
Radix Sort
• Represents keys as d-digit numbers in some
base-k
e.g., key = x1x2...xd where 0≤xi≤k-1
• Example: key=15
key10 = 15, d=2, k=10 where 0≤xi≤9
key2 = 1111, d=4, k=2 where 0≤xi≤1
22
Radix Sort
• Assumptions
d=Θ(1) and k =O(n)
• Sorting looks at one column at a time
– For a d digit number, sort the least significant
digit first
– Continue sorting on the next least significant
digit, until all digits have been sorted
– Requires only d passes through the list
23
RADIX-SORT
Alg.: RADIX-SORT(A, d)
for i ← 1 to d
do use a stable sort to sort array A on digit i
• 1 is the lowest order digit, d is the highest-order digit
24
Analysis of Radix Sort
• Given n numbers of d digits each, where each
digit may take up to k possible values, RADIX-
SORT correctly sorts the numbers in (d(n+k))
– One pass of sorting per digit takes (n+k) assuming
that we use counting sort
– There are d passes (for each digit)
25
Correctness of Radix sort
• We use induction on number of passes through each
digit
• Basis: If d = 1, there’s only one digit, trivial
• Inductive step: assume digits 1, 2, . . . , d-1 are sorted
– Now sort on the d-th digit
– If ad < bd, sort will put a before b: correct
a < b regardless of the low-order digits
– If ad > bd, sort will put a after b: correct
a > b regardless of the low-order digits
– If ad = bd, sort will leave a and b in the
same order (stable!) and a and b are
already sorted on the low-order d-1 digits
26
Bucket Sort
• Assumption:
– the input is generated by a random process that distributes
elements uniformly over [0, 1)
• Idea:
– Divide [0, 1) into n equal-sized buckets
– Distribute the n input values into the buckets
– Sort each bucket (e.g., using quicksort)
– Go through the buckets in order, listing elements in each one
• Input: A[1 . . n], where 0 ≤ A[i] < 1 for all i
• Output: elements A[i] sorted
• Auxiliary array: B[0 . . n - 1] of linked lists, each list
initially empty
27
Example - Bucket Sort
A 1 .78 B 0 /
2 .17 1 .17 .12 /
3 .39 2 .26 .21 .23 /
4 .26 3 .39 /
5 .72 4 /
6 .94 5 /
7 .21 6 .68 /
8 .12 7 .78 .72 /
9 .23 8 /
10 .68 9 .94 /
28
Example - Bucket Sort
.12 .17 .21 .23 .26 .39 .68 .72 .78 .94 /
0 /
1 .12 .17 /
2 .21 .23 .26 /
3 .39 /
4 /
5 /
6 .68 /
7 .72 .78 /
Concatenate the lists from
8 / 0 to n – 1 together, in order
9 .94 / 29
Correctness of Bucket Sort
• Consider two elements A[i], A[ j]
• Assume without loss of generality that A[i] ≤ A[j]
• Then nA[i] ≤ nA[j]
– A[i] belongs to the same bucket as A[j] or to a bucket
with a lower index than that of A[j]
• If A[i], A[j] belong to the same bucket:
– sorting puts them in the proper order
• If A[i], A[j] are put in different buckets:
– concatenation of the lists puts them in the proper order
30
Analysis of Bucket Sort
Alg.: BUCKET-SORT(A, n)
for i ← 1 to n
O(n)
do insert A[i] into list B[nA[i]]
for i ← 0 to n - 1
(n)
do sort list B[i] with quicksort sort
concatenate lists B[0], B[1], . . . , B[n -1]
together in order O(n)
return the concatenated lists
(n)
31
Radix Sort Is a Bucket Sort
32
Running Time of 2nd Step
33
Radix Sort as a Bucket Sort
34
Effect of radix k
35
Problems
• You are given 5 distinct numbers to sort.
Describe an algorithm which sorts them using at
most 6 comparisons, or argue that no such
algorithm exists.
36
Problems
• Show how you can sort n integers in the range 1
to n2 in O(n) time.
37
Conclusion
• Any comparison sort will take at least nlgn to sort an
array of n numbers
• We can achieve a better running time for sorting if we
can make certain assumptions on the input data:
– Counting sort: each of the n input elements is an integer in the
range [0 ... r] and r=O(n)
– Radix sort: the elements in the input are integers represented
with d digits in base-k, where d=Θ(1) and k =O(n)
– Bucket sort: the numbers in the input are uniformly distributed
over the interval [0, 1)
38
