Analysis of Algorithms

Linear Sorting
 Types of Sorting Algorithms
• Non-recursive/Incremental comparison sorting
– Selection sort
– Bubble sort
– Insertion sort
• Recursive comparison sorting
– Merge sort
– Quick sort
– Heap sort
• Non-comparison linear sorting
– Count sort
– Radix sort 2
– Bucket sort
 How Fast Can We Sort?
• Selection Sort, Bubble Sort, Insertion Sort  O(n2)
• Merge Sort, Quick Sort, Heap Sort  O(n lg n)

• What is common to all these algorithms?

– Make comparisons between input elements

• Lower bound for comparison based sorting

– Theorem: To sort n elements, comparison sorts must
make Ω(nlgn) comparisons in the worst case.

3
 Counting Sort
• Assumptions:
– Sort n integers which are in the range [0 ... r]
– r is in the order of n, that is, r= O(n)
• Idea:
– For each element x, find the number of elements ≤ x
– Place x into its correct position in the output array

output array

4
Step 1

(i.e., frequencies) 5
Step 2

(i.e., cumulative sums)

6
 Algorithm
• Start from the last element of A
• Place A[i] at its correct place in the output array
• Decrease C[A[i]] by one

7
 Example
1 2 3 4 5 6 7 8 0 1 2 3 4 5

A 2 5 3 0 2 3 0 3 C 2 2 4 7 7 8
0 1 2 3 4 5
(cumulative sums)
C 2 0 2 3 0 1
(frequencies)
1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8

B 3 B 0 3
0 1 2 3 4 5 0 1 2 3 4 5

C 2 2 4 6 7 8 C 1 2 4 6 7 8

1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8

B 0 3 3 B 0 2 3 3
0 1 2 3 4 5 0 1 2 3 4 5

C 1 2 4 5 7 8 C 1 2 3 5 7 8
8
 Example (cont.)
1 2 3 4 5 6 7 8

A 2 5 3 0 2 3 0 3

1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8

B 0 0 2 3 3 B 0 0 2 3 3 3 5
0 1 2 3 4 5 0 1 2 3 4 5

C 0 2 3 5 7 8 C 0 2 3 4 7 7

1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8

B 0 0 2 3 3 3 B 0 0 2 2 3 3 3 5
0 1 2 3 4 5

C 0 2 3 4 7 8

9
 COUNTING-SORT
1 j n

A
Alg.: COUNTING-SORT(A, B, n, k) 0 k

1. for i ← 0 to r C
2. do C[ i ] ← 0 B
1 n

3. for j ← 1 to n
4. do C[A[ j ]] ← C[A[ j ]] + 1
5. C[i] contains the number of elements equal to i
6. for i ← 1 to r
7. do C[ i ] ← C[ i ] + C[i -1]
8. C[i] contains the number of elements ≤ i
9. for j ← n downto 1
10. do B[C[A[ j ]]] ← A[ j ]
11. C[A[ j ]] ← C[A[ j ]] - 1
10
 Analysis of Counting Sort
Alg.: COUNTING-SORT(A, B, n, k)
1. for i ← 0 to r Θ(r)
2. do C[ i ] ← 0
3. for j ← 1 to n
Θ(n)
4. do C[A[ j ]] ← C[A[ j ]] + 1
5. C[i] contains the number of elements equal to i
6. for i ← 1 to r
Θ(r)
7. do C[ i ] ← C[ i ] + C[i -1]
8. C[i] contains the number of elements ≤ i
9. for j ← n downto 1
10. do B[C[A[ j ]]] ← A[ j ] Θ(n)

11. C[A[ j ]] ← C[A[ j ]] - 1

Overall time: Θ(n + r) 11
 Analysis of Counting Sort
• Overall time: Θ(n + r)

• In practice we use COUNTING sort when r = O(n)

⇒ running time is Θ(n)

• Counting sort is stable

• Counting sort is not in place sort

12
 Radix Sort
• Represents keys as d-digit numbers in some
base-k
e.g., key = x1x2...xd where 0≤xi≤k-1

• Example: key=15
key10 = 15, d=2, k=10 where 0≤xi≤9
key2 = 1111, d=4, k=2 where 0≤xi≤1

13
 Radix Sort
• Assumptions
d=Θ(1) and k =O(n)
• Sorting looks at one column at a time
– For a d digit number, sort the least significant
digit first
– Continue sorting on the next least significant
digit, until all digits have been sorted
– Requires only d passes through the list

14
 RADIX-SORT
Alg.: RADIX-SORT(A, d)
for i ← 1 to d
do use a stable sort to sort array A on digit i
• 1 is the lowest order digit, d is the highest-order digit

15
 Analysis of Radix Sort

• Given n numbers of d digits each, where each

digit may take up to k possible values, RADIX-

SORT correctly sorts the numbers in Θ(d(n+k))

– One pass of sorting per digit takes Θ(n+k) assuming

that we use counting sort

– There are d passes (for each digit)

16
 Bucket Sort
• Assumption:
– the input is generated by a random process that distributes
elements uniformly over [0, 1)
• Idea:
– Divide [0, 1) into n equal-sized buckets
– Distribute the n input values into the buckets
– Sort each bucket (e.g., using quicksort)
– Go through the buckets in order, listing elements in each one

• Input: A[1 . . n], where 0 ≤ A[i] < 1 for all i

• Output: elements A[i] sorted
• Auxiliary array: B[0 . . n - 1] of linked lists, each list
initially empty
17
 Example - Bucket Sort
A 1 .78 B 0 /

2 .17 1 .17 .12 /

3 .39 2 .26 .21 .23 /

4 .26 3 .39 /

5 .72 4 /

6 .94 5 /

7 .21 6 .68 /

8 .12 7 .78 .72 /

9 .23 8 /
10 .68 9 .94 /
18
 Example - Bucket Sort
.12 .17 .21 .23 .26 .39 .68 .72 .78 .94 /

0 /

1 .12 .17 /

2 .21 .23 .26 /

3 .39 /

4 /

5 /

6 .68 /

7 .72 .78 /
Concatenate the lists from
8 / 0 to n – 1 together, in order
9 .94 / 19
 Analysis of Bucket Sort
Alg.: BUCKET-SORT(A, n)

for i ← 1 to n
O(n)
do insert A[i] into list B[nA[i]]

for i ← 0 to n - 1
Θ(nlogn)
do sort list B[i] with quicksort sort

concatenate lists B[0], B[1], . . . , B[n -1]

together in order O(n)

return the concatenated lists

Θ(n)
20
 Conclusion
• Any comparison sort will take at least nlgn to sort an
array of n numbers

• We can achieve a better running time for sorting if we

can make certain assumptions on the input data:
– Counting sort: each of the n input elements is an integer in the
range [0 ... r] and r=O(n)

– Radix sort: the elements in the input are integers represented

with d digits in base-k, where d=Θ(1) and k =O(n)

– Bucket sort: the numbers in the input are uniformly distributed

over the interval [0, 1)
21