Program: B.

Tech
Course Code: AIML303/AIDS303/IOT303
Course Name: Design & Analysis of Algorithms

Selection Sort, Bubble Sort

Sorting in Linear Time – Counting
Sort
By:
Dr. Monika Bansal
Selection Sort

• The Selection Sort algorithm finds the lowest value in an array and moves it to
the front of the array.

• Time Complexity: θ(n2) (in all three cases: Best, Worst, Average)
Bubble Sort
• The simplest sorting algorithm that works by repeatedly swapping the adjacent
elements if they are in the wrong order.

• Time Complexity: θ(n2) (in all three cases: Best, Worst, Average)
Comparison Sorting Review

• Merge sort:
• Divide-and-conquer:
• Split array in half
• Recursively sort sub-arrays
• Linear-time merge step
• Pro’s:
• O(n lg n) worst case - asymptotically optimal for
comparison sorts
• Con’s:
• Doesn’t sort in place
Comparison Sorting Review

• Heap sort:
• Uses the very useful heap data structure
• Complete binary tree
• Heap property: parent key > children’s keys
• Pro’s:
• O(n lg n) worst case - asymptotically optimal for
comparison sorts
• Sorts in place
• Con’s:
• Fair amount of shuffling memory around
Comparison Sorting Review

• Quick sort:
• Divide-and-conquer:
• Partition array into two sub-arrays, recursively sort
• All of first sub-array < all of second sub-array
• Pro’s:
• O(n lg n) average case
• Sorts in place
• Fast in practice (why?)
• Con’s:
• O(n2) worst case
• Naïve implementation: worst case on sorted input
• Good partitioning makes this very unlikely.
Sorting in Linear Time

7
Non-Comparison Based Sorting

• Many times we have restrictions on our keys

• Deck of cards: Ace->King and four suites
• Social Security Numbers
• Employee ID’s
• We will examine three algorithms which under certain conditions
can run in O(n) time.
• Counting sort
• Radix sort
• Bucket sort
 Sorting in linear time

Counting sort: No comparisons between elements.

• Input: A[1 . . n], where A[ j]{1, 2, …, k} .
• Output: B[1 . . n], sorted.
• Auxiliary storage: C[1 . . k] .
Counting sort

for i  1 to k
do C[i]  0
for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
for i  2 to k
do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key  i}|
for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 Counting-sort example

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C:

B:
 Loop 1

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 0 0 0 0

B:

for i  1 to k
do C[i]  0
 Loop 2

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 0 0 0 1

B:

for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
 Loop 2

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 0 1

B:

for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
 Loop 2

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 1 1

B:

for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
 Loop 2

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 1 2

B:

for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
 Loop 2

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 2 2

B:

for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
 Loop 3

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 2 2

B: C': 1 1 2 2

for i  2 to k
do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key  i}|
 Loop 3

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 2 2

B: C': 1 1 3 2

for i  2 to k
do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key  i}|
 Loop 3

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 0 2 2

B: C': 1 1 3 5

for i  2 to k
do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key  i}|
 Loop 4

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 1 3 5

B: 3 C': 1 1 2 5

for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 Loop 4

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 1 2 5

B: 3 4 C': 1 1 2 4

for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 Loop 4

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 1 2 4

B: 3 3 4 C': 1 1 1 4

for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 Loop 4

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 1 1 1 4

B: 1 3 3 4 C': 0 1 1 4

for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
 Loop 4

1 2 3 4 5 1 2 3 4

A: 4 1 3 4 3 C: 0 1 1 4

B: 1 3 3 4 4 C': 0 1 1 3

for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
Analysis
for i  1 to k
(k) do C[i]  0
for j  1 to n
(n) do C[A[ j]]  C[A[ j]] + 1
for i  2 to k
(k) do C[i]  C[i] + C[i–1]
for j  n downto 1
(n) do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
(n + k)
Running time

If k = O(n), then counting sort takes (n) time.

•But, sorting takes (n lg n) time!
•Where’s the fallacy?
Answer:
•Comparison sorting takes (n lg n) time.
•Counting sort is not a comparison sort.
•In fact, not a single comparison between
elements occurs!
Stable sorting

Counting sort is a stable sort: it preserves

the input order among equal elements.

A: 4 1 3 4 3

B: 1 3 3 4 4

Exercise: What other sorts have this

property?
Counting Sort

• Total time: O(n + k)

• Works well if k = O(n) or k = O(1)
• This sorting is stable.
• A sorting algorithm is stable when numbers with the same values
appear in the output array in the same order as they do in the input
array.
Counting Sort

• Why don’t we always use counting sort?

• Depends on range k of elements.

• Could we use counting sort to sort 32 bit integers? Why or why not?
Counting Sort Review

• Assumption: input taken from small set of numbers of size k

• Basic idea:
• Count number of elements less than you for each element.
• This gives the position of that number – similar to selection sort.
• Pro’s:
• Fast
• Asymptotically fast - O(n+k)
• Simple to code
• Con’s:
• Doesn’t sort in place.
• Elements must be integers.
• Requires O(n+k) extra storage.
countable
Radix Sort

• How did IBM get rich originally?

• Answer: punched card readers for census tabulation in early 1900’s.
• In particular, a card sorter that could sort cards into different bins
• Each column can be punched in 12 places
• Decimal digits use 10 places
• Problem: only one column can be sorted on at a time
Radix Sort

• Intuitively, you might sort on the most significant digit, then the second
msd, etc.
• Problem: lots of intermediate piles of cards (read: scratch arrays) to
keep track of
• Key idea: sort the least significant digit first
RadixSort(A, d)
for i=1 to d
StableSort(A) on digit i
Radix Sort Example
Radix Sort Correctness

• Sketch of an inductive proof of correctness (induction on the number of

passes):
• Assume lower-order digits {j: j<i }are sorted
• Show that sorting next digit i leaves array correctly sorted
• If two digits at position i are different, ordering numbers by that
digit is correct (lower-order digits irrelevant)
• If they are the same, numbers are already sorted on the lower-
order digits. Since we use a stable sort, the numbers stay in the
right order
Radix Sort

• What sort is used to sort on digits?

• Counting sort is obvious choice:
• Sort n numbers on digits that range from 1..k
• Time: O(n + k)
• Each pass over n numbers with d digits takes time O(n+k), so
total time O(dn+dk)
• When d is constant and k=O(n), takes O(n) time
Radix Sort

• Problem: sort 1 million 64-bit numbers

• Treat as four-digit radix 216 numbers
• Can sort in just four passes with radix sort!
• Performs well compared to typical
O(n lg n) comparison sort
• Approx lg(1,000,000)  20 comparisons per number being
sorted
Radix Sort Review

• Assumption: input has d digits ranging from 0 to k

• Basic idea:
• Sort elements by digit starting with least significant
• Use a stable sort (like counting sort) for each stage
• Pro’s:
• Fast
• Asymptotically fast (i.e., O(n) when d is constant and k=O(n))
• Simple to code
• A good choice
• Con’s:
• Doesn’t sort in place
• Not a good choice for floating point numbers or arbitrary strings.
 Bucket Sort

Assumption: input elements distributed uniformly over some

known range, e.g., [0,1), so all elements in A are greater than
or equal to 0 but less than 1 . (Appendix C.2 has definition of
uniform distribution)

Bucket-Sort(A)
1. n = length[A]
2. for i = 1 to n
3. do insert A[i] into list B[floor of nA[i]]
4. for i = 0 to n-1
5. do sort list i with Insertion-Sort
6. Concatenate lists B[0], B[1],…,B[n-1]
39
Bucket Sort
Bucket-Sort(A, x, y)
1. divide interval [x, y) into n equal-sized subintervals (buckets)
2. distribute the n input keys into the buckets
3. sort the numbers in each bucket (e.g., with insertion sort)
4. scan the (sorted) buckets in order and produce output array
Running time of bucket sort: O(n) expected time
Step 1: O(1) for each interval = O(n) time total.
Step 2: O(n) time.
Step 3: The expected number of elements in each bucket is O(1),
so total is O(n)
Step 4: O(n) time to scan the n buckets containing a total of n
input elements
40
Bucket Sort Example
Bucket Sort Review

• Assumption: input is uniformly distributed across a range

• Basic idea:
• Partition the range into a fixed number of buckets.
• Toss each element into its appropriate bucket.
• Sort each bucket.
• Pro’s:
• Fast
• Asymptotically fast (i.e., O(n) when distribution is uniform)
• Simple to code
• Good for a rough sort.
• Con’s:
• Doesn’t sort in place
Summary of Linear Sorting
Non-Comparison Based Sorts
Running Time
worst-case average-case best-case in place
Counting Sort O(n + k) O(n + k) O(n + k) no
Radix Sort O(d(n + k')) O(d(n + k')) O(d(n + k')) no
Bucket Sort O(n) no
Counting sort assumes input elements are in range [0,1,2,..,k] and uses array
indexing to count the number of occurrences of each value.
Radix sort assumes each integer consists of d digits, and each digit is in range
[1,2,..,k'].
Bucket sort requires advance knowledge of input distribution (sorts n
numbers uniformly distributed in range in O(n) time).

43