7/03/2025

Faculty of Information Technology,

Monash University

FIT2004: Algorithms and Data

Structures

Week 2:
Analysis of Algorithms

1
 7/03/2025

Outline

 Proving correctness of algorithms

 Comparison-based sorting algorithms
 Selection sort
 Lower bound for comparison-based sorting
 Non-comparison-based sorting algorithms
 Counting Sort
 Radix Sort

FIT2004: Lecture 2 - Analysis of Algorithms

Algorithmic Analysis

 In algorithmic analysis, one is interested in (at least) two

things:
 The correctness of the algorithm.
 The amount of resources used by the algorithm.

 Let’s look at how to prove correctness of an algorithm.

FIT2004: Lecture 2 - Analysis of Algorithms

2
 7/03/2025

Proving Correctness of Algorithms

 Usually, we write programs and then test them.
 Testing detects inputs for which the program produces incorrect
output.
 Can we test all possible inputs?
 There are infinitely many possible inputs (for most programs) so
we cannot test them all, therefore…
 Testing cannot guarantee that the program is always correct!
 Then, what is the solution?
 Logic can prove that a program is always correct.
 This is usually achieved in two parts by showing that:
1. the program always terminates;
2. It produces correct results when it terminates.

FIT2004: Lecture 2 - Analysis of Algorithms

Example: Finding Minimum Value

//Find minimum value in an unsorted array of N>0 elements

min = array[1] // index starts at 1

index = 2

while index <= N

if array[index] < min

min = array[index]
index = index + 1

return min

FIT2004: Lecture 2 - Analysis of Algorithms

3
 7/03/2025

Does it Always Terminate?

//Find minimum value in an unsorted array of N>0 elements

min = array[1]
index = 2

while index <= N

if array[index] < min

min = array[index]
index = index + 1

return min

FIT2004: Lecture 2 - Analysis of Algorithms

Correct Result at Termination?

//Find minimum value in an unsorted array of N>0 elements

min = array[1]
index = 2
Will we get a correct
result if we replace
index <= N with
while index <= N
index < N?

if array[index] < min

min = array[index]
index = index + 1

return min

FIT2004: Lecture 2 - Analysis of Algorithms

4
 7/03/2025

Correctness using Loop Invariant

 A loop invariant is used to reason about the correctness
and behavior of an algorithm.

 It is a condition or property that holds true before and

after each iteration of a loop, throughout the execution of
that loop.

 In other words, it is a logical assertion about the state of

the program that remains consistent as the loop
progresses, even though the loop's variables might
change.

FIT2004: Lecture 2 - Analysis of Algorithms

Correctness using Loop Invariant (LI)

Invariant should be true at three points:
1. Before the loop starts (Initialisation)
2. During each loop (Maintenance)
3. After the loop terminates (Termination)

FIT2004: Lecture 2 - Analysis of Algorithms

5
 7/03/2025

Correctness using LI- Initialisation

min = array[1]
index = 2
//LI: min equals the minimum value in array[1 … index - 1]
while index <= N
if array[index] < min
min = array[index]
index = index + 1
return min

• Consider the invariant just before the loop condition is checked.

min = array[1]
index = 2
• Here, min = the minimum value in array[1 … index - 1], i.e. min = the
minimum value in array from array[1] to array[1] (i.e. the 1st element).

• The condition holds because min is initialized to array[1], and obviously

the "minimum" value of just one element is that element itself.
FIT2004: Lecture 2 - Analysis of Algorithms

Correctness using LI- Maintenance

min = array[1]
index = 2
//LI: min equals the minimum value in array[1 … index - 1]
while index <= N
if array[index] < min
min = array[index]
index = index + 1
return min

• At the start of each iteration, min equals the minimum value in

array[1 … index - 1].
• This is maintained because, the min variable holds the smallest value
encountered up to index – 1, which starts with array[1].
• If a smaller value is found during the current iteration, the min is
updated to this new smaller value, array[index].
• After the update, min still correctly represents the smallest value in the
array from array[1] to array[index - 1].

FIT2004: Lecture 2 - Analysis of Algorithms

6
 7/03/2025

Correctness using LI- Termination

min = array[1]
index = 2
//LI: min equals the minimum value in array[1 … index - 1]
while index <= N
if array[index] < min
min = array[index]
index = index + 1
return min

• We saw that LI is true for all values of index (in the previous step).
• We have set index to the value which causes termination (N+1).
• Check if the statement of LI with this index value is what we want
//LI: min equals the minimum value in array[1 … N]
• After the last iteration, min will hold the minimum value in the entire
array from array[1] to array[N].
• Thus, the final value of min after the loop ends will be the minimum
value in the range array[1 … N], which is the desired outcome.

FIT2004: Lecture 2 - Analysis of Algorithms

Another Example: Binary Search

5 10 15 20 25 30 35 40 45 50

1 2 3 4 5 6 7 8 9 10 11

lo mid mid hi
mid

mid = floor( (lo+hi)/2 )

Binary search 20 in a sorted array
 Since 20 < array[mid],
 Search from lo to mid (e.g., move hi to mid)
 Since 20 > array[mid]
 Search from mid to hi (e.g., move lo to mid)

 …

FIT2004: Lec-2: Analysis of Algorithms 14

7
 7/03/2025

Algorithm for Binary Search

lo = 1//we are assuming indices range is 1 to N inclusive
hi = N + 1

while ( lo < hi )
mid = floor( (lo+hi)/2 )
if key >= array[mid] Is this algorithm correct?
lo=mid To prove correctness, we need to
else show that
hi=mid
1. it always terminates, and
2. it returns correct result when it
if N > 0 and array[lo] == key
terminates
print(key found at index lo)
else
print(key not found)

FIT2004: Lec-2: Analysis of Algorithms 15

Does this algorithm always terminate?

lo = 1
hi = N+1

while ( lo < hi ) Let us try to fix this

mid = floor( (lo+hi)/2 )
if key >= array[mid]
lo=mid
else This algorithm may never terminate in
some cases
hi=mid mid

if N > 0 and array[lo] == key 5 10 15 20 25

print(key found at index lo)
1 2 3 4 5
else
print(key not found)
lo hi Key = 13

FIT2004: Lec-2: Analysis of Algorithms 16

8
 7/03/2025

Does it always terminate?

lo = 1//we are assuming indices range is 1 to N inclusive
hi = N + 1
Proof that it always terminates
 lo < hi – 1 implies that the difference between
while ( lo < hi - 1 ) lo and hi is always at least 2
mid = floor( (lo+hi)/2 )
 Therefore, lo < mid < hi.
if key >= array[mid]
 Hence, the search space always shrinks
lo=mid (e.g., lo and hi get closer after every iteration
else of the while loop until lo ≥ hi – 1 in which case
hi=mid the algorithm terminates)
mid

if N > 0 and array[lo] == key

print(key found at index lo) 5 10 15 20 25
else 1 2 3 4 5
print(key not found)

lo hi Key = 13

FIT2004: Lec-2: Analysis of Algorithms 17

Correctness using Loop Invariant

lo = 1
hi = N + 1
// LI: key in array[1 … N] if and only if (iff) key in array[lo … hi - 1]
while ( lo < hi - 1 )
// LI: key in array[1 … N] iff key in array[lo … hi-1]
mid = floor( (lo+hi)/2 )
if key >= array[mid]
// key in array[1 … N] iff key in array[mid … hi-1]
lo=mid
// LI: key in array[1 … N] iff key in array[lo … hi-1]
else
// key in array[1 … N] iff key in array[lo … mid-1]
hi=mid
// LI: key in array[1 … N] iff key in array[lo … hi-1]
// LI: key in array[1 … N] iff key in array[lo … hi-1]

FIT2004: Lec-2: Analysis of Algorithms 18

9
 7/03/2025

Correctness using Loop Invariant

// LI: key in array[1 … N] if and only if (iff) key in array[lo … hi - 1]
while ( lo < hi - 1 )
mid = floor( (lo+hi)/2 ) Note: lo < hi when loop terminates, because
if key >= array[mid]  lo < mid < hi in each iteration and
lo=mid  we update either lo to be mid or hi to be mid
else When lo = hi-2
hi=mid
// LI: key in array[1 … N] iff key in array[lo … hi-1]
1 2 3 4 5
// lo ≥ hi – 1 → lo + 1 ≥ hi ---- (A)
// lo < hi → lo + 1 ≤ hi ---- (B)
lo hi-1 hi
// From (A) and (B): lo + 1 = hi → lo = hi - 1
// Hence, key in array[1 … N] iff key in array[lo … lo]; (Proof Complete)
if N > 0 and array[lo] == key
print(key found at index lo)
else
print(key not found)
FIT2004: Lec-2: Analysis of Algorithms 19

Correctness using Loop Invariant

Invariant: If key is in L, it is in L[lo…hi-1]

BinarySearch(L[1..n], t)
lo = 1
Initialisation
hi = N+1
• If rightmost target is in L, then
while ( lo < hi - 1 ) target is in L[lo…hi-1] =
mid = (lo + hi) / 2 L[1…N] = L
if t ≥ L[mid]
lo = mid
if t < L[mid]
hi = mid
if L[lo] == t
return lo
return False

FIT2004: Lecture 2 - Analysis of Algorithms

10
 7/03/2025

Correctness using Loop Invariant

Invariant: If key is in L, it is in L[lo…hi-1] Maintenance
• Assume target in L[lo…hi-1]
before some loop iteration
BinarySearch(L[1..n], t)
lo = 1 • If t >= L[mid], then it must
hi = N+1 be located in the range
while ( lo < hi - 1 )
L[mid…hi-1], so we should
set lo to mid
mid = (lo + hi) / 2
if t ≥ L[mid]
• If t < L[mid], then it must be
lo = mid
in the range L[lo…mid-1], so
if t < L[mid] we should set hi to mid
hi = mid (since L[hi] is excluded)
if L[lo] == t
return lo • Since this is exactly what we do,
return False at the start of the next iteration
the invariant is still true
FIT2004: Lecture 2 - Analysis of Algorithms

Correctness using Loop Invariant

Invariant: If key is in L, it is in L[lo…hi-1] Termination
• lo == hi-1. Since the
invariant has been correctly
BinarySearch(L[1..n], t) maintained…
lo = 1
hi = N+1 • Rightmost target must be in
while ( lo < hi - 1 ) L[lo…hi-1], or it is not in the
mid = (lo + hi) / 2 list
if t ≥ L[mid]
• L[lo…hi-1] is a single
lo = mid
element, L[lo].
if t < L[mid]
hi = mid
• If the element is found at
if L[lo] == t L[lo], then return that index
return lo
return False • Otherwise return False

FIT2004: Lecture 2 - Analysis of Algorithms

11
 7/03/2025

Outline

 Proving correctness of algorithms

 Comparison-based sorting algorithms
 Selection sort
 Lower bound for comparison-based sorting

 Non-comparison sorting algorithms

 Counting Sort
 Radix Sort

FIT2004: Lecture 2 - Analysis of Algorithms

Comparison-based Sorting
 Comparison-based sorting algorithms sort the input array
by comparing the items with each other:
 Selection Sort
 Insertion Sort
 Quick Sort
 Merge Sort
 Heap Sort
 …

 The algorithms that do not require comparing items with

each other are called non-comparison sorting algorithms.
E.g., Counting sort, radix sort, bucket sort etc.

FIT2004: Lecture 2 - Analysis of Algorithms

12
 7/03/2025

Comparison Cost
 Typically, we assume that comparing two elements takes O(1), e.g.,
array[i] <= array[j]. This is not necessarily true.

 String Comparison: The worst-case cost of comparing two strings is O(L)

where L is the number of characters in the smaller string. E.g.,
 “Welcome to Faculty of IT” <= “Welcome to FIT2004” ??
 We compare strings character by character (from left to right) until the two
characters are different – all green letters are compared in above example

 Number Comparison: Generally we assume that numbers are machine-

word sized (i.e. fit in a register) and so comparison is O(1). In theory, for
very large numbers, comparison would be O(digits).

FIT2004: Lecture 2 - Analysis of Algorithms

Comparison Cost
 Typically assume comparison cost is O(1) (small values)
 Sometimes comparison cost is critical: genome sequences may have
millions of characters – expensive comparison.
 The cost of comparison-based sorting is often taken as in terms of # of
comparisons, e.g., # of comparisons in merge sort is O(N log N).
 This ignores comparison cost (which is mostly fine).

In summary:
In this unit we will generally assume that
 String comparison is O(c), where c is the number of characters which get
compared
 Numerical comparison is O(1) unless otherwise specified

FIT2004: Lecture 2 - Analysis of Algorithms

13
 7/03/2025

Stable Sorting Algorithms

 A sorting algorithm is called stable if it maintains the relative ordering of
elements that have equal keys.
Input is sorted by names
Marks 80 75 70 90 85 75
Input
Name Alice Bill Don Geoff Leo Maria

Sort on Marks using a stable algorithm

Output Marks 70 75 75 80 85 90
Name Don Bill Maria Alice Leo Geoff

Note: Output is sorted on marks then names.

Unstable sorting cannot guarantee this (e.g., Maria may appear before Bill)

FIT2004: Lecture 2 - Analysis of Algorithms

Outline

 Proving correctness of algorithms

 Comparison-based sorting algorithms
 Selection sort
 Lower bound for comparison-based sorting

 Non-comparison sorting algorithms

 Counting Sort
 Radix Sort

FIT2004: Lecture 2 - Analysis of Algorithms

14
 7/03/2025

Selection Sort (Correctness)

Sort an array (denoted as arr) in ascending order

for(i = 1; i < N; i++) {

j = index of minimum element in arr[i … N]
swap (arr[i],arr[j])
}

2 4 5 6 10 8 7 11

i
FIT2004: Lecture 2 - Analysis of Algorithms

Selection Sort (Correctness)

Sort an array (denoted as arr) in ascending order

for(i = 1; i < N; i++) {

j = index of minimum element in arr[i … N]
swap (arr[i],arr[j])
}

2 4 5 6 10 8 7 11

i j

FIT2004: Lecture 2 - Analysis of Algorithms

15
 7/03/2025

Selection Sort (Correctness)

Sort an array (denoted as arr) in ascending order

for(i = 1; i < N; i++) {

j = index of minimum element in arr[i … N]
swap (arr[i],arr[j])
}

2 4 5 6 7 8 10 11

i j

FIT2004: Lecture 2 - Analysis of Algorithms

Selection Sort (Correctness)

Sort an array (denoted as arr) in ascending order

// LI: arr[1 … i-1] is sorted AND arr[1 … i-1] <= arr[i … N]

for(i = 1; i < N; i++) {
j = index of minimum element in arr[i … N]
swap (arr[i],arr[j])
}
// i=N when the loop terminates
// LI: arr[1 … N-1] is sorted AND arr[1 … N-1] <= arr[N]

2 4 5 6 7 8 10 11

i j

FIT2004: Lecture 2 - Analysis of Algorithms

16
 7/03/2025

Selection Sort (Correctness)

Sort an array (denoted as arr) in ascending order

// LI: arr[1 … i-1] is sorted AND arr[1 … i-1] <= arr[i … N]

for(i = 1; i < N; i++) {
j = index of minimum element in arr[i … N]
swap (arr[i],arr[j])
}
// i=N when the loop terminates
// LI: arr[1 … N-1] is sorted AND arr[1 … N-1] <= arr[N]

Could we use a weaker loop invariant, e.g.,

// LI: arr[1 … i-1] is sorted (That is Insertion Sort)

FIT2004: Lecture 2 - Analysis of Algorithms

Selection Sort (Correctness)

Could we use a weaker loop invariant, e.g.,
// LI: arr[1 … i-1] is sorted (That is Insertion Sort)

While doing the proof, we need to show that if the LI is true at iteration k, it is true
at iteration k+1

j = index of minimum element in arr[i … N]

swap (arr[i],arr[j])

2 4 5 8 10 12 7 15 13
i-1 i
sorted unsorted

FIT2004: Lecture 2 - Analysis of Algorithms

17
 7/03/2025

Selection Sort (Correctness)

Could we use a weaker loop invariant, e.g.,
// LI: arr[1 … i-1] is sorted (That is Insertion Sort)

While doing the proof, we need to show that if the LI is true at iteration k, it is true
at iteration k+1

j = index of minimum element in arr[i … N]

swap (arr[i],arr[j])

2 4 5 8 10 12 7 15 13
i-1 i
sorted unsorted
The invariant currently holds

FIT2004: Lecture 2 - Analysis of Algorithms

Selection Sort (Correctness)

Could we use a weaker loop invariant, e.g.,
// LI: arr[1 … i-1] is sorted (That is Insertion Sort)

While doing the proof, we need to show that if the LI is true at iteration k, it is true
at iteration k+1

j = index of minimum element in arr[i … N]

swap (arr[i],arr[j])

j
2 4 5 8 10 12 7 15 13
i-1 i
sorted unsorted
The invariant currently holds

FIT2004: Lecture 2 - Analysis of Algorithms

18
 7/03/2025

Selection Sort (Correctness)

Could we use a weaker loop invariant, e.g.,
// LI: arr[1 … i-1] is sorted (That is Insertion Sort)

While doing the proof, we need to show that if the LI is true at iteration k, it is true
at iteration k+1

j = index of minimum element in arr[i … N]

swap (arr[i],arr[j])

j
2 4 5 8 7 12 10 15 13
i-1 i
sorted unsorted
The invariant currently holds

FIT2004: Lecture 2 - Analysis of Algorithms

Selection Sort (Correctness)

Could we use a weaker loop invariant, e.g.,
// LI: arr[1 … i-1] is sorted (That is Insertion Sort)

While doing the proof, we need to show that if the LI is true at iteration k, it is true
at iteration k+1

j = index of minimum element in arr[i … N]

swap (arr[i],arr[j])

j
2 4 5 8 7 12 10 15 13
i-1 i
sorted unsorted
The invariant should still be true… but it is not

FIT2004: Lecture 2 - Analysis of Algorithms

19
 7/03/2025

Selection Sort Analysis

for(i = 1; i < N; i++) {
j = index of minimum element in arr[i … N]
swap (arr[i],arr[j])
}
Time Complexity?
 Worst-case 2 4 5 6 10 10 7 11
 Complexity of finding minimum element at i-th iteration?
 Total complexity? O(N2)
i j
 Best-case

Input array: O(N)

Space Complexity? Space for swapping operation: O(1)

Auxiliary Space Complexity?

o Selection Sort is an in-place algorithm!

Is selection sort stable?

FIT2004: Lecture 2 - Analysis of Algorithms

Summary of Comparison-based Sorting Algorithms

Best Worst Average Stable? In-

place?
Selection Sort O(N2) O(N2) O(N2) No Yes

Insertion Sort O(N) O(N2) O(N2) Yes Yes

Heap Sort O(N) O(N log N) O(N log N) No Yes

Merge Sort O(N log N) O(N log N) O(N log N) Yes No

• Is it possible to develop a sorting algorithm with worst-case time

complexity better than O(N log N)?

FIT2004: Lecture 2 - Analysis of Algorithms

20
 7/03/2025

Outline

 Proving correctness of algorithms

 Comparison-based sorting algorithms
 Selection sort
 Lower bound for comparison-based sorting

 Non-comparison sorting algorithms

 Counting Sort
 Radix Sort

FIT2004: Lecture 2 - Analysis of Algorithms

Lower Bound Complexity

 Lower bound complexity for a problem is the lowest possible complexity any
algorithm (known or unknown) can achieve to solve the problem.
o It is important because it gives a theoretical bound on what is best possible.
o Unless stated otherwise, lower bound is for the worst-case complexity of the
algorithm.

 What is the lower bound complexity of finding the minimum element in an array
of N elements?
o Ans: Ω(N), i.e. Omega(N): We (at least) need to look at every element.

 Since the finding minimum algorithm we saw has O(N) worst-case complexity, it
is the best possible algorithm (i.e., optimal) in terms of time complexity.

 Generally, notation O(N) (Big O) is used for Upper bound (indicating worst case
or at most this time). Ω(N) (Big Omega) is used for Lower bound (best case or at
least this time).

FIT2004: Lecture 2 - Analysis of Algorithms

21
 7/03/2025

Lower Bound Complexity

 Lower bound complexity for a problem is the lowest possible complexity any
algorithm (known or unknown) can achieve to solve the problem.
 It is important because it gives a theoretical bound on what is best possible.
 Unless stated otherwise, lower bound is for the worst-case complexity of the
algorithm.

 What is the lower bound complexity for sorting?

 For comparison-based algorithm, lower bound complexity is Ω(N log N).
 Read the lecture notes to see why the lower bound is Ω(N log N) -- the
proof is not examinable.

 Next, we discuss two non-comparison sorting algorithms that do sorting in less

than O(N log N).

FIT2004: Lecture 2 - Analysis of Algorithms

Outline

 Proving correctness of algorithms

 Comparison-based sorting algorithms
 Selection sort
 Lower bound for comparison-based sorting

 Non-comparison sorting algorithms

 Counting Sort
 Radix Sort

FIT2004: Lecture 2 - Analysis of Algorithms

22
 7/03/2025

Intuition
 Suppose you are given an array containing a permutation of the
numbers 1…N to sort.

 Or given an array containing a subset of the numbers 1…N, sort them.

 Assume you are asked to enter a number between 1 to 200 (inclusive)

on Flux. The rules of the game are as follows:
 The person entering the smallest unique number wins
 If two people enter the same number, both are disqualified
 The winner is the person who enters the smallest number among the not-disqualified set

 Can you write an algorithm to determine the winner?

 What is the lower bound complexity for this problem?

FIT2004: Lecture 2 - Analysis of Algorithms

Counting Sort
 For example, consider the problem of sorting positive integers.
 Determine the maximum value in the input.
 Create an array “count” of that size.

count 1 0
2 0
3 0
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 0
6 0
7 0
8 0

FIT2004: Lecture 2 - Analysis of Algorithms

23
 7/03/2025

Counting Sort
 Iterate through the input and count the number of times each value
occurs.
 Do this by incrementing the corresponding position in “count”.
 If we see a 3, increment 𝑐𝑜𝑢𝑛𝑡[3].

count 1 0
2 0
3 0
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 0
6 0
7 0
8 0

FIT2004: Lecture 2 - Analysis of Algorithms

Counting Sort
 Iterate through the input and count the number of times each value
occurs.
 Do this by incrementing the corresponding position in “count”.
 If we see a 3, increment 𝑐𝑜𝑢𝑛𝑡[3].

count 1 0
2 0
3 1
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 0
6 0
7 0
8 0

FIT2004: Lecture 2 - Analysis of Algorithms

24
 7/03/2025

Counting Sort
 Iterate through the input and count the number of times each value
occurs.
 Do this by incrementing the corresponding position in “count”.
 If we see a 3, increment 𝑐𝑜𝑢𝑛𝑡[3].

count 1 1
2 0
3 1
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 0
6 0
7 0
8 0

FIT2004: Lecture 2 - Analysis of Algorithms

Counting Sort
 Iterate through the input and count the number of times each value
occurs.
 Do this by incrementing the corresponding position in “count”.
 If we see a 3, increment 𝑐𝑜𝑢𝑛𝑡[3].

count 1 1
2 0
3 2
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 0
6 0
7 0
8 0

FIT2004: Lecture 2 - Analysis of Algorithms

25
 7/03/2025

Counting Sort
 Iterate through the input and count the number of times each value
occurs.
 Do this by incrementing the corresponding position in “count”.
 If we see a 3, increment 𝑐𝑜𝑢𝑛𝑡[3].

count 1 1
2 0
3 2
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 0
6 0
7 1
8 0

FIT2004: Lecture 2 - Analysis of Algorithms

Counting Sort
 Iterate through the input and count the number of times each value
occurs.
 Do this by incrementing the corresponding position in “count”.
 If we see a 3, increment 𝑐𝑜𝑢𝑛𝑡[3].

count 1 1
2 0
3 2
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 1
8 0

FIT2004: Lecture 2 - Analysis of Algorithms

26
 7/03/2025

Counting Sort
 Iterate through the input and count the number of times each value
occurs.
 Do this by incrementing the corresponding position in “count”.
 If we see a 3, increment 𝑐𝑜𝑢𝑛𝑡[3].

count 1 1
2 0
3 3
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 1
8 0

FIT2004: Lecture 2 - Analysis of Algorithms

Counting Sort
 Iterate through the input and count the number of times each value
occurs.
 Do this by incrementing the corresponding position in “count”.
 If we see a 3, increment 𝑐𝑜𝑢𝑛𝑡[3].

count 1 1
2 0
3 3
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 2
8 0

FIT2004: Lecture 2 - Analysis of Algorithms

27
 7/03/2025

Counting Sort
 Iterate through the input and count the number of times each value
occurs.
 Do this by incrementing the corresponding position in “count”.
 If we see a 3, increment 𝑐𝑜𝑢𝑛𝑡[3].

count 1 1
2 0
3 3
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 2
8 1

FIT2004: Lecture 2 - Analysis of Algorithms

Counting Sort
 Create “output”.
 For each position in count:
o Append 𝑐𝑜𝑢𝑛𝑡[𝑖] copies of the value 𝑖 to output.
o So if 𝑐𝑜𝑢𝑛𝑡[7] = 2, then append 2 copies of 7.

count 1 1
2 0
3 3
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 2
8 1
Output

FIT2004: Lecture 2 - Analysis of Algorithms

28
 7/03/2025

Counting Sort
 Create “output”.
 For each position in count:
o Append 𝑐𝑜𝑢𝑛𝑡[𝑖] copies of the value 𝑖 to output.
o So if 𝑐𝑜𝑢𝑛𝑡[7] = 2, then append 2 copies of 7.

count 1 1
2 0
3 3
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 2
8 1
Output 1

FIT2004: Lecture 2 - Analysis of Algorithms

Counting Sort
 Create “output”.
 For each position in count:
o Append 𝑐𝑜𝑢𝑛𝑡[𝑖] copies of the value 𝑖 to output.
o So if 𝑐𝑜𝑢𝑛𝑡[7] = 2, then append 2 copies of 7.

count 1 1
2 0
3 3
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 2
8 1
Output 1

FIT2004: Lecture 2 - Analysis of Algorithms

29
 7/03/2025

Counting Sort
 Create “output”.
 For each position in count:
o Append 𝑐𝑜𝑢𝑛𝑡[𝑖] copies of the value 𝑖 to output.
o So if 𝑐𝑜𝑢𝑛𝑡[7] = 2, then append 2 copies of 7.

count 1 1
2 0
3 3
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 2
8 1
Output 1 3 3 3

FIT2004: Lecture 2 - Analysis of Algorithms

Counting Sort
 Create “output”.
 For each position in count:
o Append 𝑐𝑜𝑢𝑛𝑡[𝑖] copies of the value 𝑖 to output.
o So if 𝑐𝑜𝑢𝑛𝑡[7] = 2, then append 2 copies of 7.

count 1 1
2 0
3 3
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 2
8 1
Output 1 3 3 3

FIT2004: Lecture 2 - Analysis of Algorithms

30
 7/03/2025

Counting Sort
 Create “output”.
 For each position in count:
o Append 𝑐𝑜𝑢𝑛𝑡[𝑖] copies of the value 𝑖 to output.
o So if 𝑐𝑜𝑢𝑛𝑡[7] = 2, then append 2 copies of 7.

count 1 1
2 0
3 3
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 2
8 1
Output 1 3 3 3 5

FIT2004: Lecture 2 - Analysis of Algorithms

Counting Sort
 Create “output”.
 For each position in count:
o Append 𝑐𝑜𝑢𝑛𝑡[𝑖] copies of the value 𝑖 to output.
o So if 𝑐𝑜𝑢𝑛𝑡[7] = 2, then append 2 copies of 7.

count 1 1
2 0
3 3
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 2
8 1
Output 1 3 3 3 5

FIT2004: Lecture 2 - Analysis of Algorithms

31
 7/03/2025

Counting Sort
 Create “output”.
 For each position in count:
o Append 𝑐𝑜𝑢𝑛𝑡[𝑖] copies of the value 𝑖 to output.
o So if 𝑐𝑜𝑢𝑛𝑡[7] = 2, then append 2 copies of 7.

count 1 1
2 0
3 3
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 2
8 1
Output 1 3 3 3 5 7 7

FIT2004: Lecture 2 - Analysis of Algorithms

Counting Sort
 Create “output”.
 For each position in count:
o Append 𝑐𝑜𝑢𝑛𝑡[𝑖] copies of the value 𝑖 to output.
o So if 𝑐𝑜𝑢𝑛𝑡[7] = 2, then append 2 copies of 7.

count 1 1
2 0
3 3
4 0
Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w) 5 1
6 0
7 2
8 1
Output 1 3 3 3 5 7 7 8

FIT2004: Lecture 2 - Analysis of Algorithms

32
 7/03/2025

Analysis of Counting Sort

 Create “count” array of size max, where max is the maximum value in the input.

 For each value in “Input”:

o count[value] += 1

 Output = empty
 For x=1 to len(count):
o NumOfOccurrences = count[x]
o Append x to Output NumOfOccurrences times

Let N be the size of Input array and U be the domain size (e.g., max), i.e., U is the size of
count array.

Time Complexity:
 O(N+U) – worst-case, best-case, average-case all are the same.

Space Complexity:
 O(N+U)

FIT2004: Lecture 2 - Analysis of Algorithms

Analysis of Counting Sort

 Create “count” array of size max, where max is the maximum value in the input.

 For each value in “Input”:

o count[value] += 1

 Output = empty
 For x=1 to len(count):
o NumOfOccurrences = count[x]
o Append x to Output NumOfOccurrences times

Is counting sort stable?

No, because it counts the values but does not distinguishes between them. In fact, it loses
any data associated with the values, so it is much worse than unstable. Let’s fix this!

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Output 1 3 3 3 5 7 7 8

FIT2004: Lecture 2 - Analysis of Algorithms

33
 7/03/2025

Stable Counting Sort

 To fix the two problems of stability and losing data, we need a smart
idea.

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Output (1,p) (3,a) (3,c) (3,b) (5,g) (7,f) (7,d) (8,w)

 What information would you need to correctly place the data with key “5”
in the output?

FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
• For each key in input,
count
• count[key] += 1 1 1
2 0
3 3
4 0
5 1
6 0
7 2
8 1

FIT2004: Lecture 2 - Analysis of Algorithms

34
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
• For each key in input,
count position
• count[key] += 1 1 1 1 1
2 0 2 0
Construct position:
3 3 3 0
• Initialise first position as a 1
4 0 4 0
5 1 5 0
6 0 6 0
7 2 7 0
8 1 8 0

FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
• For each key in input,
count position
• count[key] += 1 1 1 1 1
2 0 2 2
Construct position:
3 3 3 0
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 0
5 1 5 0
6 0 6 0
7 2 7 0
8 1 8 0

FIT2004: Lecture 2 - Analysis of Algorithms

35
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
• For each key in input,
count position
• count[key] += 1 1 1 1 1
2 0 2 2
Construct position:
3 3 3 2
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 0
5 1 5 0
6 0 6 0
7 2 7 0
8 1 8 0

FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
• For each key in input,
count position
• count[key] += 1 1 1 1 1
2 0 2 2
Construct position:
3 3 3 2
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 0
6 0 6 0
7 2 7 0
8 1 8 0

FIT2004: Lecture 2 - Analysis of Algorithms

36
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
• For each key in input,
count position
• count[key] += 1 1 1 1 1
2 0 2 2
Construct position:
3 3 3 2
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
6 0 6 0
7 2 7 0
8 1 8 0

FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
• For each key in input,
count position
• count[key] += 1 1 1 1 1
2 0 2 2
Construct position:
3 3 3 2
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
6 0 6 6
7 2 7 0
8 1 8 0

FIT2004: Lecture 2 - Analysis of Algorithms

37
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
• For each key in input,
count position
• count[key] += 1 1 1 1 1
2 0 2 2
Construct position:
3 3 3 2
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
6 0 6 6
7 2 7 6
8 1 8 0

FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
• For each key in input,
count position
• count[key] += 1 1 1 1 1
2 0 2 2
Construct position:
3 3 3 2
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
6 0 6 6
7 2 7 6
8 1 8 8

FIT2004: Lecture 2 - Analysis of Algorithms

38
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 1
2 0 2 2
Construct position: 3 3 3 2
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) pair 7 2 7 6
from input 8 1 8 8
• Increment position[key]
Output
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 1
2 0 2 2
Construct position: 3 3 3 2
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 6
pair from input 8 1 8 8
• Increment position[key]
Output (3,a)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

39
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 1
2 0 2 2
Construct position: 3 3 3 3
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 6
pair from input 8 1 8 8
• Increment position[key]
Output (3,a)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 1
2 0 2 2
Construct position: 3 3 3 3
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 6
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

40
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 3
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 6
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 3
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 6
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a) (3,c)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

41
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 4
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 6
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a) (3,c)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 4
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 6
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a) (3,c) (7,f)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

42
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 4
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 7
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a) (3,c) (7,f)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 4
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 5
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 7
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a) (3,c) (5,g) (7,f)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

43
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 4
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 6
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 7
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a) (3,c) (5,g) (7,f)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 4
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 6
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 7
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a) (3,c) (3,b) (5,g) (7,f)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

44
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 5
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 6
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 7
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a) (3,c) (3,b) (5,g) (7,f)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 5
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 6
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 7
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a) (3,c) (3,b) (5,g) (7,f) (7,d)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

45
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 5
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 6
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 8
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a) (3,c) (3,b) (5,g) (7,f) (7,d)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 5
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 6
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 8
pair from input 8 1 8 8
• Increment position[key]
Output (1,p) (3,a) (3,c) (3,b) (5,g) (7,f) (7,d) (8,w)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

46
 7/03/2025

Stable Counting Sort

Input (3,a) (1,p) (3,c) (7,f) (5,g) (3,b) (7,d) (8,w)

Construct count:
count position
• For each key in input,
• count[key] += 1 1 1 1 2
2 0 2 2
Construct position: 3 3 3 5
• Initialise first position as a 1
• position[i] = position[i-1] + count[i-1] 4 0 4 5
5 1 5 6
Construct output
6 0 6 6
• Go through input, looking at each (key, val)
• Set output[position[key]] to the (key, val) 7 2 7 8
pair from input 8 1 8 9
• Increment position[key]
Output (1,p) (3,a) (3,c) (3,b) (5,g) (7,f) (7,d) (8,w)
1 2 3 4 5 6 7 8
FIT2004: Lecture 2 - Analysis of Algorithms

Analysis of Stable Counting Sort

 We have made count sort:
o Stable
o Able to keep associated data

 Time and space complexity?

o We have not changed the complexity (Though one is faster in practice).
o Time Complexity is O(N+U)
o Can it be O(N)?
o Yes, when U is O(N) (very efficient)

 Counting sort can also be very inefficient for certain kinds of input, any
guesses?
o Example: N = 2, U = 9223372036854775807
o Input: [0, 9223372036854775807]

 Using counting sort, can we build a fast way to sort numbers?

FIT2004: Lecture 2 - Analysis of Algorithms

47
 7/03/2025

Outline

 Proving correctness of algorithms

 Comparison-based sorting algorithms
 Selection sort
 Lower bound for comparison-based sorting

 Non-comparison sorting algorithms

 Counting Sort
 Radix Sort

FIT2004: Lecture 2 - Analysis of Algorithms

Radix Sort
Sort an array of numbers, assuming each number has k digits (why is this often reasonable?)
▪ Use stable sort to sort them on the k-th digit
▪ Use stable sort to sort them on the (k-1)-th digit
▪ …
▪ Use stable sort to sort them on 1st digit

7555 1191 3512 1182 1182

4642 4642 5412 1191 1191
3512 3512 1323 6284 1323
1323 6682 9523 1323 3512
3784 5412 4642 9356 3784
6284 1182 7555 5412 4642
6682 1323 9356 3512 5412
Sort on Sort on Sort on Sort on
9 5 2 3 4th digit 9 5 2 3 3rd digit 6 6 8 2 2nd digit 9 5 2 3 1st digit 6 2 8 4
1191 378 4 1182 755 5 6682
9356 628 4 3784 464 2 7555
5412 755 5 6284 668 2 9356
1182 935 6 1191 378 4 9523

FIT2004: Lecture 2 - Analysis of Algorithms

48
 7/03/2025

Radix Sort
What happens if we don’t use stable sort?

7555 1191 3512 1191 1323

4642 1182 5412 1182 1191
3512 3512 1323 6284 1182
1323 4642 9523 9356 3784
3784 5412 4642 1323 3512
6284 6682 9356 5412 4642
6682 1323 7555 755 5 5412
Sort on Sort on Sort on Sort on
9 5 2 3 4th digit 9 5 2 3 3rd digit 3 7 8 4 2nd digit 9 5 2 3 1st digit 6 6 8 2
1 191 3784 6284 351 2 6284
9 356 6284 1182 668 2 7555
5 412 7555 6682 464 2 9523
1182 9356 1191 3784 9356

FIT2004: Lecture 2 - Analysis of Algorithms

Radix Sort
What happens if we process left to right (or most significant
digit to least?)

7555 1323 1191 5412 1191

4642 1191 1182 3512 5412
3512 1182 6284 1323 3512
1323 3512 1323 9523 4642
3784 3784 9356 4642 1182
6284 4642 5412 9356 6682
6682 5412 3512 7555 1323
Sort on Sort on Sort on Sort on
9 5 2 3 1st digit 6 2 8 4 2nd digit 7 5 5 5 3rd digit 1 1 8 2 4th digit 9 5 2 3
1191 668 2 9523 628 4 6284
9356 755 5 4642 668 2 3784
5412 952 3 6682 378 4 7555
1182 935 6 3784 119 1 9356

FIT2004: Lecture 2 - Analysis of Algorithms

49
 7/03/2025

Analysis of Radix Sort

Sort an array of numbers, assuming each number has k digits
• Use stable sort to sort them on the k-th digit
• Use stable sort to sort them on the (k-1)-th digit
• …
• Use stable sort to sort them on 1st digit

Assume that N numbers to be sorted and each number has k digits

Assume we are using stable counting sort which has time and space
complexity 𝑂(𝑁 + 𝑈)

Time complexity of radix sort:

• Number of digits ∗ complexity of counting sort
• 𝑂(𝑘 ∗ (𝑁 + 𝑈)) = 𝑂(𝑘𝑁) because U is 10

Space complexity of radix sort:

• 𝑂(𝑘 ∗ (𝑁 + 𝑈)) = 𝑂(𝑘𝑁)

FIT2004: Lecture 2 - Analysis of Algorithms

Analysis of Radix Sort

But wait!
 Why base 10?

 Variable base, lets call the base “b”

 How many digits does a number have in base b?

 If the number has value 𝑀, then it has roughly log 𝑏 𝑀 digits

 The exact number of digits is log 𝑏 𝑀 + 1, but since we are doing
complexity analysis, we can just say this is O(log 𝑏 𝑀)

 So we would need O(log 𝑏 𝑀) count sorts to sort numbers of size M

FIT2004: Lecture 2 - Analysis of Algorithms

50
 7/03/2025

Analysis of Radix Sort

 So we would need O(log 𝑏 𝑀) count sorts to sort numbers of size 𝑀

 What would be the complexity of each counting sort?

 Each counting sort would take 𝑂(𝑁 + 𝑏)

 Total cost: O(log 𝑏 𝑀 ∗ N + b )

 As we increase 𝑏…
 Number of count sorts goes down
 Cost of each count sort goes up

FIT2004: Lecture 2 - Analysis of Algorithms

Analysis of Radix Sort

 Total cost: O(log 𝑏 𝑀 ∗ N + b )

We want to find a good balance …

 Notice that each count sort will be O(N) as long as the base b is O(N)
 As an example, pick b = N (i.e. the base is the number of elements in
the input)

 Total cost: O(log 𝑁 𝑀 ∗ N)

 What would be the value of M which makes the total cost O(N)?
 If M is O(Nc), then total cost: O(logN Nc * (N)) = O(CN) = O(N)!!!

 Of course, practical considerations also matter:

 Choosing a base which is a power of 2 is probably good
 Cache/localisation considerations…

FIT2004: Lecture 2 - Analysis of Algorithms

51
 7/03/2025

Sorting Strings
 How can we apply the count sort/radix sort idea to strings?

 Strings have “digits” which are letters

 The mapping can be done using ASCII:
 e.g., in python:
 ord(“A”) gives 65.
 ord(“B”) gives 66.
 and so on…

 The radix is now 26 (or however many characters we are using, e.g.
256)

FIT2004: Lecture 2 - Analysis of Algorithms

Radix Sort
Sort an array of words in alphabetical order assuming each word consists of M letters each
⚫ Use stable sort to sort them on the M-th column
⚫ Use stable sort to sort them on the (M-1)-th column
⚫…
⚫ Use stable sort to sort them on 1st column
GO A L HERB GOA L TAL L A I MS
TALL B I RD R I DE HERB ANT S
A I MS L I KE L I KE R I DE B I KE
ANTS B I KE B I KE L I KE B I RD
B I RD R I DE M I KE B I KE F I SH
F I SH M I KE TALL M I KE GO A L
L I KE K I NG K I NG K I NG Sort on HERV
Sort on Sort on Sort on 1st
B I KE 4th F I SH 3rd A I MS 2nd A I MS K I NG
column
column column column
K I NG GO A L HERB B I RD L I KE
R I DE TALL B I RD F I SH M I KE
HERB A I MS F I SH ANTS R I DE
M I KE ANTS ANTS GO A L TALL
FIT2004: Lec-2: Analysis of Algorithms 104

52
 7/03/2025

Algorithm’s Heroes
Betty Holberton (March 7, 1917 – December 8, 2001)

• An American mathematician and programmer

• One of the six original programmers of the first

electronic digital computer ENIAC (Electronic
Numerical Integrator and Calculator)

• She was the inventor of breakpoints in

computer debugging

• Once she said: By 1951 engineers had well

accepted the computer, but the business world was
still very sceptical about it, because they had such
questions as, "How do I know the data is on the
magnetic tape?" And "What is the legal implication of
data on a magnetic tape?"

FIT2004: Lecture 2 - Analysis of Algorithms

Reading
 Course Notes: Sections 1.1, 1.5, 3.1, 3.3 and 3.4

 You can also check algorithms’ textbooks for contents related to this
lecture, e.g.:
 CLRS: Section 2.1, Chapter 8
 Rou: Section 5.6

FIT2004: Lecture 2 - Analysis of Algorithms

53
 7/03/2025

Concluding Remarks
Summary
 Loop invariants can be used to prove correctness
 Stable sorting, in-place algorithms
 Non-comparison sorting

Coming Up Next
 Quick Sort and its best/worst/average case analysis
 How to improve worst-case complexity of Quick Sort to O(N log N)

Preparation required before next lecture

 Make sure you understand this lecture completely
 Try to implement radix sort yourself

FIT2004: Lecture 2 - Analysis of Algorithms

54