Data Structures (67109-2)- Recitation 4

26-30.4.2024
The Hebrew University of Jerusalem

1 Linear time sorting

You saw in Yossi’s lecture that a comparison based sorting algorithm has worst time complexity
œ (n log n). However, making additional input assumptions can enable sorting in linear time. Today
we will describe 3 such sorting algorithms: Counting Sort, Radix Sort and Bucket Sort.
Definition 1. Stable Sort Algorithms
-
331 IIIN ex :
A
Stable sorting algorithms maintain the relative order of records with equal keys (i.e. values). age
= 3
Avi
If a comes before b in the original list and has the same key (i) - [. . . , ia , . . . , ib , . . .], a stable
nom =

&
sort algorithm ensures a comes before b in the sorted list. age 3
=

Bina
nom =

Si tri selon age

L
:

4
/31
est
1.1 Counting Sort
on sait pas qui
qui

b
&

si
Counting Sort is a sorting technique based on keys between a specific range.
Input: n integer numbers in the range [0, . . . , k] where k = O (n) is an integer.
The idea: determine for each input element x its rank - the number of elements less than x. Once
we know the rank r of x, we can place it in position r + 1.
Algorithm 1 Counting Sort

Input: arr: unsorted array of integersœ {0} fi [k], k: largest number in the array.
Output: sorted_arr: sorted arr.
1 Function Counting-Sort(arr, k):
/* allocate counting array and set to zero */
2 C =Allocate[0 . . . k] cette allocat do mémoire vant 0(1)
3 for i = 0 to k do mettre to les ([i) a 0 0(k) =
Û « (k)
4 C [i] = 0
5 for j = 0 to length(arr)≠1 do Û « (n)
↑
E 6 C [arr [j]] + = 1 // now C [i] contain the number of elements = to i
7 for i = 1 to k do Û « (k)
2[ 8 C [i] + = C [i ≠ 1] // now C [i] contain the number of elements Æ to i
9 sorted_arr =Allocate[0 . . . n ≠ 1]

[
10 for j =length(arr)≠1 downto 0 do Û « (n)
·
11 sorted_arr [C [arr [j]] ≠ 1] = arr [j] // place element
12 C [arr [j]] ≠ = 1 // reduce by 1

[1]
 1 202 6 ①
arr =

k = 6

f On metds C
C -

-
2 120001
de
ab on a
0123456
- chaque élément
k + 1 = 6

2 Si ri
ya
:

3
on les dsarr[v
place : r+ c[i]]
0 = 234 S

Sorted aur- -
o 01 2126
A ↑
Cb st +

petit que 0
?
ab st + petit
le met que
1
? 2 éléments
> O donc on
*onc met 1 de la
-

O à la
on
de la case
Case 2 à 2+ c[1] = 3
Case c[0] = 2

aut [7 2 , 0, 2, 6 , 03
([arr(o]] C[1]
=
, = =

-c 4 117 111111 +

CCarr[1]] [2]
-
=
<
0123456 =

Clarr[3]] = <[0]
155151
x =

CCarr[n]] =
c [2]

3 Sortedarr =
2012/215
12343
:
O e ([i] + Ci -1] :

sor-arr[]] ) arr(s) 0
3
=
=

([1] + ((0] > 1+ 2 =

-

2
-
I
([2] + ((z] => 2 + 3 = 5

arr[4] S
sor [C(arr[i]
u 1] - =
=

([3] + <(2) -
> 0 + S= 5
e
S
Let’s show an example run of Counting Sort:

arr 1 4 1 2 7 5 2 3 1 7

Lines 5-6

0 1 2 3 4 5 6 7 8 9

Lines 7-11 9
8
7
6
5
4
3
2
1
0
0 1 2 3 4 5 6 7 8 9

1.1.1 Counting Sort Time Complexity

Time complexity: T (n) = « (n + k). Two loops of « (k) and two loops of « (n).
Space complexity: S (n) = « (n + k). The Space complexity for allocating sorted_arr (line 9)
is « (n). Additionally, we have allocated an array C that has k + 1 numbers (line 2). Therefore,
the space complexity is « (n + k).

1.1.2 Loop Invariant In the last few TA’s and exercises you saw a few examples for proving
correctness of algorithms. This will be a key topic in your next course in Algorithms, but it is
also important for our course. As you previously saw, we prove an algorithm’s correctness using
an induction, which is proved on the iterations of the algorithm (each iteration is accounted
as the i of the induction). For proving the correctness, we sometimes use what is called a loop
invariant. Knowing a code’s invariant is essential in understanding the effect of a loop and proving
its correctness.
Definition 2. Loop Invariant

A loop invariant is a property of a program loop that is true before (and after) each iteration.
It is a logical assertion, sometimes checked within the code by an assertion call. There are
three parts of a loop invariant that we need to show:
1. Initialization: It is true prior to the first iteration of the loop.
2. Maintenance: If it is true before an iteration of the loop, it remains true before the
next iteration.
3. Termination: When the loop terminates, the invariant gives us a useful property that
helps show that the algorithm is correct.

[2]
1.2 Radix Sort

Radix sort is a non-comparative sorting algorithm. It avoids comparison by creating and distribut-
ing elements into buckets according to their radix. For elements with more than one significant
digit, this bucketing process is repeated for each digit, while preserving the ordering of the prior
step, until all digits have been considered.
Input: n integer numbers with (at most) d digits.
The idea: look at one digit at a time and sort the numbers according to this digit only.
Start from the least significant digit, working up to the most significant one. Since there are only
10 different digits, only 10 places are used for each column, so we can use counting sort for each
call, with k = 9, and since this is constant, we definitely have k = O (1) = O (n).
Algorithm 2 Radix Sort

Input: arr: unsorted array of n integers of d digits.

Output: Sorted arr.
1 Function Radix-Sort(arr, d):
2 for i = 1 to d do
3 Use a stable sort to sort arr on digit i

Let’s us show an example run of Radix Sort:

Input Output

1.2.1 Radix Sort Complexity

Assuming we use a linear stable sort, such as Counting Sort, and d is constant.
Time complexity: Q R
const
˙˝¸˚
c d
T (n) = d · « (n) = « a d · nb = « (n)

Space complexity: S (n) = « (n). We can use one more array (« (n)) to copy the temporary
results at each iteration and pass it as the target sorting array to the counting sort in the following
iteration (line 3). In addition, if k = 10, then the counting array C can be allocated once and
reused in each iteration, resulting in a constant space complexity of O (1). Therefore, the overall
space complexity of the algorithm is « (n).

[3]
Remark 1. This algorithm MUST use a stable sort (def. 1), otherwise it fails.

1.2.2 Radix Sort Correctness - by Loop Invariant

Claim. Radix Sort is a correct, stable sorting algorithm.

Proof. We proceed by loop invariant considering the number of digits d.

First, we introduce some notation. For integer x, a d-digit number, we define xj to be the number
formed from the last j digits of x. (e.g. if x = 2345 then x1 = 5; x2 = 45; x3 = 345; x4 = x).

Loop Invariant: In the outer for-loop, just before the ith iteration, the integers in array
arr are sorted according to the values induced by their xi≠1 value.

Initialization: Before the iteration with i = 0, the array is not sorted, which is trivially
true.
Maintenance: Assume that the loop invariant holds before the ith iteration; we aim to
prove that it holds also before the i + 1 iteration. To do this, we must establish
that for any pair of integers x, y in arr, their relative order is determined by the
values of xi and yi . WLOG, assume xi < yi , it is necessary to show that x will
appear before y. There are two cases:
1. The ith digit of x < ith digit of y - Radix-Sort will put x before y in the current
round of sorting.
2. The ith digit of x=ith digit of y - Since xi < yi , by the loop invariant,
xi≠1 < yi≠1 , so x appears before y before the ith iteration, and since the ith
digits are the same, their order will not change in the new iteration, so they
will remain in the same order.
Termination: When i = d, the array is sorted. You will prove by invariant that the algo-
rithm is stable in ex4.

Remark 2. It is possible to perform Radix Sort on any basis.

1.3 Bucket Sort

Bucket sort, is a sorting algorithm that works by distributing the elements of an array into a number
of buckets. Each bucket is then sorted individually, either using a different sorting algorithm, or
by recursively applying the bucket sorting algorithm.
Input: n real numbers in the interval [0, . . . , 1), uniformly distributed. (this will be explained
further next week, but for now we may think of this as the chance of getting a number x and a
number y are equally likely).
The idea: Divide the interval [0, . . . , 1) to n buckets 0, n1 , n2 , . . . , n≠1
n
. Put each element ai into its
matching bucket, such that n Æ ai Æ n . Since the numbers are uniformly distributed, not too
i i+1

many elements will be placed in each bucket. If we insert them in order (using Insertion-Sort), the
buckets and the elements in them will always be in sorted order.

[4]
 Algorithm 3 Bucket Sort

Input: arr: unsorted array of n real numbers in the interval [0, 1).
Output: Sorted arr.
1 Function Bucket-Sort(arr):
2 n = length(arr)
3 B Ω a list of n lists // B [i] are the buckets
4 for i = 0 to n ≠ 1 do
5 insert arr [i] into list B [Ân · arr [i]Ê] // arr [i] œ [0, 1), Ân·arr[i]Ê
n
Æ arr [i] Æ Ân·arr[i]Ê+1
n

6 for i = 0 to n ≠ 1 do
7 Insertion-Sort(B [i])
8 concatenate lists B [0] , . . . , B [n ≠ 1] in order

Let’s us show an example run of Bucket Sort (without normalization):

72 17 39 26 78 94 21 12 23 68

23
12 21 78
17 26 39 68 72 94

21
12 23 72
17 26 39 68 78 94

12 17 21 23 26 39 68 72 78 94

Remark 3. Sorting buckets (line 7) can be done with any sorting algorithm. Insertion sort is effi-
cient when dealing with small inputs, which is why we prefer it.

Remark 4. We can use Bucket Sort for n real numbers in any interval [a, b). To do this, we need
to normalize the input, use bucket sort, and then unnormalize the output.

1.3.1 Bucket Sort Stability

Bucket sort is stable if the underlying sort is also stable and on our example it is (insertion sort,
line 7).

1.3.2 Bucket Sort Time Complexity

Time complexity:

[5]
The initialization and first loop (line 4) take O (n).

worst case In the worst case, n numbers end up in the same bucket and the algorithm takes O (n2 )
(line 7).
Average case In the average case only a constant number of elements fall in each bucket, at least
with very high probability, so inesrtion sort takes O (1) on each bucket, and therefore
the second loop also takes O (n).

Space complexity: S (n) = « (n). The Space complexity for allocating B (line 3) is « (n).
So to summarize this recitation, with some additional assumptions, we can sort elements in optimal
time and space œ (n).

2 (If there is time) Loop Invariant - Examples

Example 1. What is the loop invariant? i 9 4/15

j + :
=

oui autla :
S
-

1 j=9 Lère itération : i 0 j9 =

2 for i = 0 to 10 do => j+ i =
9
3 j-=1
Maintenance
:
C :
j+ i = g

Solution . i + j == 9

Example 2. What is the loop invariant?

I : max= o
A: array
-

max = ≠INF // minus infinity

1
2 for i = 0 to n do 1 -
e

3 if A [i] > max then Afi]

=> max
max = A [i]
=

Solution . max is always maximum among the first i elements of array A.

Example 3. What is the loop invariant?

[6]
 Algorithm 4 Quick Sort

Input: Unsorted array arr, array boundaries - low, high

Output: Sorted arr
1 Function QuickSort(arr, low, high):
2 if low < high then
3 pi = partition(arr, low, high)
4 QuickSort(arr, low, pi ≠1) // before pi
5 QuickSort(arr, pi +1, high) // after pi

1 Function partition(arr, low, high):

2 pivot = arr [high]
3 i = low ≠ 1
4 for j = low to high ≠ 1 do
5 if arr [j] Æ pivot then
6 i+ = 1
7 swap(arr [i], arr [j])

8 swap(arr [i + 1], arr [high])

9 return i + 1

Solution . There are three:

1. Pivot element is placed at its correct position.

2. Elements less than pivot element lie on the left side of pivot element.
3. Elements greater than pivot element lie on the right side of pivot element.

[7]