Sorting (Divide and Conquer)

“Divide and Conquer”

• Very important strategy in computer science:
– Divide problem into smaller parts
– Independently solve the parts
– Combine these solutions to get overall solution
• Idea 1: Divide array into two halves, recursively sort
left and right halves, then merge two halves
Mergesort
• Idea 2 : Partition array into items that are “small”
and items that are “large”, then recursively sort the
two sets Quicksort

Divide and Conquer Sorting -

2/19/03 2
Lecture 14
 Mergesort

8 2 9 4 5 3 1 6

• Divide it in two at the midpoint

• Conquer each side in turn (by recursively
sorting)
• Merge two halves together

Divide and Conquer Sorting -

2/19/03 3
Lecture 14
 Mergesort Example

8 2 9 4 5 3 1 6
Divide
8 2 9 4 5 3 1 6

4
 Mergesort Example

8 2 9 4 5 3 1 6
Divide
8 2 9 4 5 3 1 6
Divide
8 2 9 4 5 3 1 6
Divide

5
 Mergesort Example

8 2 9 4 5 3 1 6
Divide
8 2 9 4 5 3 1 6
Divide
8 2 9 4 5 3 1 6
Divide
1 element 8 2 9 4 5 3 1 6
Merge
2 8 4 9 3 5 1 6
Merge
2 4 8 9 1 3 5 6
Merge
1 2 3 4 5 6 8 9
6
 Auxiliary Array
• The merging requires an auxiliary array.

2 4 8 9 1 3 5 6

Auxiliary array

7
 Auxiliary Array
• The merging requires an auxiliary array.

2 4 8 9 1 3 5 6

Auxiliary array
1

8
 Auxiliary Array
• The merging requires an auxiliary array.

2 4 8 9 1 3 5 6

Auxiliary array
1 2 3 4 5

9
 Merging

i j normal

target

Left completed
copy i j
first

target

10
 Merging
first

second i j Right completed

first

target

11
 Merging

Merge(A[], T[] : integer array, left, right : integer) :

{
mid, i, j, k, l, target : integer;
mid := (right + left)/2;
i := left; j := mid + 1; target := left;
while i < mid and j < right do
if A[i] < A[j] then T[target] := A[i] ; i:= i + 1;
else T[target] := A[j]; j := j + 1;
target := target + 1;
if i > mid then //left completed//
for k := left to target-1 do A[k] := T[k];
if j > right then //right completed//
k : = mid; l := right;
while k > i do A[l] := A[k]; k := k-1; l := l-1;
for k := left to target-1 do A[k] := T[k];
}
12
 Recursive Mergesort

Mergesort(A[], T[] : integer array, left, right : integer) : {

if left < right then
mid := (left + right)/2;
Mergesort(A,T,left,mid);
Mergesort(A,T,mid+1,right);
Merge(A,T,left,right);
}

MainMergesort(A[1..n]: integer array, n : integer) : {

T[1..n]: integer array;
Mergesort[A,T,1,n];
}

13
Iterative Mergesort

Merge by 1

Merge by 2

Merge by 4

Merge by 8

14
 Iterative Mergesort

Merge by 1

Merge by 2

Merge by 4

Merge by 8

Merge by 16

Need of a last copy

15
 Mergesort Analysis
• Let T(N) be the running time for an array of N
elements
• Mergesort divides array in half and calls itself
on the two halves. After returning, it merges
both halves using a temporary array
• Each recursive call takes T(N/2) and merging
takes O(N)

16
 Mergesort Recurrence Relation
• The recurrence relation for T(N) is:
– T(1) < a
• base case: 1 element array constant time
– T(N) < 2T(N/2) + cN
• Sorting N elements takes
– the time to sort the left half
– plus the time to sort the right half
– plus an O(N) time to merge the two halves

• T(N) = O(n log n)

17
 Quicksort
• Quicksort uses a divide and conquer strategy, but
does not require the O(N) extra space that
MergeSort does
– Partition array into left and right sub-arrays
• Choose an element of the array, called pivot
• the elements in left sub-array are all less than pivot
• elements in right sub-array are all greater than pivot
– Recursively sort left and right sub-arrays
– Concatenate left and right sub-arrays in O(1) time

18
 “Four easy steps”
• To sort an array S
1. If the number of elements in S is 0 or 1, then
return. The array is sorted.
2. Pick an element v in S. This is the pivot value.
3. Partition S-{v} into two disjoint subsets, S1 = {all
values x≤v}, and S2 = {all values x≥v}.
4. Return QuickSort(S1), v, QuickSort(S2)

19
 Quick Sort

1. Pick a “pivot”
2. Divide into less-than & greater-than pivot
3. Sort each side recursively
 The steps of QuickSort

S 81
43
31 57 select pivot value
13 75
92 0
65 26

S1 0
S2 partition S
31 75
43 65
13
81
26 57 92

QuickSort(S1) and
S1 S2 QuickSort(S2)
0 13 26 31 43 57 65 75 81 92

S 0 13 26 31 43 57 65 75 81 92 S is sorted

21
 Quicksort Partitioning

• Need to partition the array into left and right

sub-arrays
– the elements in left sub-array are ≤ pivot
– elements in right sub-array are ≥ pivot

22
 Two key steps

• How to pick a pivot?

• How to partition?
 Pick a pivot
• Use the first element as pivot
– if the input is random, ok
– if the input is presorted (or in reverse order)
• all the elements go into S2 (or S1)
• this happens consistently throughout the recursive calls
• Results in O(n2) behavior (Analyze this case later)
• Choose the pivot randomly
– generally safe
– random number generation can be expensive
 Partitioning in-place
– Set pointers i and j to start and end of array
– Increment i until you hit element A[i] > pivot
– Decrement j until you hit elmt A[j] < pivot
– Swap A[i] and A[j]
– Repeat until i and j cross
– Swap pivot (at A[N-2]) with A[i]

25
 Partitioning
partition (arr[], low, high)
{
pivot = arr[low];
i = (low - 1) // Index of smaller element
for (j = low; j <= high- 1; j++)
{
// If current element is smaller than the pivot
if (arr[j] < pivot)
{
i++; // increment index of smaller element
swap arr[i] and arr[j]
}
}
swap arr[i + 1] and arr[low])
return (i + 1)
}

26
 Partitioning:Choosing the pivot

• One implementation (there are others)

27
 QuickSort Example
i j

5 1 3 9 7 0 4 2 6 8

i j

5 1 3 9 7 0 4 2 6 8

i j

5 1 3 9 7 0 4 2 6 8

i j

5 1 3 2 7 0 4 9 6 8

•Move i to the right to be larger than pivot.

•Move j to the left to be smaller than pivot.
•Swap
 i j

5 1 3 2 7 0 4 9 6 8

i j

5 1 3 2 7 0 4 9 6 8

i j

5 1 3 2 7 0 4 9 6 8 Swap I and j
i j

5 1 3 2 4 0 7 9 6 8

i j

5 1 3 2 4 0 7 9 6 8

j i

5 1 3 2 4 0 7 9 6 8

j i

0 1 4 2 4 5 7 9 6 8

S1 < pivot pivot S2 > pivot

Complexity of Quicksort?
 Example
We are given array of n integers to sort:
40 20 10 80 60 50 7 30 100
 Pick Pivot Element
There are a number of ways to pick the pivot element. In
this example, we will use the first element in the array:

40 20 10 80 60 50 7 30 100
pivot_index = 40 20 10 80 60 50 7 30 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index

pivot_index = 40 20 10 80 60 50 7 30 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index

pivot_index = 40 20 10 80 60 50 7 30 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index

pivot_index = 40 20 10 80 60 50 7 30 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index

pivot_index = 40 20 10 80 60 50 7 30 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index

pivot_index = 40 20 10 80 60 50 7 30 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]

pivot_index = 40 20 10 80 60 50 7 30 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]

pivot_index = 40 20 10 30 60 50 7 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 60 50 7 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 60 50 7 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 60 50 7 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 60 50 7 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 60 50 7 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 60 50 7 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 7 50 60 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 7 50 60 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 7 50 60 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 7 50 60 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 7 50 60 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 7 50 60 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 7 50 60 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 7 50 60 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.

pivot_index = 40 20 10 30 7 50 60 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.
5. Swap data[too_small_index] and data[pivot_index]

pivot_index = 40 20 10 30 7 50 60 80 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.
5. Swap data[too_small_index] and data[pivot_index]

pivot_index = 7 20 10 30 40 50 60 80 100
4
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 Partition Result

7 20 10 30 40 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]

<= data[pivot] > data[pivot]

Recursion: Quicksort Sub-arrays

7 20 10 30 40 50 60 80 100
[0] [1] [2] [3] [4] [5] [6] [7] [8]

<= data[pivot] > data[pivot]

 Quicksort Analysis
• Assume that keys are random, uniformly
distributed.
• Best case running time: O(n log2n)
• Worst case running time?
– Recursion:
1. Partition splits array in two sub-arrays:
• one sub-array of size 0
• the other sub-array of size n-1
2. Quicksort each sub-array
– Depth of recursion tree?
 Quicksort Analysis
• Assume that keys are random, uniformly
distributed.
• Best case running time: O(n log2n)
• Worst case running time?
– Recursion:
1. Partition splits array in two sub-arrays:
• one sub-array of size 0
• the other sub-array of size n-1
2. Quicksort each sub-array
– Depth of recursion tree? O(n)
 Quicksort Analysis
• Assume that keys are random, uniformly
distributed.
• Best case running time: O(n log2n)
• Worst case running time?
 Quicksort: Worst Case
• Assume first element is chosen as pivot.
• Assume we get array that is already in order:

pivot_index = 2 4 10 12 13 50 57 63 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.
5. Swap data[too_small_index] and data[pivot_index]

pivot_index = 2 4 10 12 13 50 57 63 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.
5. Swap data[too_small_index] and data[pivot_index]

pivot_index = 2 4 10 12 13 50 57 63 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.
5. Swap data[too_small_index] and data[pivot_index]

pivot_index = 2 4 10 12 13 50 57 63 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.
5. Swap data[too_small_index] and data[pivot_index]

pivot_index = 2 4 10 12 13 50 57 63 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.
5. Swap data[too_small_index] and data[pivot_index]

pivot_index = 2 4 10 12 13 50 57 63 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.
5. Swap data[too_small_index] and data[pivot_index]

pivot_index = 2 4 10 12 13 50 57 63 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

too_big_index too_small_index
 1. While data[too_big_index] <= data[pivot]
++too_big_index
2. While data[too_small_index] > data[pivot]
--too_small_index
3. If too_big_index < too_small_index
swap data[too_big_index] and data[too_small_index]
4. While too_small_index > too_big_index, go to 1.
5. Swap data[too_small_index] and data[pivot_index]

pivot_index = 2 4 10 12 13 50 57 63 100
0
[0] [1] [2] [3] [4] [5] [6] [7] [8]

<= data[pivot] > data[pivot]

71
Using big-Θ notation, we get the same result as for merge sort: Θ(nlogn)

72
Divide and Conquer Sorting - Lecture
73
14
Average Case:
To do average case analysis, we need to consider all possible permutation
of array and calculate time taken by every permutation which doesn’t
look easy.
We can get an idea of average case by considering the case when partition
puts O(n/9) elements in one set and O(9n/10) elements in other set. Following
is recurrence for this case.

T(n) = T(n/9) + T(9n/10) + (n)

Solution of above recurrence is also O(nLogn)

Solution of above recurrence is also O(nLogn)

75
T(n) = T(n/9) + T(9n/10) + (n)
 Quicksort Analysis
• Assume that keys are random, uniformly
distributed.
• Best case running time: O(n log2n)
• Worst case running time?
– Recursion:
1. Partition splits array in two sub-arrays:
• one sub-array of size 0
• the other sub-array of size n-1
2. Quicksort each sub-array
– Depth of recursion tree? O(n)
– Number of accesses per partition?
 Quicksort Analysis
• Assume that keys are random, uniformly
distributed.
• Best case running time: O(n log2n)
• Worst case running time?
– Recursion:
1. Partition splits array in two sub-arrays:
• one sub-array of size 0
• the other sub-array of size n-1
2. Quicksort each sub-array
– Depth of recursion tree? O(n)
– Number of accesses per partition? O(n)
 Pseudo-code
Input: an array a[left, right]

QuickSort (a, left, right) {

if (left < right) {
pivot = Partition (a, left, right)
Quicksort (a, left, pivot-1)
Quicksort (a, pivot+1, right)
}
}

Compare with MergeSort:

MergeSort (a, left, right) {
if (left < right) {
mid = divide (a, left, right)
MergeSort (a, left, mid-1)
MergeSort (a, mid+1, right)
merge(a, left, mid+1, right)
}
}