Lecture 11

Sorting Algorithms
 2

Main content of last lecture

§ Tree
§ Tree: a graph (network) without circular paths
§ Binary tree: an extended linked list with two links in each node
§ Binary search tree: an ordered binary tree
§ AVL tree: A balanced binary search tree
§ B-trees
§ m-way search tree (or call M-way tree of order m): extended
binary search tree that allowing multiple but limited data items
in one node.
§ B-tree of order m: a balanced m-way search tree
§ Heaps: a complete or nearly complete binary tree with the highest
key on the root and all subtree are heaps
 3

The most beautiful paintings of all time

No 3

van Gogh’s
The Starry
Night

No 1 No 2
Da Vinci’s Mona Lisa Vermeer’s Girl with
a Pearl Earring No 4

Klimt’s The Kiss

 4

The most beautiful algorithms of all time

o Depth/Breadth/Best-First Search Algorithms and A* Search
algorithm
o Recursive algorithms - Hanoi Tower, Fibonacci number,
binary tree
o Tree Balancing Algorithms – AVL Algorithm
o B-Tree Algorithm
o Reheapup and reheapdown algorithms
o Minimum Spanning Tree Algorithm
o Shortest Path Algorithm
o Quick Sort Algorithm
 5

Main contents The key of different sorting algorithms is

the differences of computational complexity

§ Sorting problem
What is an algorithm?
§ Selection sort An algorithm is a finite
§ Review of straight selection sort sequence of well-
defined, computer-
§ Heap sort
implementable
§ Exchange sort instructions, typically to
solve a class of
§ Review of straight bubble sort
problems or to perform
§ Quick sort a computation.
§ Insertion sort
§ Review of straight insertion sort
§ Shell sort
General Sort Concepts Data can be
any type

§ Sorting algorithms: sort a set of data according to

the key of the data, for instance, in numeric or
alphabetic order (ascending or descending)
§ Performance: time and space complexity
§ Time complexity is usually an estimation of the number
of comparisons and moves required to sort a list
§ Space complexity is the estimation of required memory
§ Performance is assessed under different situations:
worst case, best case and average case. Default is
worst case.
Typical sorting algorithms

Linear
based

Tree
based

convertible • Heap
• BST – AVL tree
• M-way tree – B-tree
Algorithm Complexity of Simple Sorting

Comparisons moves
(worst case) (worst case)

insertionSort n(n-1)/2 n(n-1)/2 O(n2)

selectionSort n(n-1)/2 3(n-1) O(n2)

bubbleSort n(n-1)/2 3n(n-1)/2 O(n2)

§ The actual steps of processing can be less than that of the worst
case as given above; e.g. if a pass with no exchanges in bubble
sort, the procedure can be terminated immediately.
§ Some sorting algorithms can be more efficient than others on
nearly sorted data.
 9

From selection sort to heap sort

§ Review of straight selection sort
§ Heap sort
 10

Selection sort
§ Selection sort in ascending order
§ Select the smallest element from
the array and swap with the
element at the 1st position.
§ Select the second smallest
element from the rest of the
array and swap with the element
at the 2nd position
§ Keep doing it until finishing the
whole array.
§ The complexity of selection sort is
O(n2).

https://www.cs.usfca.edu/~galles/visualization/ComparisonSort.html
 11

From O(n2) to O(nlog2n)

§ Can we do a better job by using a tree structure? Yes, AVL tree
§ Step 1: Build an AVL tree by inputting the data in the sequence.
» Insert one item into an AVL tree: O(log2n) in the worst case
» Build the whole tree with n items: O(nlog2n)
§ Step 2: Conduct in-order traversal to get the ordered array:
O(nlog2n).
§ Overall complexity is O(nlog2n). However, the process of
rebalancing costs a number of times of traveling over the tree
therefore has a constant times of nlog2n.

A tree structure does help but we have an

even better way!
 12

Heap
§ A heap (max/min heap) is a binary tree with the following
properties:
§ The tree is complete or nearly complete
§ The key value of root node of the tree is greater/less than
or equal to the key value in each of its descendants
§ All the subtrees are also heaps

£ £
 13

Heap Sort 23 56 45 8 78 32
Heap sort process
Step 1: Convert the array of unsorted data
into a heap by repeatedly using reheapUP.
Step 2: Exchange the root (the largest
number) with the last element.
Step 3: Reduce the size of the array by 1
and reheapDown the new array.
Step 4: Do the same with the new array.

78 56

32 56 32 45

8 23 45 8 23

45 32

32 23 8 23

8 45 56 78
 14

Complexity of heap sort

§ Heap sort algorithm:
q Step 1: Convert the array of unsorted data into a heap by
repeatedly using reheapUP, costing O(nlog2n).
q Step 2: Exchange the root (the largest number) with the last
element, costing O(1)..
q Step 3: Reduce the size of the array by 1 and reheapDown the
new array, costing O(log2n).
q Step 4: Go to step 2 with the new array, loop n times, costing
O(nlog2n) in total.
§ The worst-case complexity of heap sort is O(nlog2n).
§ Average case: O(nlog2n)
n 10 100 1,000 10,000 100,000
n2 100 10,000 1,000,000 100,000,000 10,000,000,000
nlog2n 30 700 7,000 130,000 1,700,000
 15

Heap Sort
void heapSort(int list[], int last){
int sorted, holdData, walker;
//Create a heap
for (walker = 1; walker <= last; walker++)
reheapUp (list, walker);

//Sort with the heap

sorted = last;
while (sorted > 0) {
holdData = list[0];
list[0] = list[sorted];
list[sorted] = holdData;
reheapDown (list, 0, --sorted);
} // while
return;
} // heapSort
 16

Heap Sort An example of pseudo-code

void heapSort(int list[], int last){

int sorted, holdData, walker;
//Create a heap
for (walker = 1; walker <= last; walker++)
reheapUp (list, walker); //O(logn)

//Sort with the heap

sorted = last;
while (sorted > 0) {
holdData = list[0];
list[0] = list[sorted];
list[sorted] = holdData;
reheapDown (list, 0, --sorted); //O(logn)
} // while
return;
} // heapSort
From bobble sort to quick sort
§ Review of straight bubble sort
§ Quick sort
Bubble Sort
§ Bubble sort method
§ Swap adjacent elements if out of order
from the last to the first until the least
element is bubbled to the front.
§ Reduce the size by 1 and do the same
again until the whole array is sorted.
Bubble sort
concept

https://www.cs.usfca.edu/~galles/visualization/Comparison
Sort.html
Quick Sort
§ Divide and Conquer:
§ Partition the original list into two partitions with a pivot: all
elements in the left partition are less than the pivot and all
elements in the right partition are greater than or equal to the
pivot.
§ Recursively apply the approach to each partition until the whole
list is sorted.

all < P P all ≥ P

³
Quick Sort
§ Select an element P, called the pivot, from the list. The pivot can be
selected arbitrarily but better to be in the middle and easy to calculate
(e.g. the median value of the left, the right, and the middle element).
§ Arrange the list into two partitions: the elements less than P are put in
the left partition; the elements greater than or equal to P are put in
the right partition.
Left partition P right partition

The partition is done by using two pointers that move from each side towards
the middle. Once the left pointer finds an element larger than P and the right
point finds an element smaller than P, swap the data in the pointers. Once
two pointers meet, place P in between by a swap.

62 74 14 97 87 22 21 85 76 45 84 78
Quick Sort: pivot and partition
Compare the first, middle
and last to determine a
pivot (using bubble sort)

22<62<78, thus 62 is
the pivot, put in the first
position temporarily

• Move sortLeft towards

right and move sortRight
towards left.

• If sortLeft larger than the

pivot, swap it with a
sortRight which is smaller
than the pivot
Quick Sort

>

>

Divide and
conquer for
quick sort

>
Whole Process of Quick Sort

first last
middle

The way of
choosing pivots:
the median of
first, middle and
last elements

Recursive
processing

https://www.cs.usfca.edu/~galles/visualization/ComparisonSort.html
 24

Complexity of Quicksort

Analysis of quicksort for a list of length n

In real applications, quicksort is more efficient than

heapsort, especially for random data but may not efficient
for certain types of data when the chosen pivot is not close
to the median.

t a ff e c ts
b a d p iv o
A e n c y
eff i c i
Comparison of Sorting Algorithms
Data Size Bubble Sort Selection Insertion Shell Sort Quick Sort
10,000 2,198 943 615 45 0
15,000 4,895 2,159 1,378 55 10
20,000 8,985 3,883 2,478 99 11
25,000 13,898 6,037 3,895 156 5
30,000 23,356 11,270 6,773 213 27
35,000 15,045 8,304 280 44
40,000 12,912 356 51
50,000 578 43
100,000 2,127 109
500,000 560
1,000,000 1,775

• Data were randomly generated.

• Ran on Pentium III 800 laptop.
• Time unit is millisecond.
From insertion sort to Shell sort
§ Complexity of sorting algorithms is determined by the
number of comparisons and the number of assignments
§ The approaches introduced above involve lots of
assignments (swaps). For cases when sorting requires big
objects moving, these approaches are not efficient. Shell
sort would perform better in these cases.
§ Review of straight insertion sort
§ Shell sort
From insertion sort to shell sort
§ Insertion sort method
§ Divide the whole array into two parts, sorted and unsorted, initially
the sorted part contains only the first element of the array.
§ Pick up an element from the unsorted part and insert it into the
sorted part to maintain that part to be sorted.
§ Reduce the size of unsorted part by 1 and do the same again until
the whole array is sorted.
Simply consider inserting a sequence
of data into a sorted list like you are
playing cards.

sorted unsorted
Insertion Sort Example
Insertion Sort Example
void insertionSort (int list[], int last) {
for (int i =1; i <= last; i++) { //start from the second element
int hold = list[i];
int walker = i - 1;
while ( walker>=0 && hold < list[walker] ) {
list[walker+1] = list[walker];
walker = walker - 1;
}
list[walker+1]=hold;
}
} // insertionSort

sorted unsorted
Problems of insertion sort
§ If a small item is on the far right, to move the item
to its proper place on the left, all the intervening
items (between where it is and where it should be)
must be shifted one space right. This is close to n
copies, just for one item. The average number of
item moves are n/2. Therefore the performance of
insertion sort is O(n2).

https://www.cs.usfca.edu/~galles/visualization/ComparisonSort.html
Shell Sort
§ Given a sequence of unsorted values, the Shell sort will sort separate k sub-
sequences, where k is an integer greater than 2, but no more than half of the size
of the array, called the increment, each segment is sorted using insertion sort
method

Segment 1: {A[0], A[0+k], A[0+2k],…}

Segment 2: {A[1], A[1+k], A[1+2k],…} Perform insertion

sort in each group
……

Segment k: {A[k-1], A[(k-1)+k], A[(k-1)+2k],…}

§ Repeat the sort with a smaller value of k, say reduced by half

§ When k is reduced to one, a single segment, i.e., the list is sorted

n=16
s1 s2 s3 s4 s1 s2 s3 s4 s1 s2 s3 s4 s1 s2 s3 s4
k=4
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Shell Sort Example
77 21
62 80
14 25
9 70
30 55
before

21 77
62 80
14 25
n=10, k=5: five segments; two elements in one segment 9 70
30 55
after
Still inserting one by one but doing it in groups
 Every second in a group
Shell Sort example (Cont.)

21 14 30 80 70 Grouping at the 14 21 30 70 80 Grouping at

62 9 77 25 55 beginning 9 25 55 62 77 the end
n=10, k=2: two segments; five elements in one segment
 Normal insertion sort on a
Shell Sort Example (Cont.) well sorted array

Just consider that

the previous
rounds are for
preparation.

n=10, k=1: one segment; ten elements in one segment

Shell Sort Algorithm = size -1
void shellSort(int list [], int last) {
int hold, k, walker;
k=last/2; // increment is half of the list length 78 45 23 32 56 18
while(k !=0) { // Run each pre-ordering until k=1 hold = 32
for (int i = k; i<=last; i++) {// standard insert sort for each group walker = 0
hold = list[i];
walker = i - k;
while ( walker>=0 && hold < list[walker] ) { 32 45 23 78 56 18
list[walker+k] = list[walker];
walker = walker - k; hold = 56
} walker = 1
list[walker+k]=hold;
}
k = k / 2; // halve the increment
} 32 45 23 78 56 18
} // shellSort
hold = 18
walker = 2
Comparison of Sorting Algorithms
Data Size Bubble Sort Selection Insertion Shell Sort Quick Sort
10,000 2,198 943 615 45 0
15,000 4,895 2,159 1,378 55 10
20,000 8,985 3,883 2,478 99 11
25,000 13,898 6,037 3,895 156 5
30,000 23,356 11,270 6,773 213 27
35,000 15,045 8,304 280 44
40,000 12,912 356 51
50,000 578 43
100,000 2,127 109
500,000 560
1,000,000 1,775

• Data were randomly generated. The complexity of Shell sort relies on the
• Ran on Pentium III 800 laptop. choice of increments but something close
• Time unit is millisecond. to O(n3/2) to O(n5/3) in average.
 37

Summary
§ Search algorithms may require sorted data
§ Several sorting algorithms available
§ Selection sort, insertion sort, Shellsort, quicksort and
heapsort
§ Complexity of sorting algorithms:

selection insertion bubble heap Shell* quick

Worst case O(n2) O(n2) O(n2) O(nlog2n) O(n2) O(n2)
Average case O(n2) O(n2) O(n2) O(nlog2n) O(n(log2n)2) O(nlog2n)

* The complexity of Shellsort depends on the choice of increments.

 38

Reading
§ Read textbook Chapter 10
§ Visualization of sorting algorithms:
§ https://www.cs.usfca.edu/~galles/visualization/ComparisonSort.ht
ml
§ http://www.sorting-algorithms.com
§ https://www.toptal.com/developers/sorting-algorithms
§ http://www.youtube.com/watch?v=kPRA0W1kECg
§ Download example code from vUWS and read it.
§ Complete your solution for Practical 9. All work must be checked
within this week. There is no further practical sessions for checking
your practical work after this week
§ You must demonstrate your assignment 2 solution in next week
(Week 13) to your tutor. No demonstration no marks.