‫إن هللا ال يغير ما بقوم حتى‬

‫يغيروا ما بأنفسهم‬
‫‪...‬إذا لم تعمل على تحقيق حلمك‬

‫فإن شخصا ما سيقوم بتوظيفك‬

‫!! لتعمل على تحقيق حلمه هو‬
 ‫بينما األب ينظف سيارته الجديدة بالمطرقه أخذ ابنه البالغ من العمر سنتين‬ ‫•‬
‫مسمار وخدش به في جانب السيارة‬
‫قام األب وبغضب شديد يضرب يد ابنه من دون ان يشعر بأنه كان يضربها‬ ‫•‬
‫بمقبض مطرقة !!‬
‫انتبه األب متأخرًا لما حصل وأخذ ابنه للمستشفى وفقد االبن جميع اصابعه بسبب‬ ‫•‬
‫الكسور الكثيرة التي تعرض لها !!‬
‫عندما رأى الطفل أباه قال له متى ستنبت أصابعي يا أبي ؟؟!‬ ‫•‬
‫كان السؤال وقع كالصاعقة على األب‬ ‫•‬
‫خرج األب وتوجه الى سيارته وضربها عدة مرات‬ ‫•‬
‫جلس أمام سيارته وكله ندم على ما حدث البنه‬ ‫•‬
‫ثم نظر الى المكان الذي خدشه ابنه ووجد مكتوب‬ ‫•‬
‫{ أنا أحبك يا أبى}‬ ‫•‬
‫أعلـــــــم ان هناك اشياء تنظر لها بعين عقلك واشياء تنظر لها بعين قلبك وعقلك‬ ‫•‬
‫عمرك‬ ‫فـــال تجعل عصبيتك تعيشك في ندم طوال‬ ‫•‬
‫‪Chapter 9: Sorting‬‬ ‫‪3‬‬
 CS214 – Data Structures
Lecture 6&7: Quadratic Sorting

Slides by
Mohamed El-Ramly, PhD
Basheer Youssef ,PhD
 http://en.wikipedia.org/wiki/
List_of_mathematical_series

5
6
 Agenda
0 Why Sorting?
1. Quadratic Sorting algorithms
• Selection Sort
• Insertion Sort
• Bubble Sort Included but read yourself
2. Theoretical Boundaries of Sorting Problem
3. STL support for sorting
4. Sub-quadratic Sorting
• Merge Sort I
• Shell Sort
• Quick Sort
• Heap Sort
7
 Divide and Conquer
1. Base Case, solve the problem directly
if it is small enough

2. Divide the problem into two or more

similar and smaller subproblems

3. Recursively solve the subproblems

4. Combine solutions to the subproblems

Chapter 9: Sorting 8
Chapter 9: Sorting 9
 Quadratic Sorting Algorithms
1. Take long time
2. Are O(n2), which is quite slow
3. Theoritical limits show that we can
do better.
4. It is possible to achieve O(n log(n)),
at least theoritically.

Chapter 9: Sorting 10
Shell Sort: A Better Insertion Sort
• Shell sort is a type of insertion sort but with O(n^(3/2)) or
better performance
• Named after its discoverer, Donald Shell
• the first sub quadratic sorting algorithm
• Divide and conquer approach to insertion sort
• Instead of sorting the entire array, sort many smaller
subarrays using insertion sort before sorting the entire
array

Chapter 9: Sorting 11
 Shell Sort

• A variation of the insertion sort

• But faster than O(n2)
• Done by sorting subarrays of equally spaced indices
• Instead of moving to an adjacent location an element moves several
locations away
• Results in an almost sorted array
• This array sorted efficiently with ordinary insertion sort
• Wanted to stop moving data small distances (in the case of insertion
sort and bubble sort) and stop making swaps that are not helpful (in
the case of selection sort)
• Start with sub arrays created by looking at data that is far apart and
then reduce the gap size

Chapter 9: Sorting 12
Chapter 9: Sorting 13
Chapter 9: Sorting 14
 Shell Sort

Donald Shell suggested that the initial separation

between indices be n/2 and halve this value at each
pass until it is 1.
An array has 13 elements, and the subarrays formed
by grouping elements whose indices are 6 apart.
Chapter 9: Sorting 15
 Shell Sort

The subarrays after they are sorted, and the array

that contains them.
Chapter 9: Sorting 16
 Shell Sort

The subarrays by grouping elements whose indices are

3 apart
Chapter 9: Sorting 17
 Shell Sort

The subarrays after they are sorted, and the array that
contains them.
Chapter 9: Sorting 18
 Efficiency of Shell Sort

• Efficiency is O(n2) for worst case

• If n is a power of 2
• Average-case behavior is O(n1.5)
• Shell sort uses insertion sort repeatedly.
• Initial sorts are much smaller, the later sorts are on
arrays that are partially sorted, the final sort is on an
array that is almost entirely sorted.

Chapter 9: Sorting 19
 ShellSort in practice
46 2 83 41 102 5 17 31 64 49 18
Gap of five. Sort sub array with 46, 5, and 18
5 2 83 41 102 18 17 31 64 49 46
Gap still five. Sort sub array with 2 and 17
5 2 83 41 102 18 17 31 64 49 46
Gap still five. Sort sub array with 83 and 31
5 2 31 41 102 18 17 83 64 49 46
Gap still five Sort sub array with 41 and 64
5 2 31 41 102 18 17 83 64 49 46
Gap still five. Sort sub array with 102 and 49
5 2 31 41 49 18 17 83 64 102 46
Continued on nextChapter
slide:
9: Sorting 20
 Completed Shellsort
5 2 31 41 49 18 17 83 64 102 46
Gap now 2: Sort sub array with 5 31 49 17 64 46
5 2 17 41 31 18 46 83 49 102 64
Gap still 2: Sort sub array with 2 41 18 83 102
5 2 17 18 31 41 46 83 49 102 64
Gap of 1 (Insertion sort)
2 5 17 18 31 41 46 49 64 83 102

Array sorted

Chapter 9: Sorting 21
Pseudocode of ShellSort

Chapter 9: Sorting 22
 Java ShellSort Code
public static void shellsort(Comparable[] list)
{ Comparable temp; boolean swap;
for(int gap = list.length / 2; gap > 0; gap /= 2)
for(int i = gap; i < list.length; i++)
{ Comparable tmp = list[i];
int j = i;
for( ; j >= gap &&
tmp.compareTo( list[j - gap] ) < 0;
j -= gap )
list[ j ] = list[ j - gap ];
list[ j ] = tmp;
}
}
// See C++ version in Drozdek’s Book
Chapter 9: Sorting 23
Chapter 9: Sorting 24
 Merge Sort
• A merge is a common data processing operation that is
performed on two sequences of data with the following
characteristics
• Both sequences contain items with a common
compareTo method
• The objects in both sequences are ordered in
accordance with this compareTo method

Chapter 9: Sorting 25
Chapter 9: Sorting 26
 Merge Algorithm

• Merge Algorithm
• Access the first item from both sequences
• While not finished with either sequence
• Compare the current items from the two sequences, copy the
smaller current item to the output sequence, and access the
next item from the input sequence whose item was copied
• Copy any remaining items from the first sequence to
the output sequence
• Copy any remaining items from the second sequence
to the output sequence

Chapter 9: Sorting 27
 Merge Sort: Idea
Divide into
two halves
A FirstPart SecondPart
Recursively
sort

FirstPart SecondPart

Merge

A is sorted!

Chapter 9: Sorting 28
 Merge Sort: Algorithm

Merge-Sort (A, left, right)

if left ≥ right return
else
middle ← (left+right)/2
Recursive Call
Merge-Sort(A, left, middle)
Merge-Sort(A, middle+1, right)
Merge(A, left, middle, right)

Chapter 9: Sorting 29
 Merge-Sort: Merge
Sorted

A:

merge
Sorted Sorted
FirstPart SecondPart

A:

A[left] A[middle]
Chapter 9: Sorting A[right] 30
 Merge-Sort: Merge Example

A: 2
5 3
5 7 28
15 8 30
1 4
6 5 14
10 6

L: R:
3 5 15 28 6 10 14 22

Temporary Arrays
Chapter 9: Sorting 31
 Merge-Sort: Merge Example

A:
3
1 5 15 28 30 6 10 14

k=0

L: R:
3
2 15
3 28
7 30
8 6
1 10
4 14
5 22
6

i=0 j=0
Chapter 9: Sorting 32
 Merge-Sort: Merge Example

A:
1 2
5 15 28 30 6 10 14

k=1

L: R:
3
2 5
3 15
7 28
8 6
1 10
4 14
5 22
6

i=0 j=1
Chapter 9: Sorting 33
 Merge-Sort: Merge Example

A:
1 2 3 28 30
15 6 10 14

k=2

L: R:
2 3 7 8 6
1 10
4 14
5 22
6

i=1 j=1
Chapter 9: Sorting 34
 Merge-Sort: Merge Example

A:
1 2 3 4 6 10 14

k=3

L: R:
2 3 7 8 6
1 10
4 14
5 22
6

i=2 j=1
Chapter 9: Sorting 35
 Merge-Sort: Merge Example

A:
1 2 3 4 5 6 10 14

k=4

L: R:
2 3 7 8 6
1 10
4 14
5 22
6

i=2 j=2
Chapter 9: Sorting 36
 Merge-Sort: Merge Example

A:
1 2 3 4 5 6 10 14

k=5

L: R:
2 3 7 8 6
1 10
4 14
5 22
6

i=2 j=3
Chapter 9: Sorting 37
 Merge-Sort: Merge Example

A:
1 2 3 4 5 6 7 14

k=6

L: R:
2 3 7 8 6
1 10
4 14
5 22
6

i=2 j=4
Chapter 9: Sorting 38
 Merge-Sort: Merge Example

A:
1 2 3 4 5 6 7 8
14

k=7

L: R:
3
2 5
3 15
7 28
8 6
1 10
4 14
5 22
6

i=3 j=4
Chapter 9: Sorting 39
 Merge-Sort: Merge Example

A:
1 2 3 4 5 6 7 8

k=8

L: R:
3
2 5
3 15
7 28
8 6
1 10
4 14
5 22
6

i=4 j=4
Chapter 9: Sorting 40
Merge(A, left, middle, right)
1. n1 ← middle – left + 1
2. n2 ← right – middle
3. create array L[n1], R[n2]
4. for i ← 0 to n1-1 do L[i] ← A[left +i]
5. for j ← 0 to n2-1 do R[j] ← A[middle+j]
6. k ← i ← j ← 0
7. while i < n1 & j < n2
8. if L[i] < R[j]
9. A[k++] ← L[i++]
10. else
11. A[k++] ← R[j++]
12. while i < n1
13. A[k++] ← L[i++]
14. while j < n2
15. A[k++] ← R[j++]

n = n1+n2
Space: n
Chapter 9: Sorting 41
Time : cn for some constant c
 Merge-Sort(A, 0, 7)
Divide
A: 6 2 8 4 3 3 7 7 5 5 11

Chapter 9: Sorting 42
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 0, 3) , divide
A: 3 7 5 1

6 2 88 44

Chapter 9: Sorting 43
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 0, 1) , divide
A: 3 7 5 1

8 4

6 2
2

Chapter 9: Sorting 44
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 0, 0) , base case
A: 3 7 5 1

8 4

Chapter 9: Sorting 45
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 0, 0), return
A: 3 7 5 1

8 4

6 2

Chapter 9: Sorting 46
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 1, 1) , base case
A: 3 7 5 1

8 4

Chapter 9: Sorting 47
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 1, 1), return
A: 3 7 5 1

8 4

6 2

Chapter 9: Sorting 48
 Merge-Sort(A, 0, 7)
Merge(A, 0, 0, 1)
A: 3 7 5 1

8 4

2 6

Chapter 9: Sorting 49
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 0, 1), return
A: 3 7 5 1

2 6 8 4

Chapter 9: Sorting 50
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 2, 3) , divide
A: 3 7 5 1

2 6

8 4

Chapter 9: Sorting 51
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 2, 2), base case
A: 3 7 5 1

2 6

Chapter 9: Sorting 52
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 2, 2), return
A: 3 7 5 1

2 6

8 4

Chapter 9: Sorting 53
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 3, 3), base case
A:

2 6

Chapter 9: Sorting 54
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 3, 3), return
A: 3 7 5 1

2 6

8 4

Chapter 9: Sorting 55
 Merge-Sort(A, 0, 7)
Merge(A, 2, 2, 3)
A: 3 7 5 1

2 6

4 8

Chapter 9: Sorting 56
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 2, 3), return
A: 3 7 5 1

2 6 4 8

Chapter 9: Sorting 57
 Merge-Sort(A, 0, 7)
Merge(A, 0, 1, 3)
A: 3 7 5 1

2 4 6 8

Chapter 9: Sorting 58
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 0, 3), return
A: 2 4 6 8 3 7 5 1

Chapter 9: Sorting 59
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 4, 7)
A: 2 4 6 8

3 7 5 1

Chapter 9: Sorting 60
 Merge-Sort(A, 0, 7)
Merge (A, 4, 5, 7)
A: 2 4 6 8

1 3 5 7

Chapter 9: Sorting 61
 Merge-Sort(A, 0, 7)
Merge-Sort(A, 4, 7), return
A: 2 4 6 8 1 3 5 7

Chapter 9: Sorting 62
 Merge-Sort(A, 0, 7)
Merge-Sort(A,
Merge(A, 0, 3, 0,
7)7), done!
A: 1 2 3 4 5 6 7 8

Chapter 9: Sorting 63
 Merge-Sort Analysis
n cn

n/2 n/2 2 × cn/2 = cn

log n levels
n/4 n/4 n/4 n/4 4 × cn/4 = cn

n/2 × 2c = cn
2 2 2

Total: cn log n

• Total running time: (nlogn)

• Total Space:  (n)
Chapter 9: Sorting 64
 Merge-Sort Summary

Approach: divide and conquer

Time
• Most of the work is in the merging
• Total time: (n log n)
Space:
 (n), more space than other sorts.

Chapter 9: Sorting 65
 Analysis of Merge

• For two input sequences that contain a total of n

elements, we need to move each element’s input
sequence to its output sequence
• Merge time is O(n)
• We need to be able to store both initial sequences and
the output sequence
• The array cannot be merged in place
• Additional space usage is O(n)

Chapter 9: Sorting 66
 Read the book on MergeSort
• Drozdek, Chapter 9.

Chapter 9: Sorting 67
Chapter 9: Sorting 68
 Quicksort
• Developed in 1962 by C. Hoare
• Quicksort rearranges an array into two parts so
that all the elements in the left subarray are less
than or equal to a specified value, called the
pivot
• Each subarray is sorted separately by
recursively using the same approach, and so on
for sub-subarrays, etc.
• Average case for Quicksort is O(n log n)

Chapter 9: Sorting 69
 Quick Sort
• Divide:
• Pick any element p as the pivot, e.g, the first element
• Partition the remaining elements into
FirstPart, which contains all elements < p
SecondPart, which contains all elements ≥ p

• Recursively sort the FirstPart and SecondPart

• Combine: no work is necessary since sorting

is done in place

Chapter 9: Sorting 70
 Pseudocode Quicksort
• quickSort (array [])
• if length (array) > 1
• choose pivot; // partition array into two subarray
quickSort (subarray1);
• quickSort (subarray2);

Chapter 9: Sorting 71
 How does it work?
• It partitions the array by:
• Finding a pivot
• First element
• Middle element
• Median value
• Median of first, last and middle elements

• Scanning the array to place elements in their

right positions

• Repeat recursively
Chapter 9: Sorting 72
 Finding a Good Pivot
• Assume we have the data
1 2 4 5 6 7 9 11 10 12
Which is nearly sorted.
• What happens if we pick the first element
as a pivot?
• What sorting algorithm will Quick Sort turn
to be?

Chapter 9: Sorting 73
 Finding a Good Pivot
• First element // bad for sorted data
• Middle element // ok
• Median value // expensive to calculate
• Median of first, last and middle elements
// good idea

Chapter 9: Sorting 74
 Quick Sort

A: p
pivot Partition
FirstPart SecondPart

x<p p p≤x
Recursive call
Sorted Sorted
FirstPart SecondPart

x<p p p≤x

Sorted 75
 Quick Sort
Quick-Sort(A, left, right)
if left ≥ right return
else
middle ← Partition(A, left, right)
Quick-Sort(A, left, middle–1 )
Quick-Sort(A, middle+1, right)
end if

Chapter 9: Sorting 76
 Partition

A: p

A: p x<p p≤x

A:
p x<p p p≤x

Chapter 9: Sorting 77
 Partition Example

A: 4 8 6 3 5 1 7 2

Chapter 9: Sorting 78
 Partition Example

i=0

A: 4 8 6 3 5 1 7 2

j=1

Chapter 9: Sorting 79
 Partition Example

i=0

A: 4 8 6 3 5 1 7 2

j=1

Chapter 9: Sorting 80
 Partition Example

i=0

A: 4 8 6 3 5 1 7 2

j=2

Chapter 9: Sorting 81
 Partition Example

i=0i=1

A: 4 8
3 6 3
8 5 1 7 2

j=3

Chapter 9: Sorting 82
 Partition Example

i=1

A: 4 3 6 8 5 1 7 2

j=4

Chapter 9: Sorting 83
 Partition Example

i=1

A: 4 3 6 8 5 1 7 2

j=5

Chapter 9: Sorting 84
 Partition Example

i=2

A: 4 3 1
6 8 5 6
1 7 2

j=5

Chapter 9: Sorting 85
 Partition Example

i=2

A: 4 3 1 8 5 6 7 2

j=6

Chapter 9: Sorting 86
 Partition Example

i=2i=3

A: 4 3 1 2
8 5 6 7 8
2

j=7

Chapter 9: Sorting 87
 Partition Example

i=3

A: 4 3 1 2 5 6 7 8

j=8

Chapter 9: Sorting 88
 Partition Example

i=3

A: 24 3 1 4
2 5 6 7 8

Chapter 9: Sorting 89
 Partition Example

pivot in
correct position

A: 2 3 1 4 5 6 7 8

x<4 4≤x

Chapter 9: Sorting 90
Partition(A, left, right)
1. x ← A[left]
2. i ← left
3. for j ← left+1 to right
4. if A[j] < x then
5. i ← i + 1
6. swap(A[i], A[j])
7. end if
8. end for j
9. swap(A[i], A[left])
10. return i

n = right – left +1
Time: cn for some constant c
Space: constant Chapter 9: Sorting 91
 Quick-Sort(A, 0, 7)
Partition
A: 2
4 3
8 1
6 34 55 16 77 28

Chapter 9: Sorting 92
Chapter 9: Sorting 93
Chapter 9: Sorting 94
 Quick-Sort(A, 0, 7)
Quick-Sort(A, 0, 2) , partition
A: 4 5 6 7 8

21 2
3 31

Chapter 9: Sorting 95
 Quick-Sort(A, 0, 7)
Quick-Sort(A, 0, 0) , base
returncase
4 5 6 7 8

1 2 3

Chapter 9: Sorting 96
 Quick-Sort(A, 0, 7)
Quick-Sort(A, 1, 1) , base case
4 5 6 7 8

1 2

Chapter 9: Sorting 97
 Quick-Sort(A, 0, 7)
Quick-Sort(A,2,
Quick-Sort(A, 0,2),
2),return
return
1 2 3 4 5 6 7 8

1 2 3

Chapter 9: Sorting 98
 Quick-Sort(A, 0, 7)
Quick-Sort(A,2,
Quick-Sort(A, 4,2),
7) ,return
partition
1 2 3 4

55 6 7 8

Chapter 9: Sorting 99
 Quick-Sort(A, 0, 7)
Quick-Sort(A, 5, 7) , partition
1 2 3 4

6 7 8

Chapter 9: Sorting 100

 Quick-Sort(A, 0, 7)
Quick-Sort(A, 6, 7) , partition
1 2 3 4

7 88

Chapter 9: Sorting 101

 Quick-Sort(A, 0, 7)
returncase
Quick-Sort(A, 7, 7) , base
1 2 3 4

7 8

Chapter 9: Sorting 102

Chapter 9: Sorting 103
 Quick-Sort(A, 0, 7)
Quick-Sort(A, 6, 7) , return
1 2 3 4

6 7 8

Chapter 9: Sorting 104

 Quick-Sort(A, 0, 7)
Quick-Sort(A, 5, 7) , return
1 2 3 4

5 6 7 8

Chapter 9: Sorting 105

 Quick-Sort(A, 0, 7)
Quick-Sort(A, 4, 7) , return
1 2 3 4 5 6 7 8

Chapter 9: Sorting 106

 Quick-Sort(A, 0, 7)
Quick-Sort(A, 0, 7) , done!
1 2 3 4 5 6 7 8

Chapter 9: Sorting 107

 Quick-Sort: Best Case
• Even Partition (It becomes like .......???...... algorithm)
n cn

n/2 n/2 2 × cn/2 = cn

log n levels
n/4 n/4 n/4 n/4 4 × cn/4 = cn

n/2 × 2c = cn
2 2 2

Total time: (nlogn)

Chapter 9: Sorting 108
 Quick-Sort: Worst Case
• Unbalanced Partition

n cn

n-1 c(n-1)

n-2 c(n-2)

3c
3
Happens only if
• input is sortd
2 2c
• input is reversely sorted
• and we pick the first element
as pivot always Total time: O(n2)
Chapter 9: Sorting 109
 Quick-Sort: an Average Case

• Suppose the split is 1/10 : 9/10

n cn

0.1n 0.9n cn

log10n 0.01n 0.09n 0.09n 0.81n

cn

log10/9n
2
≤cn

2 ≤cn
Total time: O(nlogn)
Chapter 9: Sorting 110
 Quick-Sort Summary
• Time
• Most of the work done in partitioning.
• Average case takes O(n log(n)) time.
• Worst case takes O(n2) time
Space
• Sorts in-place, i.e., does not require additional space

Chapter 9: Sorting 111

 Summary
• Divide and Conquer
• Merge-Sort
• Most of the work done in Merging
• O(n log(n)) time
• O(n) space

• Quick-Sort
• Most of the work done in partitioning
• Average case takes O(n log(n)) time
• Worst case takes O(n2) time
• O(1) space

Chapter 9: Sorting 112

 Testing the Sort Algorithms

• Need to use a variety of test cases

• Small and large arrays
• Arrays in random order
• Arrays that are already sorted and that are reversed
• Arrays with duplicate values
• Arrays with few unique values
• Arrays of integers and arrays of strings
• Compare performance on each type of array

Chapter 9: Sorting 113