Sorting Algorithms

Comparison Sorting/Dr.A.Sattar/1

Topics
ƒIntroduction to Sorting

ƒQuick Sort

ƒAnalysis of Quick Sort

ƒHeap Sort

ƒAnalysis of Heap Sort

ƒPriority Queues

Comparison Sorting/Dr.A.Sattar/2

1
 Sorting Algorithms
Classification
ƒ The sorting algorithms are classified into two categories on the basis of
underlying procedure used:

Sorting by comparison: These are based on comparison of keys . The method

has general applicability. Examples are selection sort, quick sort, etc.

Sorting by counting: These depend on the characteristics of individual data items.

The sort method has limited application. Examples include radix sort, bucket sort,
sort by counting
ƒThe algorithms are also categorized on the basis of storage type used to implement
the procedure

Internal Sorting : RAM is used for storage and processing. Execution speed is fast.
Used by operating systems, Real time systems

External Sorting: RAM and Secondary Storage are used. Comparatively slow execution.
Used by DBMS for large databases, Word Processors

Comparison Sorting/Dr.A.Sattar/3

Sorting Algorithms
Sorting by Comparison
ƒ The Sorting by Comparison method sorts input by comparing pairs of keys in the input.
Generally, the comparison sorting is not stable. ( An algorithm is called stable if it preserves
the order of sub-keys for any two equal keys. A stable algorithm, for example, sorts the salaries
of employees such that employees with equal salary appear in alphabetic order).
The comparison sorting are categorized as elementary and advanced.

Elementary Algorithms: The elementary algorithm are inefficient but easy to

implement.
Their running times are θ(n2).Common algorithms are
• Exchange sort
• Selection sort
• Insertion sort
• Shell sort

Advanced Algorithms: Advanced algorithms are efficient. The implementations are

usually based on recursive function calls. Their running times are θ(nlg n). The typical
advanced algorithms include
• Merge sort
• Quick sort
• Heap sort
Comparison Sorting/Dr.A.Sattar/4

2
 Quick Sort

Comparison Sorting/Dr.A.Sattar/5

Quick Sort
Procedure
ƒ The quick sorting procedure works in two phases, called partitioning and recursive calls

Phase #1: In this phase an array element is chosen as a pivot The array is partitioned into two
subarrays such that all elements in left partition are less than pivot, and all elements
in right partition are greater or equal to pivot. An illustration of partitioning is as follows:

pivot 51 67 11 78 90 13 66 80 92 30 40 59 68 71 65 23

23 11 13 30 40 51 67 78 90 66 80 92 59 68 71 65

Unsorted partition Unsorted partition

Phase #2: In Phase #2, the left and right subarrays are sorted recursively

11 13 23 30 40 51 59 65 66 67 68 71 78 80 90 92

Sorted partition Sorted partition

Comparison Sorting/Dr.A.Sattar/6

3
 Quick Sort
Pivot Selection
Choice of pivot influences the performance of quick sort algorithm Some of the choices are:
ƒLeft-most element: The first element is chosen as pivot. There is chance of bad partitioning
1 2 3 4 5 6 7 8 9 10
pivot=56
56 90 82 88 12 50 89 22 77 25

ƒRight-most element: The last element is chosen as pivot There is chance of bad partitioning.
1 2 3 4 5 6 7 8 9 10
56 90 82 88 12 50 89 22 77 25 pivot=25

ƒMedian element: An element with index that is median of left-most and right-most element
index. If l and r are the indexes of first and last element, median index m is computed as
m=└ (l + r) / 2 ┘ = └(1+10)/2 ┘=5
1 2 3 4 5 6 7 8 9 10
56 90 82 88 12 50 89 22 77 25
pivot=12

The choice of median minimizes the chances of bad partitioning.

ƒRandom element: An element is selected randomly. A random number generator is used to compute
the index of element being chosen .For example, if the function random (l,r) returns a random number
In the range l to r, the index i of the selected element is given by
i = random( l ,r ) = (1,10)= 7, say
1 2 3 4 5 6 7 8 9 10
56 90 82 88 12 50 89 22 77 25
pivot=89

The random selection minimizes the chances of bad partitioning

Comparison Sorting/Dr.A.Sattar/7

Quick Sort
Hoare’s Partitioning Algorithm
ƒ A common partitioning procedure, due originally to Hoare , is as follows

Step #1: Select left most element as pivot Define two variables i,j to scan the array

Step #2: Use variable i as left pointer, and j as right pointer to scan the array

Step #3; Scan array from right-to-left, by decreasing j, until an element smaller than
pivot is encountered

Step#4: Scan array from left-to-right, by increasing i, until an element larger than
pivot is encountered

Step#5: Exchange the elements pointed by i and j.

Step #6: Move pointers forward ( increase i by 1, and decrease j by 1)

Step #7 Repeat Step#3 through Step#6 until pointers cross-over

Step#8: Exchange pivot with element identified by the left pointer

Comparison Sorting/Dr.A.Sattar/8

4
 Quick Sort Partitioning
Example
34 61 30 65 33 59 18 10 15 45

ƒThe procedure for partitioning of the above array is as follows:

ƒ Left-most element 34 is chosen as pivot. Left pointer is set to point at element 61 and right pointer at 45

34 61 30 65 33 59 18 10 15 45

ƒScanning starts from right to left. Since element 45 is larger than pivot, the right pointer is moved to next
left element 15.

34 61 30 65 33 59 18 10 15 45

ƒSince 15 is smaller than pivot, right-to-left scanning is stopped, and left-to-right scanning is started .

34 61 30 65 33 59 18 10 15 45

ƒSince 61is larger than pivot, left-to-right scanning is stopped, and 61 and 15 are exchanged .

34 15 30 65 33 59 18 10 61 45
Comparison Sorting/Dr.A.Sattar/9

Quick Sort Partitioning Cont’d-1

Example
Left and right pointers are moved to next location.. .

34 15 30 65 33 59 18 10 61 45

The scanning is started from right to left . .

34 15 30 65 33 59 18 10 61 45

Since 10 is smaller than pivot, the right-to-left scanning is stopped and left-to-right scanning is started . .

34 15 30 65 33 59 18 10 61 45

Since 30 is smaller than pivot, the left pointer is moved forward

34 15 30 65 33 59 18 10 61 45

Comparison Sorting/Dr.A.Sattar/10

5
 Quick Sort Partitioning Cont’d-2

Example
ƒSince 65 is greater than pivot, the left-to-right scanning is stopped. . . .

34 15 30 65 33 59 18 10 61 45

ƒThe elements 65 and 10 are exchanged. . . .

34 15 30 10 33 59 18 65 61 45

ƒAfter swapping, the pointers are moved forward. . . .

34 15 30 10 33 59 18 65 61 45

ƒThe scanning is started from right to left, with pointer at element 18. . . . .

34 15 30 10 33 59 18 65 61 45

Comparison Sorting/Dr.A.Sattar/11

Quick Sort Partitioning Cont’d-3

Example
ƒSince 18 is smaller than pivot, the right to left scanning is stopped

34 15 30 10 33 59 18 65 61 45

ƒScanning is started from left to right

34 15 30 10 33 59 18 65 61 45

ƒSince 33 is smaller than pivot, the left pointer is moved forward

34 15 30 10 33 59 18 65 61 45

ƒSince 59 is larger than than pivot, the left-to-right scanning is stopped. The elements
59 and 18 are exchanged.

34 15 30 10 33 18 59 65 61 45

Comparison Sorting/Dr.A.Sattar/12

6
 Quick Sort Partitioning Cont’d-4

Example
ƒBoth the pointers are moved forward.

34 15 30 10 33 18 59 65 61 45

ƒThe pointers now cross-over. The scanning is stopped. Pivot is exchanged with element 18

18 15 30 10 33 34 59 65 61 45

ƒThe array is now properly partitioned. The pivot 34 separates the two subarrays. The left partition
contains elements { 18,15,30,10,33} which are all smaller than 34. The right partition consists of
elements {59,65,61,45}, each being larger than 34

18 15 30 10 33 34 59 65 61 45

left partition right partition

Comparison Sorting/Dr.A.Sattar/13

Quick Sort
Partitioning Code
ƒ Quick sorting is implemented using two procedures: PARTITION and QUICKSORT

ƒThe PARTITION uses two counters to scan the array from left-to-right and then from
right-to-left The first element is chosen as the pivot

PARTITION( A, p, r)
1 x ← A[p] ► left-most element is chosen as pivot. It is stored in x
2 i ← p-1 ► i scans the array from left-to-right
3 j ←r+1 ►j scans the array from right-to-left
4 while TRUE do ►An infinite loop is set up
5 repeat j ← j-1
6 until A[j] ≤ x
7 repeat i ←i+1
8 until A[i] ≥ x
9 if i < j ► loop terminates when pointers cross-over
10 then exchange A[i]↔A[j] ► Exchange elements identified by i and j
11 else return j ► Return index of pivot

Comparison Sorting/Dr.A.Sattar/14

7
 Quick Sort
Sorting Code
The QUICKSORT procedure is recursive. It is used to sort left and right partitions .

QUICKSORT(A, p ,r)
1 if p < r ► Recursion is terminated when boundaries cross-over
2 then q ← PARTITION(A, p, r ) ► The PARTITION method returns index of pivot
3 QUICKSORT(A, p, q-1) ►sort left partition recursively
4 QUICKSORT(A, q+1, r) ►sort right partition recursively

Demonstration

Comparison Sorting/Dr.A.Sattar/15

Demonstration-CD

Comparison Sorting/Dr.A.Sattar/16

8
 Analysis of Quick Sort

Comparison Sorting/Dr.A.Sattar/17

Quick Sort Analysis

Best Case Recurrence
ƒ In best case, the array is partitioned in to two nearly equal subarrays

A[1] A[2] - - A[k-1] A[k] A[k+1] - - A[n-1] A[n]

Left partition Right partition

ƒThe recurrence relation for best running time is

T(n) = 2 T( n / 2) + n-1

Time to sort Time to sort

two subarrays Cost of partitioning
array of size n
each of size n/2 subarray of size n-1

Comparison Sorting/Dr.A.Sattar/18

9
 Quick Sort Analysis
Best Case Solution
ƒ The recurrence
T(n) = 2 T( n / 2) + n - 1
can be solved by iteration-substitution method

ƒ After kth iteration:

T(n) = 2k T( n / 2k ) + k.n - ( 20 + 21+ 22+…….. + 2k-1)

ƒ Summing the series

T(n) = 2k T( n / 2k ) + k.n -( 2k - 1)

ƒ Setting n/2k =1, gives k=lg n

T(n) = 2lg n. T(1 ) + n. lg n -( 2lg n - 1)

= n + n lg n – ( n - 1)

= n lg n + 1= θ( n lg n)

ƒ The best case running time of quick sort is θ( n lg n)

Comparison Sorting/Dr.A.Sattar/19

Quick Sort Analysis

Worst Case : Example(1)
ƒThe worst case arises when the input is presorted. In this case array is scanned from right to left, until the
pointers crossover. The partitioning results in a single right partition The left partition is empty.

Presorted Array 10 25 33 45 58 78

78 larger than pivot 10 25 33 45 58 78

58 larger than pivot 10 25 33 45 58 78

45 larger than pivot 10 25 33 45 58 78

10 25 33 45 58 78
33 larger than pivot

Pointers crossover 10 25 33 45 58 78

10 25 33 45 58 78

pivot right partition

Comparison Sorting/Dr.A.Sattar/20

10
 Quick Sort Analysis
Worst Case: Example(2)
The worst case also arises when input is reverse sorted. In this case the array is scanned from left to right only
until pointers crossover. The partitioning results in a single left partition. The right partition is empty
Reverse sorted 90 85 73 66 45 33

Scan from right . 33 larger 90 85 73 66 45 33

than pivot. Stop right scan

Scan from left. 73 smaller 90 85 73 66 45 33

than pivot

90 85 73 66 45 33
Scan from left. 66 smaller
than pivot
90 85 73 66 45 33
Scan from left. 44 smaller
than pivot
90 85 73 66 45 33
Pointers crossover

Exchange pivot 33 85 73 66 45 99

left partition pivot

Comparison Sorting/Dr.A.Sattar/21

Quick Sort Analysis

Worst Case Recurrence
Consider the case when left partition is empty

A[1] A[2] A[3] - - A[n-2] A[n-1] A[n]

pivot right partition of size n - 1

ƒ The recurrence relation for worst running time is
T(n) = T(0) + T( n -1) + n-1

Time to sort Time to sort Time to sort Cost of partitioning

array of size n left partition right partition subarray of size n-1
T(0)=0 of size n-1
ƒ Since left partition is empty, recurrence simplifies to:

T(n) = T( n -1) + n – 1, n>1

ƒ The same recurrence applies when right partition is empty.

Comparison Sorting/Dr.A.Sattar/22

11
 Quick Sort Analysis
Worst Case Solution
ƒ The recurrence

T(n) = T( n -1) + n – 1, n >1

T(1) =0

can be solved by iteration method

ƒ Iteration yields the solution:

T(n) = n-1 + (n-2) + (n-3) + +……..+ 2 + 1

= n (n -1) / 2 =O(n2)

ƒ The worst case running time of quick sort is θ(n2)

Comparison Sorting/Dr.A.Sattar/23

Quick Sort Analysis

Average Case
In order to do probabilistic analysis we assume that the pivot can be in any of the n array cells, probability of being in any cell being 1/n.
The table summarizes the running time for different possibilities and associated costs.
pivot
Pivot cell Probability Sort time
T(0)+T(n-1) A[1] A[2] A[3] - - - - A[n-2] A[n-1] A[n]
ist cell 1/n
(n-1) keys

2nd cell 1/n T(1)+T(n-2) A[1] A[2] A[3] - - - - A[n-2] A[n-1] A[n]
1 key (n-2 )keys

3rd cell 1/n T(2)+T(n-3) A[1] A[2] A[3] - - - - A[n-2] A[n-1] A[n]
2 keys (n-3) keys
4th cell 1/n T(3)+T(n-4) A[1] A[2] A[3] A[4] - - - A[n-2] A[n-1] A[n]
3 keys (n-4) keys

n-2th cell 1/n T(n-3)+T(2) A[1] A[2] A[3] A[4] - - - A[n-2] A[n-1] A[n]
(n-3 )keys 2 keys

n-1th cell 1/n T(n-2)+T(1) A[1] A[2] A[3] A[4] - - - A[n-2] A[n-1] A[n]
(n-2 )keys 1 key

nth cell 1/n T(n-1)+T(0) A[1] A[2] A[3] A[4] - - - A[n-2] A[n-1] A[n]

(n-1) keys
Comparison Sorting/Dr.A.Sattar/24

12
 Quick Sort Analysis
Average Case Recurrence

A[1] A[2] A[3] - A[k-1] A[k] A[k+1] - A[n-1] A[n]

Left partition of size k-1 Pivot right partition of size n- k

ƒBy summing over running times for all probable locations of pivot and associated costs,
the expected running time, T(n), can be expressed as the recurrence:

∑ —n
1
T(n) = n–1 + [ T( k - 1) + T(n – k) ]
k =1

Average time Cost of partitioning Time to sort

to sort array Probability Time to sort subarray
subarrays of size n-1 subarray
of size n of pivot of size n - k
being in of size k-1
cell k

Comparison Sorting/Dr.A.Sattar/25

Quick Sort Analysis

Average Case Solution
n
1
ƒ Consider the recurrence: T(n) = n – 1 + ∑—n [ T( k - 1)
k =1
+ T(n – k) ]
ƒ Expanding the summation
n

∑T(n – k) = T(n-1)+T(n-2)+…..+T(2) + T(1) + T(0)

k =1
= T(0)+T(1)+T(2)+…….. + T(n-2) + T(n-1) ( Rearranging in reverse order )
n
= ∑T(k-1)
k =1
……………(1)

ƒ The recurrence can be rewritten as:

n
T(n) = n – 1 +
2
—
n ∑T( k - 1)
k =1
……………(2)

ƒ Multiplying both sides of (2) with n

n
n.T(n) = n(n – 1) + 2 ∑ T( k - 1)
k =1
……………(3)

ƒ Substituting n-1 for n in the recurrence (3)

n −1

(n-1).T(n-1) = (n-1)(n – 2) + 2 ∑ T( k - 1)
k =1
……………(4)
Comparison Sorting/Dr.A.Sattar/26

13
 Quick Sort Analysis Cont’d-1

n
Average Case Solution
n.T(n) = n(n – 1) + 2 ∑ T( k - 1)
k =1
……………(5)

n −1
(n-1).T(n-1) = (n-1)(n – 2) + 2 ∑ T( k - 1)
k =1
……………(6)

ƒ Subtracting (6) from (5), and rearranging

nT(n) - (n-1)T(n-1) = 2(n-1) + 2T(n-1) ……………(7)

ƒ Simplifying and rearranging

T(n) T(n-1) 2(n-1)
———— = ——— + ———— ……………(8)
n+1 n n(n+1)

ƒ Setting S(n)
T(n)
S(n) = ———
n+1

gives the new recurrence

2(n - 1) ……………(9)
S(n) = S(n-1)+ ——— , S(0)=0
n(n+1)
Comparison Sorting/Dr.A.Sattar/27

Quick Sort Analysis Cont’d-2

Average Case Solution

ƒThe following recurrence is solved by iteration method.
2(n - 1)
S(n) = S(n-1)+ ——— , S(0)=0 …………..(10)
n(n+1)
ƒ The solution is as follows
n n

∑ ——— ∑
2( i - 1) 2(i+1)-4
S(n) = =
i (i +1)
i =1 i =1 i(i +1)
n n

=
i
∑ -
i =1
2 4
i(i+1)
∑ n
…………..(11)
i =1
ƒPerforming summation of harmonic series ,
n
∑
i =1
1
— = ln( n )+ 0.386
i
…………..(12)
n
∑ i(i+1)
1 n
=∑
1
ƒ Also,
i =1 i =1 i
- ∑
i =1
1
(i+1)
= 1- 1/(n+1)=n/(n+1) …………..(13)

ƒ Substituting results (12) and (13) into summation (11), we get

…………..(14)
ƒ S(n) = 2( ln (n) + 0.386 ) – 4 n / (n+1)

ƒ T(n) = 1.386 n lg n – 2.846 n (converting ln n to lg n )

ƒThe Average running time of quick sort is θ(n lg n)
Comparison Sorting/Dr.A.Sattar/28

14
 Heap Sorting

Comparison Sorting/Dr.A.Sattar/29

Binary Heaps
Properties
ƒA binary heap, simply referred to as heap, is a binary tree with completeness and order
properties.:
Completeness Property: The completeness property implies that the tree has nodes at all levels,
except possibly at the lowest level, such the last level is filled from left to right up to some point

Order Property: The order property implies that key at each node is larger ( smaller ) than
or equal to the keys at child nodes.

ƒ A heap is known as max heap if key at each node is larger than or equal to the keys
stored at the child nodes. It is called min heap if key at each node is smaller than or equal
to the keys at child nodes

ƒ In a binary heap, the left subtree and right subtree are also heaps

Heap Heap

Comparison Sorting/Dr.A.Sattar/30

15
 Binary Trees and Heaps
Examples
ƒ Figure (a ) shows a max-heap ;key at each node being larger than keys at child nodes
ƒ Figure (b) shows a binary tree which is not a heap, because the tree is not complete; the bottom
level nodes are not in left most locations
ƒ Figure (c) shows a min-heap; key at each node being smaller than keys at the child nodes
ƒ Figure (d) Shows a binary tree with order property. However, it is not a heap because it
lacks completeness property

(a) max-heap (c) min-heap

(b) Binary tree – not a heap (d) Binary tree – not a heap
Comparison Sorting/Dr.A.Sattar/31

Binary Heap
Height
Theorem: The height of a heap with n nodes is └ lg(n) ┘
Proof : :Let h be the height of the heap. Figures (a) and (b) show heap structures with maximum and
minimum number of nodes in a heap of same height.

h h

(a) Maximum number of nodes in heap of height h (b) Minimum number of nodes in heap of height h

•The tree contains 20 21

nodes at level 0, nodes at level 1…2k
nodes at level k
•In case (a) of maximum nodes, the total number of nodes is given by
20 + 21+22+…………+2h = 2h+1-1
•In case (b) of minimum nodes, the total number of nodes is
20 + 21+ 22+………..+2h-1 + 1 = 2h
. Up to height h-1 Bottom level

•Since n is the number of nodes in the heap, n can equal maximum, minimum or lie somewhere in-between
Therefore, 2h ≤ n ≤ 2 h+1-1
Or, 2h ≤ n < 2h+1 (dropping -1)
Taking logarithm to base 2,
h ≤ lg(n) < h+1 ( For a floor, x≤ └ x ┘ <x+1)
• It follows, using property of the floor function, that
h=└ lg(n) ┘ Comparison Sorting/Dr.A.Sattar/32

16
 Binary Heaps
Array Representation
ƒ A heap can be implemented as an array. The array is filled with heap keys, starting with
the key at the root, and moving from top-to-bottom and left-to-right across the binary tree.

ƒ The array representation of heap in figure (a) is shown in figure (b). The numbers next to
nodes indicate the associated array indexes
1

2 3

4 5 6 7

8 9 10 11 12 13 14 15

(a) Binary Heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
76 75 73 38 56 17 54 7 24 32 31 2 15 3 29 - - -

root Children of 76 Children of 75 Children of 73 Children of 38 Children of 56 Children of 17 Children of 54

(b) Array representation of Heap

Comparison Sorting/Dr.A.Sattar/33

Binary Heap
Basic Operations
ƒThe basic operations on a heap data structure, implemented as an array, are:

PARENT(i)- returns index of parent of the node identified by the index i

LEFT(i)- returns index of left child of the node identified by index i

RIGHT(i)- returns index of right child of the node identified by index i

BUILD-HEAP(A)- converts an unordered array A into a heap

HEAPIFY(A,i)- Restores heap order property by adjusting the key stored in a node
with index i

Comparison Sorting/Dr.A.Sattar/34

17
 Basic Operations
Pseudo Code
ƒ The pseudo code for PARENT, LEFT and RIGHT procedures is as follows.

PARENT( i ) ► returns index of parent node of an array element with index i

1 k←└ i / 2 ┘
2 return k

LEFT( i ) ►returns index of left child of an element with index i

1 k ← 2i
2 return k

RIGHT( i ) ►returns index of right child of an element with index i

1 k ←2i + 1
2 return k

ƒ The procedures PARENT, LEFT, and RIGHT require fixed amount of time to do the computations
on array indexes. Thus, running time for each of these operations is θ(1)

Comparison Sorting/Dr.A.Sattar/35

Fixing Up Heap
Heapify Procedure
ƒThe heapify is procedure is used to fix up a node that violates the order property.

ƒThe following is the heapify algorithm:

Step #1: Compare the keys of left and right children of the violating node .
Select the larger of keys.

Step #2: Compare the larger key with the key held by parent node.

Step 3#: The keys of the parent node is exchanged with larger of the keys

Step #4: If the swapped key violates the heap property, repeat ,
steps #2 through step #4, until the heap order property is restored

ƒ In essence, offending key is trickled down until the heap order property is restored.

Comparison Sorting/Dr.A.Sattar/36

18
 Fixing Up Heap
Example
ƒ The key ,1, at the root violates the heap order property; it is smaller than both the keys, 87
and 74, at child nodes. In order to fix up the heap, the key 1 is exchanged with the larger of
the keys, i.e 87

Comparison Sorting/Dr.A.Sattar/37

Fixing Up Heap Cont’d-1

Example
ƒ The key, 1, in new location again violates the heap order property; it is smaller
than both the keys 79 and 75 at child nodes In order to fix the heap, the key 1 is exchanged
with the larger of the keys in the child nodes i.e 79

Comparison Sorting/Dr.A.Sattar/38

19
 Fixing Up Heap Cont’d-2

Example
ƒ The key, 1, in new location again violates the heap order property; it is smaller
than both the keys , 38 and 76, at the child nodes. In order to fix the heap, the key, 1, is
exchanged with 76, which is the larger of the two keys at child nodes

Comparison Sorting/Dr.A.Sattar/39

Fixing Up Heap Cont’d-3

Example
ƒ The key, 1, in new location is smaller than both the keys , 22 and 11, at the child nodes
The key, 1, is exchanged with 24, which is the larger of two keys stored at the child nodes
The heap is now fixed up

Comparison Sorting/Dr.A.Sattar/40

20
 Heapify Algorithm
Pseudo Code
ƒThe pseudo code for HEAPIFY(fixing up) algorithm is given below. The heap is assumed to be
implemented as an array. The array, A , and index, i, of the offending element are passed on
to the heapify procedure.

HEAPIFY(A, i)
1 l ← LEFT(i) ► l stores the index of left child of parent identified by index i
2 r ← RIGHT(i) ► r stores the index of right child of parent identified by index i
3 if l≤ HeapSize[A] and A[l] > A[i] ►checks if key at left child is larger than key at parent node
4 then largest ← l ►if so, it is saved in variable largest
5 else largest ← i ► else, key at right child is stored in largest
6 if r ≤ HeapSize[A] and A[r] > A[largest] ► larger of the keys at left and right nodes is chosen
7 then largest ← r
8 if largest ≠ i
9 then exchange A[i] ↔ A[largest] ►elements pointed by index largest and index i are exchanged
10 HEAPIFY(A,largest) ► HEAPIFY procedure is applied recursively to the key
in new location
11 return A ► Heap is fixed up

Comparison Sorting/Dr.A.Sattar/41

Analysis of Heap Fixing

Worst Case Scenario
ƒ In worst case the offending key appears at the root.

ƒTo fix up the heap the key is moved down to the leaf level. At each level two comparisons are made:
among the key at the parent node and two keys at child nodes

ƒIf h is the height of heap and c the unit cost of comparison , the running time T(n) is given by,
T(n) = 2ch
= 2c└ lg(n)┘
ƒ Since for large n, └ lg(n)┘ = lg( n), the worst case running time of fixing up the heap is θ(lg n)

Comparison Sorting/Dr.A.Sattar/42

21
 Building Heap
Algorithm
ƒAn array holding keys in a random order is converted to a heap using heap building algorithm.
ƒ The build-up procedure can be bottom-up or top-down. The former approach starts with the
key held by the parent of the last element in the array . The latter approach starts with the key
at the root.

ƒ The bottom up algorithm, as illustrated in the diagram below, is as follows:

Step # 1: Select the key at parent of the last node in the tree
Step #2 : Heapify the array using the chosen key.
Step #3: Choose the next adjacent key on the left at the same level or the key at the extreme
right node on next higher level. Heapify the array using the selected key.
Step #4: Repeat Step # 3, until the root is reached.

End procedure

Start build procedure

Bottom up heap building

Comparison Sorting/Dr.A.Sattar/43

Building Heap
Pseudo Code
ƒ The BUILD-HEAP method converts binary tree , represented by array A[1..n], into a heap
ƒ The last element in the array has index n. The parent of last element has index └ n/2 ┘ Thus, the bottom
up heapify procedure starts with key A[└ n/2┘ ] and continues through keys A[└n/2┘-1] ,
A[└n/2┘-2]…A[1]

A[1]

A[└ n/2┘]

A[n]

ƒThe pseudo code for heap-building algorithm is as follows:

BUILD-HEAP(A) ► A is the input array to be converted to heap
1 n← length[A] ► n stores the number of elements in the array A
2 for i←└ n/2 ┘ downto 1 ►Scanning begins on a level above the leaves
3 do HEAPIFY(A, i) ► HEAPIFY fixes up the node identified by index i
Comparison Sorting/Dr.A.Sattar/44

22
 Building Heap
Example
ƒ The input array consists of keys {2,3,4,5,6,7,8,9,10,11,12,13,14,15,16}.The tree
representation is shown in diagram (a)
ƒ The heap build-up procedure is illustrated in the following diagrams.

(a) Binary tree for the input array. (b) Key 8 is fixed up by exchanging with 16, which is
larger of the two keys at child nodes

(c) Key 7 is considered (d) Key 7 is fixed up by exchanging with larger key 14
at child node
Comparison Sorting/Dr.A.Sattar/45

Building Heap Cont’d-1

Example

(e) Key 6 is considered

(f) Key 6 is fixed up by exchanging with 12

(h) Key 5 is fixed up by exchanging with 10

(g) Key 5 is examined

Comparison Sorting/Dr.A.Sattar/46

23
 Building Heap Cont’d-2

Example

(i) Key 4 is considered. It is exchanged with key 16 at child node. Since 4 is smaller
than key 15 at child node it is exchanged with 15.

Comparison Sorting/Dr.A.Sattar/47

Building Heap Cont’d-3

Example

(i) Key 3 is considered. It is exchanged with key 12 at child node. Since 3 is smaller
than key 11 at child node it is exchanged with 11.

Comparison Sorting/Dr.A.Sattar/48

24
 Building Heap Cont’d-4

Example

(j) Finally, Key 2 is considered. It is exchanged with key 16 at child node. Since 2 is smaller
than key 15 at child node it is exchanged with 15. Key is still smaller than key at child.
Key 2 is exchanged with 4. The binary tree is now converted into a heap.

Comparison Sorting/Dr.A.Sattar/49

Analysis of Heap Building

Worst Case Recurrence
ƒIt is assumed that binary tree has n nodes, which is power of 2, and d is tree depth

d=lg( n)

ƒ In heapifying operation two comparisons are performed among the parent and child nodes
ƒ In worst case all the keys would be processed and shifted down to leaf level
ƒ The table summarizes the number of comparisons on different levels
Depth Number of nodes Number of comparisons
0 20 2(d-1)
1 21 2(d-2)
2 22 2(d-3)
-- -- --
k 2k 2(d-k-1)
--- --- --
d-1 2d-1 0

d −1
ƒ Total number of comparisons, T(n) = 2 ∑ 2 k (d - k- 1) + 2
k =0

Comparison Sorting/Dr.A.Sattar/50

25
 Analysis of Heap Building
Worst Case Solution
d −1
ƒ Tbuild = 2 ∑ 2 k (d - k- 1) + 2 , where d = lg( n)
k =0
d −1 d −1
= 2(d -1) ∑ 2 k - 2 ∑ k.2 k + 2
k =0 k =0

d −1

∑
k =0
2 k = 2d -1 (sum of the geometric series)
d −1

∑
k =0
k.2 k = (d-2)2d + 2 (sum of the arithmetic-geometric series) ,

ƒ Tbuild = 2(2d - 1)
= 2( 2lg n -1)
=2(n-1)
= θ(n)

ƒThe worst case heap building procedure runs in θ(n) time

Comparison Sorting/Dr.A.Sattar/51

Heap Sorting
Algorithm
The heap sort algorithm is based on the fact that in a max-heap, the root holds the largest
key. The algorithm works as follows
Step#1: Convert the array into a heap

Step #2: Exchange last element with the first element

Step #3: Consider subarray consisting of all elements, except those

at the end ,which already are exchanged with first element. Covert the
subarray into heap.

Step #4 Exchange last element of subarray with the first element of subarray.

Step #5: Repeat Step#3 and Step #4 until the subarray reduces to a single element

Comparison Sorting/Dr.A.Sattar/52

26
 Heap Sorting Example
ƒThe steps involved in heap sorting are illustrated by the following diagrams.

Step #1:Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

7 56 17 76 32 96 31 38 24 87 75 2 15 73 54 96 87 73 76 75 17 54 38 24 32 56 2 15 31 7

Unsorted array Heaped array

Step #2:Exchanging key 7 at leaf with key 96 at root

Step #3:Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
87 76 73 38 75 17 54 7 24 32 56 2 15 31 96 7 87 73 76 75 17 54 38 24 32 56 2 15 31 96

Heaped array Unsorted array

Comparison Sorting/Dr.A.Sattar/53

Heap Sorting Example Cont’d-1

Step#4: Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
31 76 73 38 75 17 54 7 24 32 56 2 15 87 96
76 75 73 38 56 17 54 7 24 32 56 2 15 87 96

Unsorted Sorted Heaped Sorted

Step #5:Exchanging key 15 at leaf with key 76 at root

Step #6: Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
75 56 73 38 32 17 54 7 24 15 31 2 76 87 96
15 75 73 38 56 17 54 7 24 32 56 2 76 87 96

Heaped Sorted Sorted

Unsorted

Comparison Sorting/Dr.A.Sattar/54

27
 Cont’d-2
Heap Sorting Example

Step #7:Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
2 56 73 38 32 17 54 7 24 15 31 75 76 87 96 73 56 54 38 32 17 2 7 24 15 31 75 76 87 96

Unsorted Sorted Heaped Sorted

Step # 8:Exchanging key 31 at leaf with key 73 at root

Step # 9: Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
56 38 54 31 32 17 2 7 24 15 73 75 76 87 96 31 56 54 38 32 17 2 7 24 15 73 75 76 87 96

Heaped Sorted Unsorted Sorted

Comparison Sorting/Dr.A.Sattar/55

Cont’d-3
Heap Sorting Example

Step #10:Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
54 38 17 31 32 15 2 7 24 56 73 75 76 87 96
15 38 54 31 32 17 2 7 24 56 73 75 76 87 96

Heaped Sorted
Unsorted Sorted
Step # 11:Exchanging key 24 at leaf with key 54 at root

Step #12:Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
24 38 17 31 32 15 2 7 54 56 73 75 76 87 96
38 32 17 31 24 15 2 7 54 56 73 75 76 87 96

Heaped Sorted Unsorted Sorted

Comparison Sorting/Dr.A.Sattar/56

28
 Cont’d-4
Heap Sorting Example

Step #13:Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 32 31 17 7 24 15 2 38 54 56 73 75 76 87 96
7 32 17 31 24 15 2 38 54 56 73 75 76 87 96
Heaped Sorted
Unsorted Sorted Step # 14:Exchanging key 2 at leaf with key 32 at root

Step #15:Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 2 31 17 7 24 15 32 38 54 56 73 75 76 87 96
31 24 17 7 2 15 32 38 54 56 73 75 76 87 96

Unsorted Sorted
Heaped Sorted

Comparison Sorting/Dr.A.Sattar/57

Cont’d-5
Heap Sorting Example
Step #16:Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
15 24 17 7 2 31 32 38 54 56 73 75 76 87 96 24 15 17 7 2 31 32 38 54 56 73 75 76 87 96

Unsorted Sorted Heaped Sorted

Step # 17:Exchanging key 2 at leaf with key 24 at root

Step #18:Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

17 15 2 7 24 31 32 38 54 56 73 75 76 87 96 2 15 17 7 24 31 32 38 54 56 73 75 76 87 96

Heaped Sorted Unsorted Sorted

Comparison Sorting/Dr.A.Sattar/58

29
 Cont’d-6
Heap Sorting Example

Step #19:Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 15 7 2 17 24 31 32 38 54 56 73 75 76 87 96
7 15 2 17 24 31 32 38 54 56 73 75 76 87 96

Heaped Sorted
Unsorted Sorted
Step # 20:Exchanging key 2 at leaf with key 15 at root

Step #21:Converting to heap

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

7 2 15 17 24 31 32 38 54 56 73 75 76 87 96
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

2 7 15 17 24 31 32 38 54 56 73 75 76 87 96
Heaped Sorted
Sorted

Comparison Sorting/Dr.A.Sattar/59

Cont’d-7
Heap Sorting Example
Step # 22:Exchanging key 2 at leaf with key 7 at root

root

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

2 7 15 17 24 31 32 38 54 56 73 75 76 87 96

Sorted

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

2 7 15 17 24 31 32 38 54 56 73 75 76 87 96

Sorted

Comparison Sorting/Dr.A.Sattar/60

30
 Heap Sorting
Pseudo Code
The HEAPSORT method sorts an input array A[1..n] into ascending order by using order property of
max heap. It uses two other heap procedures ,BUILD-HEAP and HEAPIFY. The former converts the
array into a max heap. The latter fixes the heap. In each cycle the key at the root is exchanged with
the last element of the subarary

HEAPSORT( A)
1 BUILD-HEAP(A) ► Call BUILD-HEAP procedure to convert array into a heap
2 for j ← length[A] downto 2 ► Use bottom-up procedure
3 do exchange A[1] ↔ A[j] ► Exchange last element in the sub-array with the largest key
4 heap-size ← heap-size – 1 ► Reduce the size of array (heap) by one
5 HEAPIFY(A, j ) ► Fix up the key at root by calling HEAPIFY procedure

Demonstration

Comparison Sorting/Dr.A.Sattar/61

Heap Sort Demo CD

Comparison Sorting/Dr.A.Sattar/62

31
 Analysis of Heap Sorting
Worst Case Recurrence
The heap sorting algorithm uses two procedures : BUILD-HEAP and HEAPIFY
Thus, total T(n) to sort an array of size n is
T(n) = Tb(n) + Th(n)

Time to Time to Time to

Sort array build heap fix up heap

ƒ It was shown earlier that, in worst case, the BUILD-HEAP procedure takes θ(n) time.
Thus,
Tb= θ(n)

ƒThe HEAPIFY procedure is applied repeatedly to sub-arrays of sizes n-1, n-2, ….2, 1
It was shown earlier that fixing up a heap of size n takes 2lg( n) time. Thus, total time for
fixing up the heap is give by:
n −1
Th(n) = 2( lg(n-1) + lg(n-2)+…+lg(2).+lg(1) ) = ∑
k =1
2 lg(k)

ƒ The logarithmic series can be evaluated approximately by converting the sum into an integral

Comparison Sorting/Dr.A.Sattar/63

Analysis of Heap Sorting

Worst Case Solution
n −1
Th(n) = 2( lg(n-1) + lg(n-2)+…+lg(2).+lg(1) ) = 2 ∑ lg(k)
k =1
ƒ Converting sum into integral :
n −1 n

2 ∑
k =1
2 lg(k) ≈ 2 ∫ lg( e) ln x dx ( integrating by parts )
1
= 2(lg e) (n ln n –n )

= 2(n lg n ) - 1.443n (converting to binary logarithm )

Th(n) = 2( n lg n) - 1.443 n = θ(nlg n)

T(n) = Tb(n) + Th(n) = θ(n) +θ( n lg n ) = θ( n lg n)

ƒThe worst case running time of heap sort is θ (nlg n)

Comparison Sorting/Dr.A.Sattar/64

32
 Priority Queues

Comparison Sorting/Dr.A.Sattar/65

Priority Queues
Definition
A priority queue is data collection in which each item has an associated
priority, called key. Items with highest priority are retrieved first from the queue
If two items with equal priorities exist, the one which was inserted earlier is retrieved first.

ƒ The Figure shows an example of priority queue. The priorities are denoted by a set of
integers. Data is a set of characters. In this example, characters A and X would be retrieved
before other characters.

data A X Y C Z H I D
priorities 10 10 1 3 2 9 7 3

Example of a priority queue

ƒ A priority queue can be naturally implemented as heap, in which the keys are
the queue priorities..

Comparison Sorting/Dr.A.Sattar/66

33
 Priority Queues
Heap Representation
The heap representation of a priority queue is shown in the figure.

Data A X Y C Z H K P Q R S T V D
Priorities 10 10 1 3 2 9 7 4 6 5 8 4 2 3

(a) Priority queue

A
10

X
H
10
9

S
K Q R
8
7 6 5

T P C D Z V Y
4 4 3 3 2 2 1

(b) Heap representation

Comparison Sorting/Dr.A.Sattar/67

Priority Queues
Operations
The major operations on a priority queue are:

ƒINSERT (x) – inserts a new key x into the priority queue

ƒEXTRACT- returns and removes element with the highest key in the queue

ƒMAXIMUM- returns element with highest key without removing it from the queue

These operations can be easily implemented using a heap.

Comparison Sorting/Dr.A.Sattar/68

34
 Priority Queues Insertion
Algorithm
The insertion algorithm adds an element to a priority queue. It proceeds as follows: :

Step # 1: Insert the new element at the last position in the heap

Step #2: Increase the heap size by one

Step #3: Use HEAPFY operation to fix the new element.

Comparison Sorting/Dr.A.Sattar/69

Priority Queues
Insertions
ƒA new data element ‘L’ with priority 12 is inserted. The insertion procedure is illustrated by figures (a)-(d)

(a) The element is inserted at bottom level

The heap order property is violated (b) The key 12 being larger, it is swapped
with the key 5 at the parent node

(c) The key 12 being larger, it is swapped

with the key 9 at the parent node (d) Finally, new key is moved to root. The heap
order property is restored
Comparison Sorting/Dr.A.Sattar/70

35
 Priority Queues
Insertion Code
The INSERT Method inserts a new key into priority queue, and restores the heap property.
The pseudo code is as follow

INSERT(A,)
1 heap-size[A]←heap-size[A] + 1 ►Increase heap size to accommodate new key
2 j← heap-size[A] ►j stores index of last element in the heap
3 while j >1 and A[PAREN(j)]< key ►Compare keys of parent and child nodes
4 do A[j] ←A[PARENT(j)] ►Exchange keys
5 j←PARENT(j)
6 A[j]←key ► insert key

ƒIn worst case, the new element is moved from bottom level to the root node. Since,
The height of heap with n elements is lg(n), the worst case running time for the
insertion operation is θ( lg n)

Comparison Sorting/Dr.A.Sattar/71

Priority Queues
Extraction Algorithm
ƒ The extraction algorithm removes an element with highest priority. It proceeds as follows: :

Step # 1: Remove element at the root

Step #2: Move element in last position to the root

Step #3: Decrease the heap size by one

Step #4: Use HEAPFY operation to fix the element at the root.

Comparison Sorting/Dr.A.Sattar/72

36
 Priority Queues
Extraction
ƒAn element with highest priority is extracted from the queue. The last element in the heap is moved to the root. The
resulting tree is converted into heap by fixing up procedure. The procedure is illustrated in figures (a)-(e)

(a) Element ‘A’ with priority 10 is extracted from the queue

(b) Last element ‘Y’ with priority 1 is moved to root
Heap property is violated.

(c) To fix up, element ‘Y’ with priority 1 is swapped with

element ‘X’ with priority 10 at the child node
(d) Element ‘Y’ with priority 1 is swapped with
element ‘S’ at the child node

(e) Element ‘Y’ with priority 1 is swapped with

element ‘T’ at the child node. The heap property
is now restored.
Comparison Sorting/Dr.A.Sattar/73

Priority Queue
Extraction Code
ƒ The EXTRACT method removes an element with the largest key at the root. It is replaced by
the last element in the heap. The heap size is reduced by 1. The resulting tree is fixed up to
restore heap property. The following is the pseudo code

EXTRACT(A)
1 max←A[1] ►Store largest key at root into max
2 A[1]←A[heap-size[A]] ►Move last key into root
3 heap-size[A]←heap-size[A]-1 ►Decrease heap size by 1
4 HEAPIFY(A) ►Re-fix the heap
5 return max ► return the largest key

ƒ In the worst case extraction procedure runs in θ(lg n) time.

Comparison Sorting/Dr.A.Sattar/74

37