Lecture 2 Sorting

Sorting Problem
Input : A sequence of n numbers {a1 , a2 ,..., an }.
Output : A permutation {a '1 , a '2 ,..., a 'n } of input
sequence such that a '1 a '2  a 'n .
e.g.,
Input : 5, 2, 4, 6, 1, 3.
Output :1, 2, 3, 4, 5, 6.
Insertion Sort, Merge Sort
 Efficiency
• Running time from receiving the input to
producing the output.

Running time
2
Insertion Sort O(n )
Merge Sort O(n log n)
 Heapsort
• Heap, a data structure
• Max-Heapify procedure
• Building a max-heap
• Heapsort
 Quiz Sample
• Is array a data structure?
 Is array a data structure?
• No!
• A data structure is a data organization
associated with a set of operations
(standard algorithms) for efficiently using
the data.
• What data structures do you know on
array?
 Is array a data structure?
• No!
• A data structure is a standard part in
construction of algorithms.
• What data structures do you know on
array?
• Stack, queue, list, …, heap.
 A Data Structure Heap
• A heap is a nearly complete binary tree
which can be easily implemented on an
array. 1
6
2 3
5 3 6 5 3 2 4 1

4 5 6
2 4 1
 Nearly complete binary tree
• Every level except bottom is complete.
• On the bottom, nodes are placed as left as
possible. 1
1
6
6
2 3
2 3
5 3
5 3
4 5 6
4 5 6
2 4 1
2 1 4

No! Yes!
Algorithms associated with Heap

Parent (i )
return i / 2;

Left(i )
return 2i;

Right (i )
return 2i  1;
 Two Special Heaps
• Max-Heap
• Min-Heap
 Max-Heap

In a max - heap, every node i other than the

root satisfies the following property :
A[Parent(i )]  A[i ].
 Min-Heap

In a min - heap, every node i other than the

root satisfies the following property :
A[Parent (i )]  A[i ].
Algorithms associated with Max-Heap

• Three algorithms associated with heap.

• In addition, it is associated with two more
algorithms: Max-Heapify and Build-Max-
Heap.
 Max-Heapify
• Max-Heapify(A,i) is a subroutine.
• When it is called, two subtrees rooted at
Left(i) and Right(i) are max-heaps, but A[i]
may not satisfy the max-heap property.
• Max-Heapify(A,i) makes the subtree
rooted at A[i] become a max-heap by
letting A[i] “float down”.
 4 14

4 7
14 7

2 8 1 2 8 1

14

8 7

2 4 1
Max - Heapify( A, i )
l  Left(i );
r  Right (i );
if l heap  size[ A] and A[l ]  A[i ]
then largest  l
else largest  i;
if r heap  size[ A] and A[r ]  A[largest ]
then largest  r ;
if largest i
then begin exchange A[i ]  A[largest ];
Max - Heapify( A, largest );
end - if
 Running time
height lg n  where n heap - size[ A]
Hence, Max - Heapify runs in time O(lg n).

Let h height. Then h is the largest integer

satisfying n 1  2    2 h  1  1,
that is, n 2 h.
Hence, h lg n .
 Building a Max-Heap

Build - Max - Heap( A)

heap - size[ A]  length[ A];
for i  length[ A] / 2 downto 1
do Max - Heapify( A, i );

e.g., 4, 1, 3, 2, 16, 9, 10, 14, 8, 7.

 The last location who has a child is  n / 2.

1 3

2 16 9 10

 n / 2

14 8 7
n

Proof. It is parent of location n

 Building a Max-Heap

Build - Max - Heap( A)

heap - size[ A]  length[ A];
for i  length[ A] / 2 downto 1
do Max - Heapify( A, i );

e.g., 4, 1, 3, 2, 16, 9, 10, 14, 8, 7.

 4

1 3

2 16 9 10

14 8 7
 4

1 3

2 16 9 10

14 8 7
 4

1 3

16 9 10
14

2 8 7
 4

1 3

14 16 9 10

2 8 7
 4

1 10

14 16 9 3

2 8 7
 4

1 10

14 16 9 3

2 8 7
 4

16 10

14 1 9 3

2 8 7
 4

16 10

14 7 9 3

2 8 1
 16

4 10

14 7 9 3

2 8 1
 16

14 10

4 7 9 3

2 8 1
 16

14 10

7 9 3
8

2 4 1
 Analysis
h
running time  # node(i ) O(h  i )
i 0

where # node(i ) is the number of nodes

at level i and h height ( A).

h lg n  where n heap - size( A)

i
# node(i ) 2
 16 i 0

14 i 1
10

h i
7 9 3
8

2 4 1 i h
 Running time
h
 h i  h
(h  i )

i 0
2 O(h  i ) O  2 (h  i )  O(2  h  i ) O(n)
i

 i 0 
h

i 0 2

because

l 1/ 2

l 0 2
l

(1  1 / 2) 2
2,

2 h O(n).

    
l 1 1 1 1

l 0 2
l
 k   k   k   k  
k 1 2 k 2 2 k 3 2 k 4 2

1/2 1 / 22 1 / 23 1 / 24 
a
   
1 1/ 2 1 1/ 2 1 1/ 2 1 1/ 2
 
k 0
aq k

1 q
1 1 1 1 for q  1.
1   2  3    2
2 2 2 1 1/ 2
 Heapsort
Heapsort ( A)
Buid - Max - Heap( A);
for i  length[ A] downto 2
do begin
exchange A[1]  A[i ];
heap - size[ A]  heap - size[ A]  1;
Max - Heapify( A,1);
end - for
Input: 4, 1, 3, 2, 16, 9, 10, 14, 8, 7.
Build a max-heap
16

14 10

7 9 3
8

2 4 1

16, 14, 10, 8, 7, 9, 3, 2, 4, 1.

 1

14 10

7 9 3
8

2 4 16
 1

14 10

7 9 3
8

2 4 16

1, 14, 10, 8, 7, 9, 3, 2, 4, 16.

 14

8 10

7 9 3
4

2 1 16
 14

8 10

7 9 3
4

2 1 16

14, 8, 10, 4, 7, 9, 3, 2, 1, 16.

 1

8 10

7 9 3
4

2 14 16
 1

8 10

7 9 3
4

2 14 16

1, 8, 10, 4, 7, 9, 3, 2, 14, 16.

 10

8 9

7 1 3
4

2 14 16
 10

8 9

7 1 3
4

2 14 16

10, 8, 9, 4, 7, 1, 3, 2, 14, 16.

 2

8 9

7 1 3
4

10 14 16
 2

8 9

7 1 3
4

10 14 16

2, 8, 9, 4, 7, 1, 3, 10, 14, 16.

 9

8 3

7 1 2
4

10 14 16
 9

8 3

7 1 2
4

10 14 16

9, 8, 3, 4, 7, 1, 2, 10, 14, 16.

 2

8 3

7 1 9
4

10 14 16
 8

7 3

4 2 1 9

10 14 16
 1

7 3

4 2 8 9

10 14 16
 7

4 3

1 2 8 9

10 14 16
 2

4 3

1 7 8 9

10 14 16
 4

2 3

1 7 8 9

10 14 16
 1

2 3

4 7 8 9

10 14 16
 3

2 1

4 7 8 9

10 14 16
 1

2 3

4 7 8 9

10 14 16
 2

1 3

4 7 8 9

10 14 16
 1

2 3

4 7 8 9

10 14 16
 Running Time
Heapsort ( A)
Buid - Max - Heap( A);
for i  length[ A] downto 2 O(n)
do begin
exchange A[1]  A[i ];
heap - size[ A]  heap - size[ A]  1;
Max - Heapify( A,1); O(lg n)
end - for
O(n lg n)
 Quicksort
2
• Worst-case running time ( n )

• Expected running time O(n lg n)

• The best practical choice (why?)
 Divide and Conquer
• Divide the problem into subproblems.
• Conquer the subproblems by solving
them recursively.
• Combine the solutions to subproblems
into the solution for original problem.
 Idea of Quicksort
Partition array A[ p..r ] into subarrays
A[ p..q  1] and A[q  1..r ] such that
A[i ]  A[q ] for p i q  1
A[q ]  A[i ] for q  1 i r.
Then after sort A[ p..q  1] and A[q  1..r ],
no work is needed to combine them.
 Example

3, 4, 2, 1, 5̂, 8, 9, 7, 6.
1, 2, 3, 4, 5̂, 6, 7, 8, 9.
 Quicksort
Quicksort ( A, p, r )
if p  r
then begin
q  Partition ( A, p, r );
Quicksort ( A, p, q  1);
Quicksort ( A, q  1, r );
end - then
 How to find such a partition?
• Such a partition may not exist.
e.g., 5, 4, 3, 2, 1.
• Hence, we may need to make such a
partition.

• Take a A[i].
• Classify other A[j] by comparing it with A[i].

, 2, 8, 7, 1, 3, 5, 6, 4̂


2, 8, 7, 1, 3, 5, 6, 4̂

2, 8, 7, 1, 3, 5, 6, 4̂
 
2, 8, 7, 1 , 3, 5, 6, 4̂
 
2, 1 , 7, 8, 3, 5, 6, 4̂
 
2, 1, 3, 8, 7, 5, 6, 4̂
 
2, 1, 3, 8, 7, 5, 6, 4̂

2, 1, 3, 8, 7, 5, 6, 4̂
2, 1, 3, 4̂, 7, 5, 6, 8
Partition ( A, p, r )
x  A[r ];
i  p  1;
for j  p to r  1
do if A[ j ]  x
then i  i  1 and exchange A[i ]  A[ j ];
exchange A[i  1]  A[r ];
return i  1;
running time (n)
 
2, 8, 7, 1, 3, 5, 6, 4̂
 
2, 8, 7, 1, 3, 5, 6, 4̂
 
2, 3, 7, 1, 8, 5, 6, 4̂

2, 3, 7, 1 , 8, 5, 6, 4̂
 
2, 3, 1 , 7, 8, 5, 6, 4̂

2, 3, 1 , 7, 8, 5, 6, 4̂

2, 3, 1 , 4, 8, 5, 6, 7̂
 Expected Partition
If all possible permutations are considered,
then each number appears in the last position
has the probability 1 / n. Thus, the expected
size of each subtree is
1 n 1
(0  1    n  1)  ,
n 2
which is in a balanced partition.
Expected Running Time
Assume : the running time of partition c1n
Guess : the expected running time of Quicksort
E[T (n)] cn log n
Induction :
1
E[T (n)]  (E[T (0)]  E[T (n  1)]  c1n)
n
1
 (E[T (1)]  E[T (n  2)]  c1n)
n

1
 (E[T (n  1)]  E[T (0)]  c1n)
n
2c n  1
c1n   i lg i
n i 1
 2c n  1
c1n   i lg i
n i 1
n 1 2 /( n ( n  1))
 i
c1n  c(n  1) lg i 
 i 1 
12  2 2    (n  1) 2
c1n  c(n  1) lg
n(n  1) / 2
2n  1
c1n  c(n  1) lg
3
2n
c1  cn lg cn lg n  (c1  c lg(3 / 2))n
3
(Choose c c1 / lg(3 / 2).)
Randomized Quicksort

Randomized - Partition( A, p, r )
i  Random( p, r );
exchange A[i ]  A[r ];
Partition( A, p, r );
 Randomized Quicksort

Randomized - Quicksort( A, p, r )
q  Randomized - Partition ( A, p, r );
Ramdomized - Quichsort ( A, p, q  1);
Ramdomized - Quichsort ( A, q  1, r );
 What we learnt in this lecture?

• What is heap, max-heap and min-heap?

• Heapsort and Quicksort.
• What is expected running time?