CS212: Design & Analysis of Algorithms 3rd Semester

Lecture 7: August 21
Instructor: Prof. Prateek Vishnoi Indian Institute of Technology, Mandi

So far, we have seen different types of sorting algorithms like Insertion Sort, Merge Sort, and Quick Sort.
On average, we obtained a time complexity of O(n log n) and a space complexity of O(log n).

Again, the question arises, “Can we do better ?”.Unfortunately, the answer is “NO”for this model of compu-
tation.

Until now, our focus has been on the question, “Given a problem X and an algorithm Y, what is the upper
bound on the running time of the algorithm Y on X ?? ”. Now, we are shifting our focus to the question,
“Given any problem X, what is the lower bound on the running time of any algorithm on problem X ?”
First we define the term “comparison based sorting algorithm”.

Definition 7.1. Comparison-based sorting is a type of sorting algorithm where the ordering of elements
is determined by comparing pairs of elements. These algorithms make decisions based on the outcome of
comparisons between the elements in the input list, determining their relative order. There is no other way
to gain information about the items.

Merge Sort, Quick Sort, Insertion Sort are the comparision based sorting algorithm.

Theorem 7.2. Any deterministic comparision based sorting algorithm must perform Ω(n log n) comparisions
to sort n elements in the worst case.

Proof. To prove this theorem, we cannot assume the sorting algorithm is going to necessarily choose a pivot
as in Quicksort, or split the input as in Mergesort — we need to somehow analyze any possible (comparison-
based) algorithm that might exist.

Recall that the sorting algorithm must output a permutation of the input [a1 ,a2 ,...,an ]. The key to the ar-
gument is that

(a) there are n! different possible permutations the algorithm might output.

(b) For each of these permutations, there exists an input for which that permutation is the only correct
answer. For instance, the permutation [a3 ,a1 ,a4 ,a2 ] is the only correct answer for sorting the input [2, 4,
1, 3]. In fact, if you fix a set of n distinct elements, then there will be a 1-1 correspondence between the
different orderings the elements might be in and the permutations needed to sort them.
Given (a) and (b) above, this means we can fix some set of n! inputs (e.g., all orderings of 1, 2,... ,n), one
for each of the n! output permutations.
Let S be the set of these inputs that are consistent with the answers to all comparisons made so far (so,
initially, |S| = n!). We can think of a new comparison as splitting S into two groups: those inputs for which
the answer would be YES and those for which the answer would be NO. Now, suppose an adversary always
gives the answer to each comparison corresponding to the larger group. Then, each comparison will cut down
the size of S by at most a factor of 2. Since S initially has size n!, and by construction, the algorithm at the

7-1
7-2 Lecture 7: August 21

end must have reduced |S| down to 1 in order to know which output to produce, the algorithm must make
at least log2 (n!) comparisons before it can halt. We can then solve:

log2 (n!) = Ω(n ln n)

Let’s do an example with n = 3, and S as initially consisting of the 6 possible orderings of {1, 2, 3}:

(123),(132),(213),(231),(312),(321)

Suppose the sorting algorithm initially compares the first two elements a1 and a2 . Half of the possibilities
have a1 > a2 and half have a2 > a1 . So, the algorithm can answer either way and let’s say it answers that
a2 > a1 . This narrows down the input to the three possibilities:

(123),(132),(231).

Suppose the next comparison is between a2 and a3 . In this case, the most popular answer is that a2 > a3 , so
the adversary returns that answer which removes just one ordering, leaving the algorithm with (132),(231).
It now takes one more comparison to finally isolate the input ordering and determine the correct permutation
to output.

Judgement Question

What if we were to solve the problem “Arrange the input array so that the smallest n/2 elements comes before
the largest n/2 elements? In that case, does our lower bound still hold, or does it break down somewhere?
How quickly can you figure it out?

Theorem 7.3. Any deterministic comparision based sorting algorithm must perform Ω(n log n) comparisions
to sort n elements in the average case.

Proof. Let S be the set of all n! possible orderings of n distinct elements. As noted in the previous argument,
these each require a different permutation to be produced as output. Let’s now build out the entire decision
tree for algorithm A on S.

This tree has n! leaves, where the depth of a leaf is the number of comparisons performed by the sorting
algorithm on that input. Our goal is to show that the average depth of the leaves must be at least log2 n!. If
the tree is completely balanced, then each leaf is at depth log2 (n!) and we are done. To prove the theorem,
we just need to show that out of all binary trees on a given number of leaves, the one that minimizes their
average depth is a completely balanced tree. This is not too hard to see: given some unbalanced tree, we
take two sibling leaves at largest depth and move them to be children of the leaf of smallest depth. Since
the difference between the largest depth and the smallest depth is at least 2 (otherwise the tree would be
balanced), this operation reduces the average depth of the leaves. Specifically, if the smaller depth is d and
the larger depth is D, we have removed two leaves of depth D and one of depth d, and we have added two
leaves of depth d + 1 and one of depth D − 1. Since any unbalanced tree can be modified to have a smaller
average depth, such a tree cannot be one that minimizes average depth, and therefore the tree of smallest
average depth must in fact be balanced.

Theorem 7.4. Any randomised comparision based sorting algorithm must perform Ω(n log n) comparisions
to sort n elements in the average case.
Lecture 7: August 21 7-3

Proof. The argument here is a bit subtle. The first step is to argue that with respect to counting comparisons,
we can think of a randomized algorithm A as a probability distribution over deterministic algorithms. In
particular, we can think of a randomized algorithm A as a deterministic algorithm with access to a special
“random bit tape”: every time A wants to flip a coin, it just pulls the next bit off that tape. In that case,
for any given run of algorithm A, say reading bit-string from that tape, there is an equivalent deterministic
algorithm As with those bits hardwired in. Algorithm A is then a probability distribution over all those
deterministic algorithms As .

This means that the expected number of comparisons made by randomized algorithm A on some input I is
just
X
P r(s)(Running time of As on I).
s

If you recall the definition of expectation, the running time of the randomized algorithm is a random variable
and the sequences s correspond to the elementary events. So, the expected running time of the randomized
algorithm is just an average over deterministic algorithms. Since each deterministic algorithm has average-
case running time at least Ω(n log n), any average over them must too.

7.1. References

a) Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. 2009. Introduction to
Algorithms, 3rd Edition. MIT Press. isbn:978-0-262-03384-8 http://mitpress.mit.edu/books/introduction-
algorithms.
b) https://web.stanford.edu/class/archive/cs/cs161/cs161.1176/