Algorithms: Correctness and Complexity. Slide set 12.

Problem Complexity

c Theodore Norvell

Why time complexity is important

The time complexity of an algorithm can make a big
difference as to whether it is practical to apply it to large
instances.
Suppose the most frequent operations take 1 ns

n = 10 n = 50 n = 100 n = 1000
log2 n 3 ns 5 ns 6 ns 10 ns
n 10 ns 50 ns 100 ns 1 µs
n log2 n 33 ns 282 ns 664 ns 10 µs
n2 100 ns 2.5 µs 10 µs 1 ms
n3 1 µs 125 µs 1 ms 1s
n100 3 × 1083y 2.5 × 10179y 3 × 10209y 3 × 10310y
1.1n 2.6 ns 117 ns 13 µs 8 × 1050y
2n 1 µs 3. 5 × 1024y 4 × 1039y 3 × 10310y
n! 3 ms 10 × 1073y 3 × 10167y 1.3 × 102577y
n
22 6 × 10317y big Bigger HUGE
Another way to look at it is how big an instance can be
solved in a given time

Typeset March 18, 2020 1

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

In 1s In 1hr In 1day
log2 n 10300,000,000 big really big
n 109 3.6 × 1012 8.64 × 1013
n log2 n 4 × 107 1011 2 × 1012
n2 3 × 104 2 × 106 9 × 106
n3 1000 15, 000 44, 000
2n 29 42 46
n! 12 15 16
n
22 4 5 5
We broadly classify functions according to how they
behave
• Super exponential functions grow faster than any
exponential function.
2n
∗ n!, 2 , The Ackermann/Peter function.
• Exponential functions
∗ 2n, 3n etc
∗ 2Θ(1)n where Θ(1) represents some positive coeffi-
cient
∗ (The next time someone refers to “exponential
growth” ask yourself if they know what they are
talking about.)
• Polynomial functions n , n log2 n, n2, n3 etc
∗ While n log n is not really a polynomial, it is bounded
by two polynomials and so is considered a polyno-
mial.
∗ nΘ(1) where Θ(1) represents some positive coeffi-
cient
Typeset March 18, 2020 2
Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

• Polylog functions (polynomials of logs).

∗ log2 n, (log2 n)2, ...
∗ (log n)Θ(1)

Feasible and infeasible

As a general rule, we say that any algorithm that is
• polynomial time or better is feasible
• superpolynomial time is infeasible
However, as you can see from the tables, if you are
dealing with billions of bits then even an n2 algorithm is
impractical.
In compiling, it is a general rule that any ‘optimization’
method should be Θ(n2) or better in the size of a
subroutine.
Obviously any Θ(n100) time algorithm will be impractical.
In practice polynomial-time algorithms on RAMs are
usually not worse than Θ(n6), since each increase in
the exponent corresponds to some complication in the
algorithm such as an additional loop nesting.
One benefit of drawing the line at polynomial time is
that the model of computation is not important. E.g.,
any polynomial RAM algorithm can be translated to
a polynomial time algorithm on a Turing machine and
conversely.

Typeset March 18, 2020 3

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

Upper and lower bounds on function

complexity
Suppose we don’t know whether or not f ∈ Θ(g).
It is possible that still that we know something about their
relationship.
Definition: Function f is asymptotically dominated by
g if and only if there exist positive numbers b, and m such
that
∀n ∈ N · n > m ⇒ f (n) < b × g(n)
Notation: We write f g to mean that f is
asymptotically dominated by g .
In terms of sets, we say f ∈ O(g).
I.e. O(g) is the set of all functions dominated by g.
O(n) ⊆ O(n log n) ⊆ O(n2) ⊆ O(n3) ⊆ O(2n)
Definition: Function f asymptotically dominates by g
if and only if there exist positive numbers a, and m such
that
∀n ∈ N · n > m ⇒ a × g(n) < f (n)
Notation: We write f g to mean that f asymptotically
dominates g .
In terms of sets, we say f ∈ Ω(g).
I.e. Ω(g) is the set of all functions that dominate g.
Ω(n) ⊇ Ω(n log n) ⊇ Ω(n2) ⊇ Ω(n3) ⊇ Ω(2n)
We have f g if and only if g f ; in other words
f ∈ O(g) if and only if g ∈ Ω(f )
Typeset March 18, 2020 4
Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

Relation to Θ. For any g :

• Θ(g) ⊂ O(g)
• Θ(g) ⊂ Ω(g)
• Θ(g) = O(g) ∩ Ω(g)
Comparing O, Θ, and Ω in three Venn diagrams

Typeset March 18, 2020 5

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

Problem complexity
A problem consists
• an input set (called the set of instances)
• a set of outputs.
• an acceptability relation between these sets
• a measure of input size.
Using precondition P and postcondition R:
• P identifies inputs out of a (possibly) larger set.
• R defines the acceptable outputs for each input.
An algorithm A solves problem B iff, for every input, A
terminates with an acceptable output.
Exact problem complexity
The worst-case time complexity of a problem is the
worst-case time complexity of the fastest algorithm that
solves it.
For the rest of this slide deck, we are concerned only
with worst-case time complexity.
Note that problem complexity is model dependent.
Usually the RAM model is considered the standard.
When possible, the complexity of a problem should be
stated using Θ(f ) notation.

Typeset March 18, 2020 6

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

Upper-bounds
Often the exact problem complexity of a problem is hard
to calculate exactly as we must consider every algorithm
for the problem — including ones that no one has yet
thought of.
You may know a fast algorithm for a problem.
• But that doesn’t mean that Fred won’t come up with a
faster one tomorrow
If you know an algorithm has a worst-case time function
in O(g) then the problem complexity is in O(g).
For example,
• merge sort sorts n numbers in Θ(n log n) time
• therefore (since Θ(g) ⊆ O(g)) merge sort sorts in
O(n log n) time
• therefore the fastest sorting algorithm sorts in
O(n log n) time
• therefore (by definition) the problem complexity of
sorting is in O(n log n)
• in other words O(n log n) is an upper bound for sorting
n numbers.
Proof of the third step: Assume (falsely) that the fastest
sorting algorithm does not take O(n log n) time. Then
merge sort is faster that the fastest sorting algorithm.
Contradiction.

Typeset March 18, 2020 7

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

Note that we can not (yet) conclude that the fastest

sorting algorithm sorts in Θ(n log n).
Nor can we (yet) conclude that the complexity of sorting
is in Θ(n log n)
When problem complexity is stated using O, we say the
result is an upper-bound.
Lower-bounds
What if we can prove that no algorithm can be quicker
than Θ(f ) time?
• Then we have that Ω(f ) is a lower-bound for the
problem.

Case study — Matrix multiplication

Matrix multiplication (MM). Calculate A × B where A and
B are n by n matrices
From your high-school education you know that we can
compute MM with
for i, j ∈ {0, ..n}
C(i, j) := 0
for k ∈ {0, ..n} C(i, j) := C(i, j) + A(i, k) × B(k, j)
end for
end for
This is a Θ(n3) algorithm and so we know that the time
complexity of MM ∈ O(n3).

Typeset March 18, 2020 8

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

Since any algorithm has to at least look at each input

number and there are 2n2 of them, any algorithm for MM
can not have a complexity of better than Ω(n2).
Conclusion:
• The complexity of MM is in Ω(n2) ∩ O(n3).
This conclusion is secure against the future.
• Discovery of new algorithms will not invalidate it
• However new ideas may improve on it.
In 1969 Volker Straßen (to much surprise) found a way
to multiply matrices that takes Θ(nlog2 7) (approximately
Θ(n2.807)) and so we can improve the result.
• The complexity of MM is in Ω(n2) ∩ O(nlog2 7)
In 1990 Coopersmith and Winograd found an algorithm
that is in (approx) O(n2.376).
And so we can conclude that
• The complexity of MM is in (approx) Ω(n2) ∩ O(n2.376)
Recently Le Gall lowered the upper bound to O(n2.3728639)
It is an open problem whether or not the complexity of
MM is in O(n2) (and hence in Θ(n2))

Typeset March 18, 2020 9

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

Case study: sorting

As seen above, the complexity of sorting n numbers is
upper bounded by O(n log n).
In this section we will show that:
• In a restricted model of computation, the complexity is
lower bounded by Ω(n log n)
• and so (since merge sort fits into this model), the
complexity of sorting (in the restricted model) is
Θ(n log n)

A restricted model of computation

In this model of computation, two items may be compared
but we will not otherwise access the value of an item,
other than to copy it
An algorithm in this model can be thought of as a set of
trees, one for each tree for size of input.
Each tree node either represents an action or a
comparison. Action nodes have zero or one child,
comparison nodes have 1 or 2 children. (A comparison
whose conclusion is forgone has 1 child.)
For simplicity, I’ll assume that no two items of the input
are the same.
[Considering only a restricted set of inputs is kosher,
because any lower bound we find for this restricted
problem must also hold for the unrestricted problem.]
Selection sort
Typeset March 18, 2020 10
Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

for i ← [0, ..n − 1] do

var j := i
for k ← [i + 1, ..n] do if( a[k] < a[j] then j := k end if
end for
swap(i, j)
end for
Here is a tree for selection sort with n = 3. Assume that
a < b < c the leaves show the input that corresponds to
each path.

Next is a directed acyclic graph that can be expanded to

a tree with 24 leaves representing merge sort for n = 4

Typeset March 18, 2020 11

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

Example:

Typeset March 18, 2020 12

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

For a given n
• the best case time is the length of the shortest path
from root to leaf
• the worst case time is the length of the longest path
from root to leaf
To simplify, we’ll only count comparisons.
[Since we are investigating lower bounds, it is kosher to
ignore whole classes of operations. If a sorting algorithm
requires at least f (n) comparisons, then it must require
at least f (n) operations.]
Thus the worst (and best, and average) case time for
2
selection sort is n 2−n comparisons.
But selection sort is not the best algorithm. We want a
result that even the best algorithm can not beat.
Merge sort, for n = 2k , requires Θ(n log n) comparisons.
Can we do better?
The best algorithm has the shortest trees (as n
approaches ∞)
The key question is:
• How short can a tree for an input of size n be?
The inputs [2, 1, 3] and [4, 1, 6] look the same in this model,
as the only way to access the data is by comparing.
There are n! distinct inputs, as there are n! permutations
of n distinct values.

Typeset March 18, 2020 13

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

Any two permutations of the input require the algorithm

to do something different. Each possible input requires a
different path through the tree and hence a different leaf.
There are n! leaves in each tree, for inputs of size n,
regardless of the algorithm.
So our key question becomes
• If a binary tree has n! leaves, what is the shortest its
longest path can be?
We need to consider the squattiest trees.
Lemma 0 a binary tree of height x has at most 2x leaves.
Corollary 0 a binary tree with y leaves has height at
least log2 y.
Lemma 1 log2(n!) ∈ Ω(n log n)
Proof of lemma 1:

log2(n!)
=
log2(1 × 2 × ...n)
=
log2 2 + log2 3 + ... + log2 n
>

n+1
log2 (x − 1) dx
2

Typeset March 18, 2020 14

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

To justify the last step, I provide a “proof by picture”:

3
y
2

0
1 2 3 4 5 6 7 8
x

 n+1
log2 2 + log2 3... + log2 n > 2 log2 (x − 1) dx

n+1
log2 (x − 1) dx
2
=
n
1
ln x dx
ln 2 1

= “ ln x dx = x ln x − x”
1
((n ln n − n) − (1 ln 1 − 1))
ln 2
=
1
(n ln n − n + 1)
ln 2
1
∈ “The dominant term is n ln n”
ln 2
Ω(n log n)

Typeset March 18, 2020 15

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

Proof of the main result

The height of a binary tree with n! leaves

≥ “Corollary 0”
log2 n!
∈ “Lemma 1”
Ω(n log n)
Therefore, for every algorithm, there is at least one input
that requires at least Ω(n log n) comparisons to sort.
So Ω(n log n) is a lower bound on the complexity of
sorting.
• There is no point trying to find an algorithm that is
significantly better than merge sort or heap sort (that
accesses the data only by comparison).
Since O(n log n) is an upper bound and Ω(n log n) is a
lower bound, Θ(n log n) is an exact bound.
We now know the complexity of sorting.
How good is merge sort quantitatively?
How many comparisons does merge-sort use? For n a
power of 2 we have (worst-case)
m(1) = 0
m(2k ) = 2m(2k−1) + 2k − 1
So
m(1024) = 9217

Typeset March 18, 2020 16

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

For comparison
log2(1024!) = 8770
So merge sort is quite close to optimal.
(The reason quicksort is usually quicker than merge-sort
is that it has a smaller number of moves.)

Typeset March 18, 2020 17

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

Sorting in Θ(n) time

For this section we will assume that the items to be
sorted are natural numbers and are no bigger than some
constant 2m.
If it is not known a priori, a preliminary pass can be made
to determine m.
Radix sorting msb to lsb
Radix sorting is a lot like quick sort. Look at the
most-significant bit position of each number. Move the
numbers with 0 at this bit position to the front of the array.
Now do a radix sort of each ‘half’, only this time, using
the next-most-significant bit and so on down until all bits
are taken care of.

This takes Θ(mn) time (and can be very fast in practice).

This is linear in several senses
• For each particular m, we have Θ(n). E.g. sorting n
32-bit numbers can be done in Θ(n).
• If m is not fixed, then a realistic measure of the input
Typeset March 18, 2020 18
Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

size is not numbers (n), but rather bits (mn).

• A straight-forward generalization to variable sized data
(e.g. variable length strings in lexicographic order) can
be done in time Θ(N + n) where N is the total number
of characters.
The algorithm generalizes to bases bigger than 2, e.g.
10 or 256.
• If the base is 10, split the data into ten groups based
on the first digit.
• Then, recursively sort each group starting with the
second digit.
Radix sorting (lsb to msb)
A similar method works from least significant bit to the
most significant bit.
• First sort on the lsb
• Now sort on the 2ndlsb in such a way that values equal
on this bit remain in the same order
• And so on until all bits are processed.
Again the generalization to bigger bases is
straightforward.

Typeset March 18, 2020 19

Algorithms: Correctness and Complexity. Slide set 12. Problem Complexity

c Theodore Norvell

This can be implemented “by hand” using cards and a

skewer.
• One edge of each card is perforated with a pattern of
holes and notches.
• A notch is a 1 a hole is a 0
• The spindle is thrust trough the lsb, and all cards with
a 0 are picked up and moved to the front.
• Repeat until all bits are processed.
Libraries, businesses, and researchers often used this
hand technique at least until the 1980s.
This is again Θ(mn) (assuming your hand moves with
velocity independent of n). However it feels much more
like Θ(m) because much of the work is independent of n,
i.e., the time function looks rather like this
cm + kmn
where c is big and k is small.
A similar method was used for electromechanically
sorting punched cards from about 1901 until
superceeded by electronic computers.

Typeset March 18, 2020 20