0% found this document useful (0 votes)
188 views37 pages

LB&UB

The document discusses lower and upper bound theory for analyzing algorithms. It provides examples of using lower bound theory to determine: - The minimum number of comparisons needed to sort an array is Ω(nlogn), as every comparison-based sorting algorithm requires at least this many comparisons in the worst case. - The minimum number of comparisons needed to search a sorted array is Ω(logn), as this is the minimum information needed to identify an element. It also describes methods for establishing lower bounds, including trivial counting arguments, information theory which models the minimum information gained from comparisons, and decision trees which model algorithm execution as comparisons.

Uploaded by

Tsuki Zombina
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
188 views37 pages

LB&UB

The document discusses lower and upper bound theory for analyzing algorithms. It provides examples of using lower bound theory to determine: - The minimum number of comparisons needed to sort an array is Ω(nlogn), as every comparison-based sorting algorithm requires at least this many comparisons in the worst case. - The minimum number of comparisons needed to search a sorted array is Ω(logn), as this is the minimum information needed to identify an element. It also describes methods for establishing lower bounds, including trivial counting arguments, information theory which models the minimum information gained from comparisons, and decision trees which model algorithm execution as comparisons.

Uploaded by

Tsuki Zombina
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 37

Lower and Upper Bound Theory

Prof:Dr. Adnan YAZICI


Dept. of Computer Engineering
Middle East Technical Univ.
Ankara - TURKEY

1
A.Yazıcı Lower&Upper Bound, Fall 2015
Lower and Upper Bound Theory
How fast can we sort?
Lower-Bound Theory can help us answer this
question.
• Most of the sorting algorithms are comparison
sorts: only use comparisons to determine the
relative order of elements.
Examples: insertion sort, merge sort, quicksort,
heapsort.
• The best worst-case running time that we’ve
seen for comparison sorting is O(nlgn) .
Is O(nlgn) the best we can do? 2
A.Yazıcı Lower&Upper Bound, Fall 2015
Lower and Upper Bound Theory
• Lower Bound, L(n), is a property of the specific
problem, i.e. sorting problem, MST, matrix
multiplication, not of any particular algorithm
solving that problem.
• Lower bound theory says that no algorithm can
do the job in fewer than L(n) time units for
arbitrary inputs.
• For example, every comparison-based sorting
algorithm must take at least L(n), nlgn, time in
the worst case.
• L(n) is the minimum over all possible
algorithms, of the maximum complexity.
3
A.Yazıcı Lower&Upper Bound, Fall 2015
Lower and Upper Bound Theory
• Upper bound theory says that for any arbitrary inputs,
we can always sort in time at most U(n). How long it
would take to solve a problem using one of the known
algorithms with worst-case input gives us a upper
bound.
• U(n) is the minimum over all known algorithms, of the
maximum complexity.
• For example: Finding the maximum element in an
unsorted array is U(n), O(n).
• Both upper and lower bounds are minima over the
maximum complexity of inputs of size n.
• Improving an upper bound means finding a new
algorithm with better worst-case performance.
• The ultimate goal is to make these two functions
coincide. When this is done, the optimal algorithm will
have L(n) = U(n). 4
A.Yazıcı Lower&Upper Bound, Fall 2015
Lower Bounds
Examples:
• an estimate on a minimum amount needed to
multiply two n-by-n matrices of work needed to
solve a given problem.
• number of comparisons needed to find the mean
element in a set of n numbers.
• number of comparisons needed to sort an array of
size n.
• number of comparisons necessary for searching in a
sorted array.
• number of multiplications for n-digit integers.
5
A.Yazıcı Lower&Upper Bound, Fall 2015
Lower Bounds (cont.)
• Lower bound can be
– an exact count
– an efficiency class (Ω)

• Tight lower bound: there exists an algorithm


with the same efficiency as the lower bound

Problem Lower bound Tightness


sorting Ω(nlogn) yes
searching in a sorted array Ω(logn) yes
element uniqueness Ω(nlogn) yes
n-digit integer multiplication Ω(n) unknown
multiplication of n-by-n matrices Ω(n2) unknown
6
A.Yazıcı Lower&Upper Bound, Fall 2015
Methods for Establishing Lower Bounds
There are few techniques for finding lower bounds:

• trivial lower bound

• information-theory

• decision tree

• problem reduction

• adversary arguments
7
A.Yazıcı Lower&Upper Bound, Fall 2015
Trivial Lower Bound Method

1) Trivial Lower Bounds: For many problems it is possible to


easily observe that a lower bound identical to number of inputs
(n) exists, (or possibly outputs) to the problem.

• The method is based on counting the number of items that must


be processed in input and generated as output.

• Note that any algorithm must, at least, read its inputs and write
its outputs.

Example-1: Multiplication of a pair of nxn matrices


• requires that 2n2 inputs be examined and
• n2 outputs be computed, and
• the lower bound for this matrix multiplication problem is
therefore Ω(n2).
8
A.Yazıcı Lower&Upper Bound, Fall 2015
Trivial Lower Bound Method
Example: Finding maximum (or minimum) of
unordered array requires examining each input,
so it is Ω(n).

• A simple counting arguments shows that any


comparison-based algorithm for finding the maximum
value of an element in a list of size n must perform at
least n-1 comparisons for any input.

Comment:
• This method may and may not be useful

• be careful in deciding how many elements must be


processed
9
A.Yazıcı Lower&Upper Bound, Fall 2015
Information Theory
• The information theory method establishing
lower bounds by computing the limitations on
information gained by a basic operation and then
showing how much information is required
before a given problem is solved.
• This is used to show that any possible algorithm
for solving a problem must do some minimal
amount of work.
• The most useful principle of this kind is that the
outcome of a comparison between two items
yields one bit of information.
10
A.Yazıcı Lower&Upper Bound, Fall 2015
Lower and Upper Bound Theory
Example: For the problem of searching an
ordered list with n elements for the position
of a particular item,
<a,b,c,d>


>

<a,b> <c,d>

≤ > ≤ >

a b c d

For any outcome, at least 2 comparisions, therefore,


2 bits of information are necessary.
11
A.Yazıcı Lower&Upper Bound, Fall 2015
Information Theory
Example: The lower bound of the problem of
searching an ordered list with n elements for
the position of a particular item is Θ(lgn).
Proof: There are n possible outcomes, input
strings
– In this case lgn comparisons are necessary,
– If a comparison between two items yields one bit
of information, then unique identification of an
index in the list requires lgn bits.
– Therefore, lgn bits of information are necessary
to specify one of the n possibilities.
12
A.Yazıcı Lower&Upper Bound, Fall 2015
Information Theory
Example-2: By using information theory, the lower bound
for the problem of comparison-based sorting problem.
input

≤ >

≤ > ≤ >

<1,2,3>
<2,1,3>
≤ >
≤ >
111

<1,3,2> <3,2,1> 000


<3,1,2> <2,3,1>

Figure: There are 3! = 6 possible permutations of the n


input elements, so lgn! bits are required for lgn!
comparisons for sorting n things.
13
A.Yazıcı Lower&Upper Bound, Fall 2015
Information Theory
Example: By using information theory, the lower bound
for the problem of comparison-based sorting problem is
Θ(nlgn)
Proof:
– If we only know that the input is orderable, then
there are n! possible outcomes – each of the n!
permutations of n things.
– Within the comparison-swap model, we can only use
comparisons to derive information
– Since each comparison only yields one bit of
information, the question is what the minimum
number of bits of information needed to allow n!
different outcomes is, which is lgn! bits.
14
A.Yazıcı Lower&Upper Bound, Fall 2015
Information Theory
• How fast lgn! grow?
• We can bind n! from above by overestimating every
term of the product, and bind it below by
underestimating the first n/2 terms.

n/2*n/2*…*n/2*…*2*1 ≤ n*(n-1)*…*2*1 ≤ n* n*…*n


(n/2)n/2 ≤ n! ≤ nn
½(nlgn-n) ≤ lgn! ≤ nlgn
½nlgn- ½ n ≤ lgn! ≤ nlgn
½nlgn ≤ lgn! ≤ nlgn

• This follows that lgn!∈ Θ(nlgn)


15
A.Yazıcı Lower&Upper Bound, Fall 2015
Decision-tree model

• This method can model the execution of any


comparison based problem.
• One tree for each input size n.
• View the algorithm as splitting whenever it
compares two elements.
• The decision tree contains the comparisons along
all possible instruction traces.
• The running time of the algorithm = the length of
the path taken.

16
A.Yazıcı Lower&Upper Bound, Fall 2015
Decision-tree model
• Decision tree is a convenient model of algorithms
involving comparisons in which:
• Internal nodes represent comparisons
• leaves represent outcomes.
• There must be n internal nodes in all of these trees
corresponding to the n possible successful
occurrences of x in A.
• Because every leaf in a valid decision tree must be
reachable, the worst-case number of comparisons
done by such a tree is the number of nodes in the
longest path from the root to a leaf in the decision
tree consisting of the comparison nodes.
• Worst-case running time = the length of the longest
path of tree = the height of tree.
17
A.Yazıcı Lower&Upper Bound, Fall 2015
Decision-tree Model for Searching

Example: Comparison-based searching


problem
Proof: Let us consider all possible decision
trees which model algorithms to solve
the searching problem.
– FIND(n) is bounded below by the
distance of the longest path from the root
to a leaf in such a tree.

18
A.Yazıcı Lower&Upper Bound, Fall 2015
Decision-tree model for Searching
x:A(lg(n+1)/2)
Height=log2n

>
x:A(lg(n+1)/4) x:A(lg(3n+1)/4)

≤ >
> ≤ ….

x:A(1) …. x:A(lg(n+1)/4-1) x:A(lg(n+1)/4 +1) … x:A(n)

≤ ≤ ≤
> > ≤ > >
Return Failure Return Failure Return Failure Return Failure

A decision tree for a search algorithm

19
A.Yazıcı Lower&Upper Bound, Fall 2015
Decision-tree Model for Search on Sorted Elements

3
< >
=
1 3 5 Height=log2(7)=3
< > <
= >
=
0 1 2 4 5 6
<
= > <
=
> <
=
> < >
=
F 0 F F 2 F F 4 F F 6 F

Example decision tree for search on sorted list


<0,1,2,3,4,5,6>

20
A.Yazıcı Lower&Upper Bound, Fall 2015
Decision-tree Model for Searching
Example: Comparison-based searching problem
• A decision tree, with n leaves and height h, n ≤ 2h.
Indeed, a decision tree of height h with largest
number of leaves has all its leaves on the last
level. Hence, the largest number of leaves in such
a tree is 2h. FIND(n) = h ≥ log (n)
Since, n ≤ 2h and.
Such a longest path is Θ(lgn).
• Therefore, the lower bound for comparison-
based searching on ordered input is Θ(lgn).
21
A.Yazıcı Lower&Upper Bound, Fall 2015
Decision-tree model for Searching
x:A(1)
= ≠ Height = n
Return(A(1)) x:A(2)

=
Return(A(2))
x:A(n)
= ≠
Return(A(n)) Return(A(0))

A decision tree for search on unsorted list

The number of comparisons in the worst-case is equal to the


height (h) of the algorithm’s decision tree. h ≥ n and h ∈ Ω(n).

22
A.Yazıcı Lower&Upper Bound, Fall 2015
Decision-tree for Search on Unsorted Elements
4
= ≠
4 1
= ≠
1 5
= ≠
5 3
= ≠
Height = 5
3
=
2

2 0
=
0 F

Example decision tree for search on unsorted


list <4,1,5,3,2,0>, that is, h = n.
• Therefore, the lower bound for comparison-
based searching on unordered input is Θ(n).
23
A.Yazıcı Lower&Upper Bound, Fall 2015
Decision-tree model for Sorting
a1,a2,a3
yes no
a1<a2

a1a2a3 a2a1a3
yes no yes no
a2<a3 a1<a3

a1a3a2 a2a3a1
a1<a2<a3 a2<a1<a3
yes no yes no
a1<a3 a2<a3

a1<a3<a2 a3<a1<a2 a2<a3<a1 a3<a2<a1

• Each internal node is labelled for i,j element-of {1,2,...,n}.


• The left subtree shows subsequent comparisons if ai ≤ aj.
• The right subtree shows subsequent comparisons if ai ≥ aj.

24
A.Yazıcı Lower&Upper Bound, Fall 2015
Decision-tree model for sorting

Each permutation represents a distinct sorted order and the tree


muct have n! leaves. Since a binary tree of height h has no more
than 2h leaves, we have n! ≤ 2h.
25
A.Yazıcı Lower&Upper Bound, Fall 2015
26
A.Yazıcı Lower&Upper Bound, Fall 2015
Problem Reduction
• Prob. 1 (to be solved) (reduction) Prob 2 (solvable by Alg.A)
• Prob 2 (solvable by Alg.A) (Alg.A) Soln. to Prob. 1
• Another elegant means of proving a lower bound on a
problem P is to show that problem P is at least as hard
as another problem Q with a known lower bound.
• We need to reduce Q to P (not P to Q).
• In other words, we should show that an arbitrary
instance of problem Q can be transformed (in a
reasonably efficient way) to an instance of problem P,
so that any algorithm solving P would also solve Q as
well.
• Then a lower bound for Q will be a lower bound for P.
• Prob. Q (LB is known) (reduction) Prob.P (whose LB to be known)
27
A.Yazıcı Lower&Upper Bound, Fall 2015
Problem Reduction
Example: Euclidean minimum spanning tree (MST) problem:
Given n points in the Cartesian plane construct an Euclidian MST,
whose vertices are the given points. As a problem with a known LB,
we use the element uniqueness problem (EUP), whose LB is known
as Ω(nlgn). The reduction is as follows:
• Transform any set x1,x2,…,xn of n real numbers into a set of
points in the Cartesian plane by simply adding 0 as the points’s y
coordinate: (x1,0), (x2,0),…,(xn,0).
• This reduction implies that Ω(nlgn) is a LB for the Euclidian
MST problem as well.
• Solution of EUP can now be found using the solution of
Euclidean MST: Let T be any Euclidean MST found for this set of
points, (x1,0), (x2,0),…,(xn,0).
• Since T must contain a shortest edge, checking whether T
contains a zero length edge will answer the question about
uniqueness of the given numbers, x1,x2,…,xn. 28
A.Yazıcı Lower&Upper Bound, Fall 2015
Example for reduction: Convex Hull

• Example: Computing the convex hull in


Computational Geometry is perhaps the most
basic, elementary function on a set of points.
• A Convex Hull is the smallest convex polygon
that contains every point of the set S.
– A polygon P is convex if and only if, for any
two points A and B, inside the polygon, then
the line segment AB is inside P.

29
A.Yazıcı Lower&Upper Bound, Fall 2015
Example for reduction Convex Hull (cont.)

• One way to visualize a convex hull is to put


a "rubber band" around all the points, and
let it wrap as tight as it can.

• We can think of a rubber


balloon for the three-
dimensional case.

30
A.Yazıcı Lower&Upper Bound, Fall 2015
Example for reduction Convex Hull (cont.)
Theorem: The comparison-based sorting problem is
transformable (in linear time) to the convex hull
problem, therefore; the problem of finding the
ordered convex hull of n points in the plane requires
Ω(nlogn) time. That is, LB of the problem of
convex hull is also Ω(nlogn).

Proof: First we introduce a proposition:


Proposition (Lower Bounds via Transformability):
If a problem A is known to require T(n) time and A
is τ(n)- transformable to B (A ∝τ(n)B) then B
requires at least T(n) – O(τ(n)) time.
31
A.Yazıcı Lower&Upper Bound, Fall 2015
Example for reduction Convex Hull (cont.)
• Reduction: Given n real sorted numbers x1, x2, ..., xn, all
positive, we determine corresponding points (x1,x12),
(x2,x22),…, (xn,xn2).
• All of these points lie on the parabola y = x2 and they all
together result in a convex hull.
• Since LB of comparison-based sorting problem is known,
Ω(nlogn), the LB of the problem of convex hull is also
Ω(nlogn).
• So, when n real numbers x1, x2, ..., xn are given, we can
show how a convex hull algorithm can be used to sort these
numbers with only linear overhead.
• The convex hull of this set, in standard form, will consist of
a list of the points sorted.
• One pass through the list will enable us to read off the xi in
order (in sorted form).
32
A.Yazıcı Lower&Upper Bound, Fall 2015
Example for reduction Convex Hull (cont.)

y y = x2

*
*
*
*
*
*
x
x1 x2 x3 x4 x5 x6
Sorted  
33
A.Yazıcı Lower&Upper Bound, Fall 2015
Example for reduction Convex Hull (cont.)

Proposition (Upper Bounds via Transformability): If a


problem B can be solved in T(n) time and problem A is
τ(n)- transformable to B (A ∝τ(n)B) then A can be
solved in at most T(n) + O(τ(n)) time.

34
A.Yazıcı Lower&Upper Bound, Fall 2015
Oracles and Adversary Arguments
• Another technique for obtaining LBs consists of making use of an oracle or
adversary arguments.
• Adversary arguments establish (worst-case) lower bounds by dynamically
creating inputs to a given algorithm that force the algorithm to perform
many basic operations.
• In order to derive a good LB, the adversary tries its best to cause the
algorithm to work as hard as it might.
• We only require that the adversary gives answers that are consistent and
truthful.
• We also assume that the algorithm uses a basic operation when querying the
adversary.
• Clever adversaries force any given algorithm to make many queries before
the algorithm has enough information to output the solution.
• LBs for the worst-case complexity are then obtained by counting the
number of basic operations that the adversary forces the algorithm to
perform.

35
A.Yazıcı Lower&Upper Bound, Fall 2015
Oracle and Adversary Arguments
• Adversary argument: a method of proving a lower bound by playing role of
adversary that makes algorithm work the hardest by adjusting input.
• LBs for the worst-case complexity are then obtained by counting the
number of basic operations that the adversary forces the algorithm to
perform.
Example-1: “Guessing” a number between 1 and n with yes/no
questions.
Adversary:Put the number in a larger of the two subsets
generated by last question.
For example; Adversary (or Oracle) knows your guess, which in the order
of <4,7,9,4,8>. So, you put the the numbers in that order, 8 at the end, to
force the algorithm the hardest way. That way, output is guessed as 8 only
after n-1 guesses. That means the lower bound for this problem is Ω(n).

Example-2: Merging two sorted lists of size n


a1 < a2 < … < an and b1 < b2 < … < bn
• Adversary: ai < bj iff i < j.
• There output: b1 < a1 < b2 < a2 < … < bn < an requires 2n-1 comparisons
of adjacent elements. That means the lower bound for this problem is Ω(n).36
A.Yazıcı Lower&Upper Bound, Fall 2015
Oracles and Adversary Arguments
Example: (Merging Problem) Given the sets A(1:m) and B(1:n), where the items
in A and in B are sorted. Investigate lower bounds for algorithms merging these
two sets to give a single sorted set.
• Assume that all of the m+n elements are distinct and A(1)<A(2)< …<A(m)
and B(1) < B(2) < …< B(n).
• Elementary combinatorics tells us that there are C((m+n), n)) ways that the
A’s and B’s may merge together while still preserving the ordering within A
and B.
• Thus, if we use comparison trees as our model for merging algorithms, then
there will be C((m+n), n)) external nodes and therefore log C((m+n), m)
comparisons are required by any comparison-based merging algorithm in
worst-case.
• If we let MERGE(m,n) be the minimum number of comparisons need to
merge m items with n items then we have the inequality
log C((m+n), m) ≤ MERGE(m,n) ≤ m+n-1.
• The upper bound and lower bound can get arbitrarily far apart as m gets
much smaller than n.
37
A.Yazıcı Lower&Upper Bound, Fall 2015

You might also like