Chapter 3

With Question/Answer Animations

Copyright © McGraw-Hill Education. All rights reserved. No reproduction or distribution without the prior written consent of McGraw-Hill Education.
Section 3.2
Section Summary Donald E. Knuth
( )
 Big-O Notation
 Big-O Estimates for Important Functions
 Big-Omega and Big-Theta Notation

Edmund Landau Paul Gustav Heinrich Bachmann

( ) ( )
The Growth of Functions
 In both computer science and in mathematics, there
are many times when we care about how fast a
function grows.
 In computer science, we want to understand how
quickly an algorithm can solve a problem as the size of
the input grows.
 We can compare the efficiency of two different
algorithms for solving the same problem.
 We can also determine whether it is practical to use a
particular algorithm as the input grows.
Big-O Notation
Definition: Let f and g be functions from the set of
integers or the set of real numbers to the set of real
numbers. We say that f(x) is O(g(x)) if there are constants
C and k such that

whenever x > k. (illustration on next slide)

 This is read as “f(x) is big-O of g(x)” or “g asymptotically
dominates f.”
 The constants C and k are called witnesses to the
relationship f(x) is O(g(x)). Only one pair of witnesses is
needed.
Illustration of Big-O Notation
f(x) is O(g(x)
Some Important Points about Big-
O Notation
 If one pair of witnesses is found, then there are infinitely
many pairs. We can always make the k or the C larger and
still maintain the inequality .
 Any pair C and k where C < C and k < k is also a pair of
witnesses since whenever x > k > k.
You may see “ f(x) = O(g(x))” instead of “ f(x) is O(g(x)).”
 But this is an abuse of the equals sign since the meaning is
that there is an inequality relating the values of f and g, for
sufficiently large values of x.
 It is ok to write f(x) O(g(x)), because O(g(x)) represents
the set of functions that are O(g(x)).
 Usually, we will drop the absolute value sign since we will
always deal with functions that take on positive values.
Using the Definition of Big-O Notation
Example: Show that is .
Solution: Since when x > , x < x and < x

 Can take C = and k = as witnesses to show that

(see graph on next slide)
 Alternatively, when x > , we have x x and < x .
Hence,
when x >
 C= and k = as witnesses instead.
Illustration of Big-O Notation
is
Big-O Notation
 Both and
are such that and .
We say that the two functions are of the same order. (More on this
later)

 If and h(x) is larger than g(x) for all positive real

numbers, then .

 Note that if for x > k and if

for all x, then if x > k. Hence, .

 For many applications, the goal is to select the function g(x) in O(g(x))
as small as possible (up to multiplication by a constant, of course).
Using the Definition of Big-O Notation
Example: Show that x is O(x ).
Solution: When x > , x x Take C and k
as witnesses to establish that x is O(x ).
Would C and k work?)
Example: Show that n is not O(n).
Solution: Suppose there are constants C and k for
which n Cn, whenever n > k. Then (by dividing
both sides of n Cn) by n, then n C must hold for
all n > k. A contradiction!
Big-O Estimates for Polynomials
Example: Let
where are real numbers with an .
Then f x xn). Uses triangle inequality,
an exercise in Section 1.8.
Proof: |f(x)| = |anx + an-1 x + ∙∙∙ + a1x + a0|
n n-1 1

Assuming x > 1 |a n |x n + |a | xn-1 + ∙∙∙ + |a |x1 + |a |

n-1 1 0
xn (|an| + |an-1| /x + ∙∙∙ + |a1|/xn-1 + |a0|/ xn)
xn (|an| + |an-1| + ∙∙∙ + |a1|+ |a0|)
 Take C = |an| + |an-1| + ∙∙∙ + |a0| and k = f x xn).
 The leading term anxn of a polynomial dominates its
growth.
Big-O Estimates for some
Important Functions
Example: Use big-O notation to estimate the sum of
the first n positive integers.
Solution:

Example: Use big-O notation to estimate the factorial

function
Solution:

Continued
Big-O Estimates for some
Important Functions
Example: Use big-O notation to estimate log n!
Solution: Given that (previous slide)
then .
Hence, log(n!) is O(n C
Display of Growth of Functions

Note the difference in behavior of functions as n gets larger

Useful Big-O Estimates Involving
Logarithms, Powers, and Exponents
 If d > c > , then
nc is O(nd), but nd is not O(nc).
 If b > and c and d are positive, then
(logb n)c is O(nd), but nd is not O((logb n)c).
 If b > and d is positive, then
nd is O(bn), but bn is not O(nd).
 If c > b > , then
bn is O(cn), but cn is not O(bn).
Combinations of Functions
 If f x O g (x)) and f x O g (x)) then
( f + f )(x) is O(max( g (x) g (x)

 If f x and f x O g(x)) then

( f + f )(x) is O(g(x)
 See text for argument
 If f x O g (x)) and f x O g (x)) then
( f f )(x) is O(g (x)g (x)
 See text for argument
Ordering Functions by Order of Growth
 Put the functions below in order so that each function is
big-O of the next function on the list.
 f1(n) = ( )n We solve this exercise by successively finding the function that
grows slowest among all those left on the list.
 f2(n) = 8
• f (n) = 10000 (constant, does not increase with n)
f3(n) = ( )
9

•f (n) = log (log n) (grows slowest of all the others)
f4(n) =
5

•f (n) = (log n ) 2
(grows next slowest)
f5(n) =
3

•f (n) = n (log n) (next largest, (log n) factor smaller than any power of n)
2 3 3

f6(n) = ( )3 6
 •f (n) = 8n +17n +111 (tied with the one below)
2
3 2

 f7(n) = •f (n) = n + n(log n)

8
3 2
(tied with the one above)

 f8(n) = •f (n) = (1.5) 1 (next largest, an exponential function)

n

 f9(n) = •f (n) = 2 n
(grows faster than one above since 2 > 1.5)
4

 f10(n) = n •f (n) = 2 (n +1) (grows faster than above because of the n +1 factor)
7
n 2 2

•f10(n) = n! ( n! grows faster than cn for every c)

Big-Omega Notation
Definition: Let f and g be functions from the set of
integers or the set of real numbers to the set of real
numbers. We say that
if there are constants C and k such that
when x > k.
 We say that “f(x) is big-Omega of g(x).”
 Big-O gives an upper bound on the growth of a function,
while Big-Omega gives a lower bound. Big-Omega tells us
that a function grows at least as fast as another.
 f(x) is (g(x)) if and only if g(x) is O(f(x)). This follows
from the definitions. See the text for details.
Big-Omega Notation
Example: Show that is
where .
Solution: for all
positive real numbers x.
 Is it also the case that is ?
Big-Theta Notation
 Definition: Let f and g be functions from the set of
integers or the set of real numbers to the set of real
numbers. The function if
and .

 We say that “f is big-Theta of g(x)” and also that “f(x) is of

order g(x)” and also that “f(x) and g(x) are of the same
order.”
 if and only if there exists constants C , C
and k such that C g(x) < f(x) < C g(x) if x > k. This follows
from the definitions of big-O and big-Omega.
Big-Theta Notation
Example: Sh0w that f(x) = x + 8x log x is x ).
Solution:
 x + 8x log x x
8x log x x
 x + 8x log x is x ).
 x is clearly x + 8x log x)
 x + 8x log x is x ).
Big-Theta Notation
 When it must also be the case that

 Note that if and only if it is the case

that and .
 Sometimes writers are careless and write as if big-O
notation has the same meaning as big-Theta.
Big-Theta Estimates for
Polynomials
Theorem: Let
where are real numbers with an .
Then f x xn (or x )).
(The proof is an exercise.)
Example:
The polynomial is order of x (or
x )).
The polynomial
is order of x (or x ) ).
Section 3.3
Section Summary
 Time Complexity
 Worst-Case Complexity
 Algorithmic Paradigms
 Understanding the Complexity of Algorithms
The Complexity of Algorithms
 Given an algorithm, how efficient is this algorithm for
solving a problem given input of a particular size? To
answer this question, we ask:
 How much time does this algorithm use to solve a problem?
 How much computer memory does this algorithm use to solve
a problem?
 When we analyze the time the algorithm uses to solve the
problem given input of a particular size, we are studying
the time complexity of the algorithm.
 When we analyze the computer memory the algorithm
uses to solve the problem given input of a particular size,
we are studying the space complexity of the algorithm.
The Complexity of Algorithms
 In this course, we focus on time complexity. The space
complexity of algorithms is studied in later courses.
 We will measure time complexity in terms of the number
of operations an algorithm uses and we will use big-O and
big-Theta notation to estimate the time complexity.
 We can use this analysis to see whether it is practical to use
this algorithm to solve problems with input of a particular
size. We can also compare the efficiency of different
algorithms for solving the same problem.
 We ignore implementation details (including the data
structures used and both the hardware and software
platforms) because it is extremely complicated to consider
them.
Time Complexity
 To analyze the time complexity of algorithms, we determine the
number of operations, such as comparisons and arithmetic
operations (addition, multiplication, etc.). We can estimate the
time a computer may actually use to solve a problem using the
amount of time required to do basic operations.
 We ignore minor details, such as the “house keeping” aspects of
the algorithm.
 We will focus on the worst-case time complexity of an algorithm.
This provides an upper bound on the number of operations an
algorithm uses to solve a problem with input of a particular size.
 It is usually much more difficult to determine the average case
time complexity of an algorithm. This is the average number of
operations an algorithm uses to solve a problem over all inputs of
a particular size.
Complexity Analysis of Algorithms
Example: Describe the time complexity of the algorithm
for finding the maximum element in a finite sequence.
procedure max(a1, a2, …., an: integers)
max := a1
for i := 2 to n
if max < ai then max := ai
return max{max is the largest element}

Solution: Count the number of comparisons.

• The max < ai comparison is made n
• i n.
• One last comparison determines that i > n.
• (n

Θ(n).
Worst-Case Complexity of Linear Search
Example: Determine the time complexity of the
linear search algorithm.
procedure linear search(x:integer,
a1, a2, …,an: distinct integers)
i := 1
while (i ≤ n and x ≠ ai)
i := i + 1
if i ≤ n then location := i
else location := 0
return location{location is the subscript of the term that equals x, or is 0 if
x is not found}

Solution: Count the number of comparisons.

• At each step two comparisons are made; i n and x ≠ ai .
• To end the loop, one comparison i n is made.
• After the loop, one more i n comparison is made.
If x = ai , i + comparisons are used. If x is not on the list, n +
then an additional comparison is used to
exit the loop. So, in the worst case n +
Hence, the complexity is Θ(n).
Average-Case Complexity of Linear Search
Example: Describe the average case performance of the
linear search algorithm. (Although usually it is very
difficult to determine average-case complexity, it is easy for
linear search.)
Solution: Assume the element is in the list and that the
possible positions are equally likely. By the argument on
the previous slide, if x = ai , the number of comparisons is
i+
 Worst-Case Complexity of Binary Search
Example: Describe the time complexity of binary
search in terms of the number of comparisons used.
procedure binary search(x: integer, a1,a2,…, an: increasing integers)
i := 1 {i is the left endpoint of interval}
j := n {j is right endpoint of interval}
while i < j
m := ⌊(i + j)/2⌋
if x > am then i := m + 1
else j := m
if x = ai then location := i
else location := 0
return location{location is the subscript i of the term ai equal to x, or 0 if x is not found}

Solution: Assume (for simplicity) n = k elements. Note that k = log n.

• Two comparisons are made at each stage; i < j, and x > am .
• At the first iteration the size of the list is k and after the first iteration it is k-1 Then k-2

and so on until the size of the list is 1 = .

• At the last step, a comparison tells us that the size of the list is the size is 0 = and the
element is compared with the single remaining element.
• Hence, at most k + = log n + comparisons are made.
• Therefore, the time complexity is Θ (log n), better than linear search.
Worst-Case Complexity of Bubble Sort
Example: What is the worst-case complexity of bubble
sort in terms of the number of comparisons made?
procedure bubblesort(a1,…,an: real numbers
with n ≥ )
for i := to n
for j := to n i
if aj >aj+1 then interchange aj and aj+1
{a1,…, an is now in increasing order}
Solution: A sequence of n passes is made through the list. On each pass n i
comparisons are made.

Θ(n2) since .
Worst-Case Complexity of Insertion Sort
Example: What is the worst-case complexity of
insertion sort in terms of the number of comparisons
made? procedure insertion sort(a1,…,an:
real numbers with n ≥ )
Solution: The total number of for j := to n
comparisons are: i :=
while aj > ai
i := i +
m := aj
Therefore the complexity is Θ(n2). for k := to j i
aj-k := aj-k-1
ai := m
Matrix Multiplication Algorithm
 The definition for matrix multiplication can be expressed
as an algorithm; C = A B where C is an m n matrix that is
the product of the m k matrix A and the k n matrix B.
 This algorithm carries out matrix multiplication based on
its definition.

procedure matrix multiplication(A,B: matrices)

for i := to m
for j := to n
cij :=
for q := to k
cij := cij + aiq bqj
return C{C = [cij] is the product of A and B}
Complexity of Matrix Multiplication
Example: How many additions of integers and
multiplications of integers are used by the matrix
multiplication algorithm to multiply two n n
matrices.
Solution: There are n
n n
n n n

O(n ).
Boolean Product Algorithm
 The definition of Boolean product of zero-one
matrices can also be converted to an algorithm.

procedure Boolean product(A,B: zero-one matrices)

for i := to m
for j := to n
cij :=
for q := to k
cij := cij aiq bqj)
return C{C = [cij] is the Boolean product of A and B}
Complexity of Boolean Product
Algorithm
Example: How many bit operations are used to find
A B, where A and B are n n zero-one matrices?
Solution: There are n A B

O(n )
Algorithmic Paradigms
 An algorithmic paradigm is a a general approach
based on a particular concept for constructing
algorithms to solve a variety of problems.
 Greedy algorithms were introduced in Section .
 We discuss brute-force algorithms in this section.
 We will see divide-and-conquer algorithms (Chapter 8),
dynamic programming (Chapter 8), backtracking
(Chapter ), and probabilistic algorithms (Chapter ).
There are many other paradigms that you may see in
later courses.
Brute-Force Algorithms
 A brute-force algorithm is solved in the most
straightforward manner, without taking advantage of
any ideas that can make the algorithm more efficient.
 Brute-force algorithms we have previously seen are
sequential search, bubble sort, and insertion sort.
Computing the Closest Pair of
Points by Brute-Force
Example: Construct a brute-force algorithm for
finding the closest pair of points in a set of n points in
the plane and provide a worst-case estimate of the
number of arithmetic operations.
Solution: Recall that the distance between (xi,yi) and
(xj, yj) is . A brute-force algorithm
simply computes the distance between all pairs of
points and picks the pair with the smallest distance.
Note: There is no need to compute the square root, since the square of the
distance between two points is smallest when the distance is smallest.

Continued
Computing the Closest Pair of
Points by Brute-Force
 Algorithm for finding the closest pair in a set of n points.
procedure closest pair((x1, y1), (x2, y2), … (xn, yn): xi, yi real numbers)
min =
for i := to n
for j := to i
if (xj xi)2 + (yj yi)2 < min
then min := (xj xi)2 + (yj yi)2
closest pair := (xi, yi), (xj, yj)
return closest pair

 The algorithm loops through n(n )/ pairs of points, computes the value
(xj xi ) 2
+ (yj yi)2
and compares it with the minimum, etc. So, the algorithm
uses Θ(n2) arithmetic and comparison operations.
 We will develop an algorithm with O(log n) worst-case complexity in Section
.
Understanding the Complexity of
Algorithms
Understanding the Complexity of
Algorithms

Times of more than 100 years are indicated with an *.

Complexity of Problems
 Tractable Problem: There exists a polynomial time
algorithm to solve this problem. These problems are
said to belong to the Class P.
 Intractable Problem: There does not exist a
polynomial time algorithm to solve this problem
 Unsolvable Problem : No algorithm exists to solve this
problem, e.g., halting problem.
Exercises
 Algorithms
 Chapter 3.2. (7th Global Edition): 1, 9, 21, 28
 Chapter 3.3. (7th Global Edition): 2, 3, 7, 13, 29, 40

47
References
 Discrete Mathematics and Its Applications, Seventh Edition, Kenneth Rosen

48