Discrete Math lec5

• Determinants of a matrix
• Inverse of a matrix
• Analysis of Algorithms
 Note the difference in the matrix
Determinants and the determinant of the
matrix!

1 3
• Every square matrix has
a determinant. -½ 0
• The determinant of a
matrix is a number.
• We will consider the -3 8 ¼
determinants only of
2x2 and 3x3 matrices. 2 0 -¾

4 180 11
 Why do we need the determinant
• It is used to help us calculate
the inverse of a matrix and it
is used when finding the area
of a triangle
 Finding Determinants of Matrices

Notice the different symbol:

3 2
the straight lines tell you to
5 4
find the determinant!!

3 2
= (3 * 4) - (-5 * 2)
5 4
= 12 - (-10)
= 22
 Finding Determinants of Matrices
2 0 3 2 0
1 2 5 1 -2
1 4 2 -1 4

= [(2)(-2)(2) + (0)(5)(-1) + (3)(1)(4)]

- [(3)(-2)(-1) + (2)(5)(4) + (0)(1)(2)]
= [-8 + 0 +12] - [6 + 40 + 0]
= 4 – 6 - 40 = -42
 Using matrix equations
Identity matrix: Square matrix with 1’s on the diagonal
and zeros everywhere else

 1 0 2 x 2 identity matrix
 
 0 1
1 0 0
0 1 0
  3 x 3 identity matrix
0 0 1 

The identity matrix is to matrix multiplication as

___
1 is to regular multiplication!!!!
Multiply:

 1 0  5  2 5  2
  3 4 = 
3

4
 0 1

5  2  1 0 5  2
    =  
3 4  0 1 3 4

So, the identity matrix multiplied by any matrix

lets the “any” matrix keep its identity!

Mathematically, IA = A and AI = A !!
 Using matrix equations

Inverse Matrix: 2 x 2
 a b
A   
 c d
1 1  d  b
A  ad  bc  c a 

In words:
•Take the original matrix.
•Switch a and d.
•Change the signs of b and c.
•Multiply the new matrix by 1 over the determinant of the original matrix.
 Using matrix equations

Example: Find the inverse of A.

 2 4
A   
 4  10

1 1  10  4
A  (2)( 10)  ( 4)( 4)  4 2 

5 1 
1 1  10  4
A   4  4 2  =
2 
 1
 1  
 2
Find the inverse matrix.

 8  3 
1  Matrix 

  Inverse =
 5 2 det  Reloaded
 
Matrix A

Det A = 8(2) – (-5)(-3) = 16 – 15 = 1

1 2 3 2 3
=
1   =  
5 8 5 8
 What happens when you multiply a matrix by its inverse?

1
1st: What happens when you multiply a number by its inverse? 7
7

A & B are inverses. Multiply them.

 8  3 2 3  1 0
    =
 
 5 2 5 8  0 1

So, AA-1 = I
 Algorithms
•What is an algorithm?

•An algorithm is a finite set of precise instructions for

performing a computation or for solving a problem.

12
 Algorithms
• Properties of algorithms:

• Input from a specified set,

• Output from a specified set (solution),
• Definiteness of every step in the computation,
• Correctness of output for every possible input,
• Finiteness of the number of calculation steps,
• Effectiveness of each calculation step and
• Generality for a class of problems.

13
 Algorithm Examples
•We will use a pseudocode to specify algorithms, which
slightly reminds us of Basic and Pascal.
•Example: an algorithm that finds the maximum element in
a finite sequence

procedure max(a1, a2, …, an: integers)

max := a1
for i := 2 to n
if max < ai then max := ai
{max is the largest element}

14
 Algorithm Examples
•Another example: a linear search algorithm, that is, an
algorithm that linearly searches a sequence for a particular
element.
procedure linear_search(x: integer; a1, a2, …, an:
integers)
i := 1
while (i  n and x  ai)
i := i + 1
if i  n then location := i
else location := 0
{location is the subscript of the term that equals x, or is
zero if x is not found}
15
 Algorithm Examples

•If the terms in a sequence are ordered, a binary search

algorithm is more efficient than linear search.

•The binary search algorithm iteratively restricts the

relevant search interval until it closes in on the position of
the element to be located.

16
Algorithm Examples

binary search for the letter ‘j’

search interval

a c d f g h j l m o p r s u v x z

center element

17
 Algorithm Examples

binary search for the letter ‘j’

search interval

a c d f g h j l m o p r s u v x z

center element

18
Algorithm Examples

binary search for the letter ‘j’

search interval

a c d f g h j l m o p r s u v x z

center element

19
Algorithm Examples

binary search for the letter ‘j’

search interval

a c d f g h j l m o p r s u v x z

center element

20
Algorithm Examples

binary search for the letter ‘j’

search interval

a c d f g h j l m o p r s u v x z

center element

found !
21
 Algorithm Examples
procedure binary_search(x: integer; a1, a2, …, an:
integers)
i := 1 {i is left endpoint of search interval}
j := n {j is right endpoint of search interval}
while (i < j)
begin
m := (i + j)/2
if x > am then i := m + 1
else j := m
end
if x = ai then location := i
else location := 0
{location is the subscript of the term that equals x, or is
zero if x is not found}
22
 Bubble Sort
• The bubble sort is one of the simplest sorting algorithms,
but not one of the most efficient.
• It puts a list into increasing order by successively
comparing adjacent elements, interchanging them if they
are in the wrong order.
 Bubble Sort
• To carry out the bubble sort, we perform the basic
operation, that is, interchanging a larger element with a
smaller one following it, starting at the beginning of the list,
for a full pass.
• We iterate this procedure until the sort is complete.
 Bubble Sort
• Pseudocode for the bubble sort is given as Algorithm 4.
• We can imagine the elements in the list placed in a column.
• In the bubble sort, the smaller elements “bubble” to the top
as they are interchanged with larger elements.
• The larger elements “sink” to the bottom.
Bubble Sort
Bubble Sort
Bubble Sort
 Insertion sort
• The insertion sort is a simple sorting algorithm, but it is
usually not the most efficient.
• To sort a list with n elements, the insertion sort begins with
the second element.
• The insertion sort compares this second element with the
first element and inserts it before the first element if it does
not exceed the first element and after the first element if it
exceeds the first element.
 Insertion sort
• At this point, the first two elements are in the correct order.
• The third element is then compared with the first element,
and if it is larger than the first element, it is compared with
the second element; it is inserted into the correct position
among the first three elements.
 Insertion sort
• In general, in the j th step of the insertion sort, the j th
element of the list is inserted into the correct position in the
list of the previously sorted j − 1 elements.
• To insert the j th element in the list, a linear search
technique is used; the j th element is successively compared
with the already sorted j − 1 elements at the start of the list
until the first element that is not less than this element is
found or until it has been compared with all j − 1 elements;
the j th element is inserted in the correct position so that the
first j elements are sorted.
 Insertion sort
• The algorithm continues until the last element is placed in
the correct position relative to the already sorted list of the
first n − 1 elements. The insertion sort is described in
pseudocode in Algorithm 5.
 Insertion sort
• Use the insertion sort to put the elements of the list 3, 2, 4,
1, 5 in increasing order.
• Solution: The insertion sort first compares 2 and 3. Because
3 > 2, it places 2 in the first position, producing the list 2, 3,
4, 1, 5 (the sorted part of the list is shown in color). At this
point, 2 and 3 are in the correct order.
 Insertion sort
• Next, it inserts the third element, 4, into the already sorted
part of the list by making the comparisons 4 > 2 and 4 > 3.
• Because 4 > 3, 4 remains in the third position.
• At this point, the list is 2, 3, 4, 1, 5 and we know that the
ordering of the first three elements is correct.
• Next, we find the correct place for the fourth element, 1,
among the already sorted elements, 2, 3, 4.
• Because 1 < 2, we obtain the list 1, 2, 3, 4, 5.
 Insertion sort
• Finally, we insert 5 into the correct position by successively
comparing it to 1, 2, 3, and 4.
• Because 5 > 4, it stays at the end of the list, producing the
correct order for the entire list
Insertion sort
 What’s next?
• The growth of functions
• Complexity of Algorithms
• Big-O notation
 Growth of functions
Objectives:
• study the number of operations used by these algorithms.
• estimate the number of comparisons used by the linear and
binary search algorithms to find an element in a sequence of
n elements.
 Growth of functions
• The time required to solve a problem depends on more than
the number of operations it uses.
• The time also depends on the hardware and software used to
run the program that implements the algorithm.
• However, when we change the hardware and software used
to implement an algorithm, we can closely approximate the
time required to solve a problem of size n by multiplying
the previous time required by a constant.
 Growth of functions
• One of the advantages of using big-O notation is that we
can estimate the growth of a function without worrying
about constant multipliers or smaller order terms.
• This means that, using big-O notation, we do not have to
worry about the hardware and software used to implement
an algorithm.
• Furthermore, using big-O notation, we can assume that the
different operations used in an algorithm take the same
time, which simplifies the analysis considerably
 Growth of functions
• Big-O notation is used extensively to estimate the number
of operations an algorithm uses as its input grows.
• With the help of this notation, we can determine whether it
is practical to use a particular algorithm to solve a problem
as the size of the input increases.
 Growth of functions
• Furthermore, using big-O notation, we can compare two
algorithms to determine which is more efficient as the size
of the input grows.
• For instance, if we have two algorithms for solving a
problem, one using 100n2 + 17n + 4 operations and the
other using n3 operations, big-O notation can help us see
that the first algorithm uses far fewer operations when n is
large, even though it uses more operations for small values
of n, such as n = 10.
 Complexity

•In general, we are not so much interested in the time and

space complexity for small inputs.

•For example, while the difference in time complexity

between linear and binary search is meaningless for a
sequence with n = 10, it is gigantic for n = 230.

43
 Complexity

•For example, let us assume two algorithms A and B that

solve the same class of problems.
•The time complexity of A is 5,000n, the one for B is
1.1n for an input with n elements.
•For n = 10, A requires 50,000 steps, but B only 3, so B
seems to be superior to A.
•For n = 1000, however, A requires 5,000,000 steps,
while B requires 2.51041 steps.

44
 Complexity

•This means that algorithm B cannot be used for large

inputs, while algorithm A is still feasible.

•So what is important is the growth of the complexity

functions.

•The growth of time and space complexity with

increasing input size n is a suitable measure for the
comparison of algorithms.

45
 Complexity
• Comparison: time complexity of algorithms A and B

Input Size Algorithm A Algorithm B

n 5,000n 1.1n

10 50,000 3

100 500,000 13,781

1,000 5,000,000 2.5.1041

1,000,000 5x109 4.8.1041392

46
 Complexity

•This means that algorithm B cannot be used for large

inputs, while running algorithm A is still feasible.

•So what is important is the growth of the complexity

functions.

•The growth of time and space complexity with

increasing input size n is a suitable measure for the
comparison of algorithms.

47
 The Growth of Functions
•The growth of functions is usually described using the
big-O notation.

•Definition: Let f and g be functions from the integers or

the real numbers to the real numbers.
•We say that f(x) is O(g(x)) if there are constants C and k
such that
•|f(x)|  C|g(x)|
•whenever x > k.

48
 The Growth of Functions

•When we analyze the growth of complexity functions,

f(x) and g(x) are always positive.
•Therefore, we can simplify the big-O requirement to
•f(x)  Cg(x) whenever x > k.

•If we want to show that f(x) is O(g(x)), we only need to

find one pair (C, k) (which is never unique).

49
 The Growth of Functions
•The idea behind the big-O notation is to establish an
upper boundary for the growth of a function f(x) for
large x.
•This boundary is specified by a function g(x) that is
usually much simpler than f(x).
•We accept the constant C in the requirement
•f(x)  Cg(x) whenever x > k,
•because C does not grow with x.
•We are only interested in large x, so it is OK if
f(x) > Cg(x) for x  k.

50
 The Growth of Functions
•Example:
•Show that f(x) = x2 + 2x + 1 is O(x2).

•For x > 1 we have:

•x2 + 2x + 1  x2 + 2x2 + x2
• x2 + 2x + 1  4x2
•Therefore, for C = 4 and k = 1:
•f(x)  Cx2 whenever x > k.

• f(x) is O(x2).

51
 The Growth of Functions
•Example:
•Show that 7x2 is O(x3).
•For x > 7 we have:
•7x2  x3
•Therefore, for C = 1 and k = 7:
•f(x)  Cx2 whenever x > k.

• f(x) is O(x3).

52
 The Growth of Functions

•Question: If f(x) is O(x2), is it also O(x3)?

•Yes. x3 grows faster than x2, so x3 grows also faster than

f(x).

•Therefore, we always have to find the smallest simple

function g(x) for which f(x) is O(g(x)).

53
 The Growth of Functions
“Popular” functions g(n) are
n log n, 1, 2n, n2, n!, n, n3, log n

Listed from slowest to fastest growth:

• 1
• log n
• n
• n log n
• n2
• n3
• 2n
• n!

54
 The Growth of Functions

•A problem that can be solved with polynomial worst-case

complexity is called tractable.

•Problems of higher complexity are called intractable.

•Problems that no algorithm can solve are called

unsolvable.

55
 Useful Rules for Big-O
For any polynomial f(x) = anxn + an-1xn-1 + … + a0, where
a0, a1, …, an are real numbers,
f(x) is O(xn).

If f1(x) is O(g1(x)) and f2(x) is O(g2(x)), then

(f1 + f2)(x) is O(max(g1(x), g2(x)))

If f1(x) is O(g(x)) and f2(x) is O(g(x)), then

(f1 + f2)(x) is O(g(x)).

If f1(x) is O(g1(x)) and f2(x) is O(g2(x)), then

(f1f2)(x) is O(g1(x) g2(x)).
56
 Complexity Examples
What does the following algorithm compute?
procedure who_knows(a1, a2, …, an: integers)
m := 0
for i := 1 to n-1
for j := i + 1 to n
if |ai – aj| > m then m := |ai – aj|
{m is the maximum difference between any two numbers in
the input sequence}
Comparisons: n-1 + n-2 + n-3 + … + 1
= (n – 1)n/2 = 0.5n2 – 0.5n

Time complexity is O(n2).

57
 Complexity Examples
Another algorithm solving the same problem:
procedure max_diff(a1, a2, …, an: integers)
min := a1
max := a1
for i := 2 to n
if ai < min then min := ai
else if ai > max then max := ai
m := max - min
Comparisons: 2n - 2
Time complexity is O(n).

58
 Analysis of Algorithms

Input Algorithm Output

An algorithm is a step-by-step procedure for

solving a problem in a finite amount of time.
© 2004 Goodrich, Tamassia
Running Time (§3.1)
Most algorithms transform best case
average case
input objects into output worst case
objects. 120

The running time of an 100

algorithm typically grows

Running Time
80
with the input size.
60
Average case time is often
difficult to determine. 40

We focus on the worst case 20

running time. 0
1000 2000 3000 4000
 Easier to analyze Input Size
 Crucial to applications such as
games, finance and robotics

© 2004 Goodrich, Tamassia Analysis of Algorithms 60

Experimental Studies
Write a program 9000

implementing the 8000

algorithm 7000

Run the program with 6000

Time (ms)
inputs of varying size and 5000
composition 4000
Use a method like 3000
System.currentTimeMillis() to 2000
get an accurate measure
1000
of the actual running time
0
Plot the results 0 50 100
Input Size

© 2004 Goodrich, Tamassia Analysis of Algorithms 61

Limitations of Experiments
It is necessary to implement the
algorithm, which may be difficult
Results may not be indicative of the
running time on other inputs not included
in the experiment.
In order to compare two algorithms, the
same hardware and software
environments must be used

© 2004 Goodrich, Tamassia Analysis of Algorithms 62

Theoretical Analysis
Uses a high-level description of the
algorithm instead of an implementation
Characterizes running time as a function
of the input size, n.
Takes into account all possible inputs
Allows us to evaluate the speed of an
algorithm independent of the
hardware/software environment
© 2004 Goodrich, Tamassia Analysis of Algorithms 63
Pseudocode (§3.2)
High-level description Example: find max
of an algorithm element of an array
More structured than Algorithm arrayMax(A, n)
English prose Input array A of n integers
Less detailed than a Output maximum element of A
program
Preferred notation for currentMax  A[0]
describing algorithms for i  1 to n  1 do
Hides program design if A[i]  currentMax then
issues currentMax  A[i]
return currentMax

© 2004 Goodrich, Tamassia Analysis of Algorithms 64

Pseudocode Details
Control flow Method call
 if … then … [else …] var.method (arg [, arg…])
 while … do … Return value
 repeat … until … return expression
 for … do … Expressions
 Indentation replaces braces  Assignment
(like  in Java)
Method declaration  Equality testing
Algorithm method (arg [, arg…]) (like  in Java)
Input … n2 Superscripts and other
Output … mathematical
formatting allowed

© 2004 Goodrich, Tamassia Analysis of Algorithms 65

The Random Access Memory (RAM)
Model
A CPU

An potentially unbounded
bank of memory cells, 1
2
each of which can hold an 0
arbitrary number or
character
Memory cells are numbered and accessing
any cell in memory takes unit time.
© 2004 Goodrich, Tamassia Analysis of Algorithms 66
Primitive Operations
Basic computations
Examples:
performed by an algorithm
 Evaluating an
Identifiable in pseudocode expression
Largely independent from the  Assigning a value
to a variable
programming language
 Indexing into an
Exact definition not important array
(we will see why later)  Calling a method
Returning from a
Assumed to take a constant 

method
amount of time in the RAM
model

© 2004 Goodrich, Tamassia Analysis of Algorithms 67

 Counting Primitive
Operations (§3.4)
By inspecting the pseudocode, we can determine the
maximum number of primitive operations executed by
an algorithm, as a function of the input size
Algorithm arrayMax(A, n) # operations
currentMax  A[0] 2
for i  1 to n  1 do 2n
if A[i]  currentMax then 2(n  1)
currentMax  A[i] 2(n  1)
{ increment counter i } 2(n  1)
return currentMax 1
Total 8n  2

© 2004 Goodrich, Tamassia Analysis of Algorithms 68

Estimating Running Time
Algorithm arrayMax executes 8n  2 primitive
operations in the worst case. Define:
a = Time taken by the fastest primitive operation
b = Time taken by the slowest primitive operation
Let T(n) be worst-case time of arrayMax. Then
a (8n  2)  T(n)  b(8n  2)
Hence, the running time T(n) is bounded by two
linear functions

© 2004 Goodrich, Tamassia Analysis of Algorithms 69

Growth Rate of Running Time
Changing the hardware/ software
environment
 Affects T(n) by a constant factor, but
 Does not alter the growth rate of T(n)
The linear growth rate of the running
time T(n) is an intrinsic property of
algorithm arrayMax

© 2004 Goodrich, Tamassia Analysis of Algorithms 70

Seven Important Functions (§3.3)
Seven functions that
often appear in 1E+29
algorithm analysis: 1E+27 Cubic

 Constant  1 1E+25
Quadratic
1E+23
 Logarithmic  log n 1E+21 Linear
 Linear  n 1E+19
 N-Log-N  n log n 1E+17

T(n)
1E+15
 Quadratic  n2 1E+13
 Cubic  n3 1E+11
 Exponential  2n 1E+9
1E+7
1E+5
In a log-log chart, the 1E+3
slope of the line 1E+1

corresponds to the 1E-1

1E-1 1E+2 1E+5 1E+8
growth rate of the n
function
© 2004 Goodrich, Tamassia Analysis of Algorithms 71
Constant Factors
The growth rate is 1E+25
1E+23
Quadratic

not affected by 1E+21

Quadratic
Linear
1E+19
 constant factors or 1E+17 Linear

 lower-order terms 1E+15

T(n)
1E+13
Examples 1E+11
 102n + 105 is a linear 1E+9
1E+7
function 1E+5
 105n2 + 108n is a 1E+3
quadratic function 1E+1
1E-1
1E-1 1E+1 1E+3 1E+5 1E+7 1E+9

© 2004 Goodrich, Tamassia Analysis of Algorithms 72

Big-Oh Notation (§3.4)
10,000
Given functions f(n) and 3n

g(n), we say that f(n) is 2n+10

1,000
O(g(n)) if there are
positive constants n

c and n0 such that 100

f(n)  cg(n) for n  n0

10
Example: 2n + 10 is O(n)
 2n + 10  cn
1
 (c  2) n  10 1 10 100 1,000
 n  10/(c  2) n
 Pick c  3 and n0  10

© 2004 Goodrich, Tamassia Analysis of Algorithms 73

Big-Oh Example
1,000,000
n^2
Example: the function 100n
100,000
n2 is not O(n) 10n

 n2  cn 10,000 n

 nc
 The above inequality 1,000
cannot be satisfied
since c must be a 100
constant
10

1
1 10 100 1,000
n

© 2004 Goodrich, Tamassia Analysis of Algorithms 74

 More Big-Oh Examples
7n-2
7n-2 is O(n)
need c > 0 and n0  1 such that 7n-2  c•n for n  n0
this is true for c = 7 and n0 = 1

 3n3 + 20n2 + 5
3n3 + 20n2 + 5 is O(n3)
need c > 0 and n0  1 such that 3n3 + 20n2 + 5  c•n3 for n  n0
this is true for c = 4 and n0 = 21

 3 log n + 5
3 log n + 5 is O(log n)
need c > 0 and n0  1 such that 3 log n + 5  c•log n for n  n0
this is true for c = 8 and n0 = 2
© 2004 Goodrich, Tamassia Analysis of Algorithms 75
Big-Oh and Growth Rate
The big-Oh notation gives an upper bound on the
growth rate of a function
The statement “f(n) is O(g(n))” means that the growth
rate of f(n) is not more than the growth rate of g(n)
We can use the big-Oh notation to rank functions
according to their growth rate

f(n) is O(g(n)) g(n) is O(f(n))

g(n) grows more Yes No
f(n) grows more No Yes
Same growth Yes Yes
© 2004 Goodrich, Tamassia Analysis of Algorithms 76
 Big-Oh Rules

If is f(n) a polynomial of degree d, then f(n) is

O(nd), i.e.,
1. Drop lower-order terms
2. Drop constant factors
Use the smallest possible class of functions
 Say “2n is O(n)” instead of “2n is O(n2)”
Use the simplest expression of the class
 Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”

© 2004 Goodrich, Tamassia Analysis of Algorithms 77

Asymptotic Algorithm Analysis
The asymptotic analysis of an algorithm determines
the running time in big-Oh notation
To perform the asymptotic analysis
 We find the worst-case number of primitive operations
executed as a function of the input size
 We express this function with big-Oh notation
Example:
 We determine that algorithm arrayMax executes at most
8n  2 primitive operations
 We say that algorithm arrayMax “runs in O(n) time”
Since constant factors and lower-order terms are
eventually dropped anyhow, we can disregard them
when counting primitive operations

© 2004 Goodrich, Tamassia Analysis of Algorithms 78

Computing Prefix Averages
We further illustrate 35
asymptotic analysis with X
two algorithms for prefix 30 A
averages 25
The i-th prefix average of 20
an array X is average of the
first (i + 1) elements of X: 15
A[i]  (X[0] + X[1] + … + X[i])/(i+1) 10

Computing the array A of 5

prefix averages of another 0
array X has applications to 1 2 3 4 5 6 7
financial analysis

© 2004 Goodrich, Tamassia Analysis of Algorithms 79

 Prefix Averages (Quadratic)
The following algorithm computes prefix averages in
quadratic time by applying the definition
Algorithm prefixAverages1(X, n)
Input array X of n integers
Output array A of prefix averages of X #operations
A  new array of n integers n
for i  0 to n  1 do n
s  X[0] n
for j  1 to i do 1 + 2 + …+ (n  1)
s  s + X[j] 1 + 2 + …+ (n  1)
A[i]  s / (i + 1) n
return A 1
© 2004 Goodrich, Tamassia Analysis of Algorithms 80
Arithmetic Progression
7
The running time of
6
prefixAverages1 is
O(1 + 2 + …+ n) 5
The sum of the first n 4
integers is n(n + 1) / 2
3
 There is a simple visual
proof of this fact 2
Thus, algorithm 1
prefixAverages1 runs in 0
O(n2) time
1 2 3 4 5 6

© 2004 Goodrich, Tamassia Analysis of Algorithms 81

 Prefix Averages (Linear)
The following algorithm computes prefix averages in
linear time by keeping a running sum
Algorithm prefixAverages2(X, n)
Input array X of n integers
Output array A of prefix averages of X #operations
A  new array of n integers n
s0 1
for i  0 to n  1 do n
s  s + X[i] n
A[i]  s / (i + 1) n
return A 1
Algorithm prefixAverages2 runs in O(n) time
© 2004 Goodrich, Tamassia Analysis of Algorithms 82
Math you need to Review
Summations
Logarithms and Exponents
properties of logarithms:
logb(xy) = logbx + logby
logb (x/y) = logbx - logby
logbxa = alogbx
logba = logxa/logxb
properties of exponentials:
a(b+c) = aba c
abc = (ab)c
Proof techniques
ab /ac = a(b-c)
Basic probability b = a logab
bc = a c*logab

© 2004 Goodrich, Tamassia Analysis of Algorithms 83

 Relatives of Big-Oh
big-Omega
 f(n) is (g(n)) if there is a constant c > 0

and an integer constant n0  1 such that

f(n)  c•g(n) for n  n0

big-Theta
 f(n) is (g(n)) if there are constants c’ > 0 and c’’
> 0 and an integer constant n0  1 such that
c’•g(n)  f(n)  c’’•g(n) for n  n0

© 2004 Goodrich, Tamassia Analysis of Algorithms 84

 Intuition for Asymptotic
Notation
Big-Oh
 f(n) is O(g(n)) if f(n) is asymptotically
less than or equal to g(n)
big-Omega
 f(n) is (g(n)) if f(n) is asymptotically
greater than or equal to g(n)
big-Theta
 f(n) is (g(n)) if f(n) is asymptotically
equal to g(n)

© 2004 Goodrich, Tamassia Analysis of Algorithms 85

 Example Uses of the
Relatives of Big-Oh
 5n2 is (n2)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0  1
such that f(n)  c•g(n) for n  n0
let c = 5 and n0 = 1
 5n2 is (n)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0  1
such that f(n)  c•g(n) for n  n0
let c = 1 and n0 = 1
 5n2 is (n2)
f(n) is (g(n)) if it is (n2) and O(n2). We have already seen the former,
for the latter recall that f(n) is O(g(n)) if there is a constant c > 0 and an
integer constant n0  1 such that f(n) < c•g(n) for n  n0
Let c = 5 and n0 = 1

© 2004 Goodrich, Tamassia Analysis of Algorithms 86

 Time and space

 To analyze an algorithm means:

 developing a formula for predicting how fast an
algorithm is, based on the size of the input (time
complexity), and/or
 developing a formula for predicting how much memory
an algorithm requires, based on the size of the input
(space complexity)
 Usually time is our biggest concern
 Most algorithms require a fixed amount of space

87
 What does “size of the input” mean?
 If we are searching an array, the “size” of the input could be the size of the
array
 If we are merging two arrays, the “size” could be the sum of the two array
sizes
 If we are computing the nth Fibonacci number, or the nth factorial, the
“size” is n
 We choose the “size” to be the parameter that most influences the actual
time/space required
 It is usually obvious what this parameter is
 Sometimes we need two or more parameters

88
Characteristic operations
 In computing time complexity, one good approach
is to count characteristic operations
 What a “characteristic operation” is depends on the
particular problem
 If searching, it might be comparing two values
 If sorting an array, it might be:
 comparing two values
 swapping the contents of two array locations
 both of the above
 Sometimes we just look at how many times the
innermost loop is executed

89
Exact values
 It is sometimes possible, in assembly language, to
compute exact time and space requirements
 We know exactly how many bytes and how many cycles
each machine instruction takes
 For a problem with a known sequence of steps (factorial,
Fibonacci), we can determine how many instructions of
each type are required
 However, often the exact sequence of steps cannot
be known in advance
 The steps required to sort an array depend on the actual
numbers in the array (which we do not know in advance)

90
Higher-level languages
 In a higher-level language (such as Java), we do not
know how long each operation takes
 Which is faster, x < 10 or x <= 9 ?
 We don’t know exactly what the compiler does with this
 The compiler probably optimizes the test anyway
(replacing the slower version with the faster one)
 In a higher-level language we cannot do an exact
analysis
 Our timing analyses will use major oversimplifications
 Nevertheless, we can get some very useful results

91
 Average, best, and worst cases
 Usually we would like to find the average time to perform an algorithm
 However,
 Sometimes the “average” isn’t well defined
 Example: Sorting an “average” array
 Time typically depends on how out of order the array is

How out of order is the “average” unsorted array?



 Sometimes finding the average is too difficult

 Often we have to be satisfied with finding the worst (longest) time required
 Sometimes this is even what we want (say, for time-critical operations)
 The best (fastest) case is seldom of interest

92
Constant time
 Constant time means there is some constant k such
that this operation always takes k nanoseconds
 A Java statement takes constant time if:
 It does not include a loop

 It does not include calling a method whose time is

unknown or is not a constant

 If a statement involves a choice (if or switch)
among operations, each of which takes constant
time, we consider the statement to take constant time
 This is consistent with worst-case analysis

93
Linear time
 We may not be able to predict to the nanosecond how long a
Java program will take, but do know some things about
timing:
for (i = 0, j = 1; i < n; i++) {
j = j * i;
}
 This loop takes time k*n + c, for some constants k and c
k : How long it takes to go through the loop once
(the time for j = j * i, plus loop overhead)
n : The number of times through the loop
(we can use this as the “size” of the problem)
c : The time it takes to initialize the loop
 The total time k*n + c is linear in n

94
 Constant time is (usually)
better than linear time
 Suppose we have two algorithms to solve a task:
 Algorithm A takes 5000 time units
 Algorithm B takes 100*n time units
 Which is better?
 Clearly, algorithm B is better if our problem size is small,
that is, if n < 50
 Algorithm A is better for larger problems, with n > 50
 So B is better on small problems that are quick anyway
 But A is better for large problems, where it matters more
 We usually care most about very large problems
 But not always!
95
The array subset problem
 Suppose you have two sets, represented as unsorted arrays:
int[] sub = { 7, 1, 3, 2, 5 };
int[] super = { 8, 4, 7, 1, 2, 3, 9 };
and you want to test whether every element of the first set
(sub) also occurs in the second set (super):
System.out.println(subset(sub, super));
 (The answer in this case should be false, because sub
contains the integer 5, and super doesn’t)
 We are going to write method subset and compute its time
complexity (how fast it is)
 Let’s start with a helper function, member, to test whether
one number is in an array

96
member
static boolean member(int x, int[] a) {
int n = a.length;
for (int i = 0; i < n; i++) {
if (x == a[i]) return true;
}
return false;
}
 If x is not in a, the loop executes n times, where
n = a.length
 This is the worst case
 If x is in a, the loop executes n/2 times on average
 Either way, linear time is required: k*n+c
97
subset
 static boolean subset(int[] sub, int[] super) {
int m = sub.length;
for (int i = 0; i < m; i++)
if (!member(sub[i], super) return false;
return true;
}
 The loop (and the call to member) will execute:
 m = sub.length times, if sub is a subset of super
 This is the worst case, and therefore the one we are most interested in

 Fewer than sub.length times (but we don’t know how few)

 We would need to figure this out in order to compute average time complexity
 The worst case is a linear number of times through the loop
 But the loop body doesn’t take constant time, since it calls
member, which takes linear time
98
 Analysis of array subset algorithm
 We’ve seen that the loop in subset executes m = sub.length times (in
the worst case)
 Also, the loop in subset calls member, which executes in time linear in n
= super.length
 Hence, the execution time of the array subset method is m*n, along with
assorted constants
 We go through the loop in subset m times, calling member each time

 We go through the loop in member n times

 If m and n are similar, this is roughly quadratic

99
 What about the constants?
 Forget the constants!
 An added constant, f(n)+c, becomes less and less
important as n gets larger
 A constant multiplier, k*f(n), does not get less
important, but...
 Improving k gives a linear speedup (cutting k in half cuts the
time required in half)
 Improving k is usually accomplished by careful code
optimization, not by better algorithms
 We aren’t that concerned with only linear speedups!

100
Simplifying the formulae
 Throwing out the constants is one of two things we
do in analysis of algorithms
 By throwing out constants, we simplify 12n2 + 35 to
just n2
 Our timing formula is a polynomial, and may have
terms of various orders (constant, linear, quadratic,
cubic, etc.)
 We usually discard all but the highest-order term
 We simplify n2 + 3n + 5 to just n2

101
 Big O notation
 When we have a polynomial that describes the time requirements of an
algorithm, we simplify it by:
 Throwing out all but the highest-order term
 Throwing out all the constants
 If an algorithm takes 12n3+4n2+8n+35 time, we simplify this formula
to just n3
 We say the algorithm requires O(n3) time
 We call this Big O notation
 (More accurately, it’s Big , but we’ll talk about that later)

102
 Big O for subset algorithm
 Recall that, if n is the size of the set, and m is the size of the (possible)
subset:
 We go through the loop in subset m times, calling member each time
 We go through the loop in member n times
 Hence, the actual running time should be k*(m*n) + c, for some constants k
and c
 We say that subset takes O(m*n) time

103
 Can we justify Big O notation?
 Big O notation is a huge simplification; can we
justify it?
 It only makes sense for large problem sizes
 For sufficiently large problem sizes, the
highest-order term swamps all the rest!
 Consider R = x2 + 3x + 5 as x varies:
x= 0 x2 = 0 3x =0 5 =5 R= 5
x= 10 x2 = 100 3x = 30 5 =5 R= 135
x= 100 x2 = 10000 3x = 300 5 =5 R= 10,305
x= 1000 x2 = 1000000 3x = 3000 5 =5 R= 1,003,005
x= 10,000 x2 = 108 3x = 3*104 5 =5 R= 100,030,005
x= 100,000 x2 = 1010 3x = 3*105 5 =5 R= 10,000,300,005

104
y = x2 + 3x + 5, for x=1..10

105
y = x2 + 3x + 5, for x=1..20

106
Common time complexities
BETTER  O(1) constant time
 O(log n) log time
 O(n) linear time
 O(n log n) log linear time
 O(n2) quadratic time
 O(n3) cubic time
 O(2n) exponential time
WORSE

107
FINDING RUNNING TIME OF AN ALGORITHM /
ANALYZING AN ALGORITHM
 Running time is measured by number of steps/primitive
operations performed

 Steps means elementary operation like

 ,+, *,<, =, A[i] etc

 We will measure number of steps taken in term of size of

input

108
 SIMPLE EXAMPLE (1)

// Input: int A[N], array of N integers

// Output: Sum of all numbers in array A

int Sum(int A[], int N)

{
int s=0;
for (int i=0; i< N; i++)
s = s + A[i];
return s;
}

How should we analyse this?

109
 SIMPLE EXAMPLE (2)

// Input: int A[N], array of N integers

// Output: Sum of all numbers in array A

int Sum(int A[], int N){

int s=0; 1
for (int i=0; i< N; i++)
2 3 4
s = s + A[i];
5 1,2,8: Once
return s;
6 7
3,4,5,6,7: Once per each iteration
}
8 of for loop, N iteration
Total: 5N + 3
The complexity function of the
algorithm is : f(N) = 5N +3
110
SIMPLE EXAMPLE (3) GROWTH OF 5N+3
Estimated running time for different values of N:

N = 10 => 53 steps
N = 100 => 503 steps
N = 1,000 => 5003 steps
N = 1,000,000 => 5,000,003 steps

As N grows, the number of steps grow in linear proportion to N for

this function “Sum”

111
 WHAT DOMINATES IN PREVIOUS
EXAMPLE?

What about the +3 and 5 in 5N+3?

 As N gets large, the +3 becomes insignificant
 5 is inaccurate, as different operations require varying amounts of time and
also does not have any significant importance

What is fundamental is that the time is linear in N.

Asymptotic Complexity: As N gets large, concentrate on the

highest order term:
 Drop lower order terms such as +3
 Drop the constant coefficient of the highest order term i.e. N

112
 ASYMPTOTIC COMPLEXITY

 The 5N+3 time bound is said to "grow asymptotically"

like N

 This gives us an approximation of the complexity of the

algorithm

 Ignores lots of (machine dependent) details, concentrate

on the bigger picture

113
 COMPARING FUNCTIONS:
ASYMPTOTIC NOTATION
 Big Oh Notation: Upper bound

 Omega Notation: Lower bound

 Theta Notation: Tighter bound

114
 BIG OH NOTATION [1]

If f(N) and g(N) are two complexity functions, we say

f(N) = O(g(N))

(read "f(N) is order g(N)", or "f(N) is big-O of g(N)")

if there are constants c and N0 such that for N > N0,
f(N) ≤ c * g(N)
for all sufficiently large N.

115
BIG OH NOTATION [2]

116
O(F(N))

117
 EXAMPLE (2): COMPARING FUNCTIONS

 Which function is better? 4000

10 n2 Vs n3 3500

3000

2500

10 n^2
2000
n^3

1500

1000

500

0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

118
 COMPARING FUNCTIONS

 As inputs get larger, any algorithm of a smaller order will

be more efficient than an algorithm of a larger order

0.05 N2 = O(N2)
Time (steps)

3N = O(N)

Input (size)
N = 60
119
 BIG-OH NOTATION

 Even though it is correct to say “7n - 3 is O(n3)”, a better

statement is “7n - 3 is O(n)”, that is, one should make the
approximation as tight as possible

 Simple Rule:
Drop lower order terms and constant factors
7n-3 is O(n)
8n2log n + 5n2 + n is O(n2log n)

120
 BIG OMEGA NOTATION
 If we wanted to say “running time is at least…” we use Ω

 Big Omega notation, Ω, is used to express the lower bounds on a

function.

 If f(n) and g(n) are two complexity functions then we can say:

f(n) is Ω(g(n)) if there exist positive numbers c and n0 such that 0<=f(n)>=cΩ (n) for all n>=n0

121
 BIG THETA NOTATION
 If we wish to express tight bounds we use the theta notation, Θ

 f(n) = Θ(g(n)) means that f(n) = O(g(n)) and f(n) = Ω(g(n))

122
 WHAT DOES THIS ALL MEAN?

 If f(n) = Θ(g(n)) we say that f(n) and g(n) grow at the same
rate, asymptotically

 If f(n) = O(g(n)) and f(n) ≠ Ω(g(n)), then we say that f(n) is

asymptotically slower growing than g(n).

 If f(n) = Ω(g(n)) and f(n) ≠ O(g(n)), then we say that f(n) is

asymptotically faster growing than g(n).

123
 WHICH NOTATION DO WE USE?
 To express the efficiency of our algorithms which of the
three notations should we use?

 As computer scientist we generally like to express our

algorithms as big O since we would like to know the
upper bounds of our algorithms.

 Why?

 If we know the worse case then we can aim to improve it

and/or avoid it.
124
 PERFORMANCE CLASSIFICATION
f(n) Classification
1 Constant: run time is fixed, and does not depend upon n. Most instructions are executed once, or
only a few times, regardless of the amount of information being processed

log n Logarithmic: when n increases, so does run time, but much slower. Common in programs which
solve large problems by transforming them into smaller problems. Exp : binary Search

n Linear: run time varies directly with n. Typically, a small amount of processing is done on each
element. Exp: Linear Search

n log n When n doubles, run time slightly more than doubles. Common in programs which break a problem
down into smaller sub-problems, solves them independently, then combines solutions. Exp: Merge

n2 Quadratic: when n doubles, runtime increases fourfold. Practical only for small problems; typically
the program processes all pairs of input (e.g. in a double nested loop). Exp: Insertion Search

n3 Cubic: when n doubles, runtime increases eightfold. Exp: Matrix

2n Exponential: when n doubles, run time squares. This is often the result of a natural, “brute force”
solution. Exp: Brute Force.
Note: logn, n, nlogn, n2>> less Input>>Polynomial
n3, 2n>>high input>> non polynomial
125
 SIZE DOES MATTER[1]

What happens if we double the input size N?

N log2N 5N N log2N N2 2N
8 3 40 24 64 256
16 4 80 64 256 65536
32 5 160 160 1024 ~109
64 6 320 384 4096 ~1019
128 7 640 896 16384 ~1038
256 8 1280 2048 65536 ~1076

126
 COMPLEXITY CLASSES

Time (steps)

127
 SIZE DOES MATTER[2]

 Suppose a program has run time O(n!) and the run time for
n = 10 is 1 second

For n = 12, the run time is 2 minutes

For n = 14, the run time is 6 hours

For n = 16, the run time is 2 months

For n = 18, the run time is 50 years

For n = 20, the run time is 200 centuries

128
 STANDARD ANALYSIS TECHNIQUES

 Constant time statements

 Analyzing Loops

 Analyzing Nested Loops

 Analyzing Sequence of Statements

 Analyzing Conditional Statements

129
 CONSTANT TIME STATEMENTS

 Simplest case: O(1) time statements

 Assignment statements of simple data types

int x = y;

 Arithmetic operations:
x = 5 * y + 4 - z;

 Array referencing:
A[j] = 5;

 Array assignment:
 j, A[j] = 5;
130
 Most conditional tests:
if (x < 12) ...
 ANALYZING LOOPS[1]
 Any loop has two parts:
 How many iterations are performed?
 How many steps per iteration?

int sum = 0,j;

for (j=0; j < N; j++)
sum = sum +j;

 Loop executes N times (0..N-1)

 4 = O(1) steps per iteration

 Total time is N * O(1) = O(N*1) = O(N) 131

 ANALYZING LOOPS[2]

 What about this for loop?

int sum =0, j;
for (j=0; j < 100; j++)
sum = sum +j;

 Loop executes 100 times

 4 = O(1) steps per iteration

 Total time is 100 * O(1) = O(100 * 1) = O(100) = O(1)

132
 ANALYZING LOOPS – LINEAR LOOPS
 Example (have a look at this code segment):

 Efficiency is proportional to the number of iterations.

 Efficiency time function is :
f(n) = 1 + (n-1) + c*(n-1) +( n-1)
= (c+2)*(n-1) + 1
= (c+2)n – (c+2) +1
 Asymptotically, efficiency is : O(n)
133
 ANALYZING NESTED LOOPS[1]
 Treat just like a single loop and evaluate each level of nesting as
needed:

int j,k;
for (j=0; j<N; j++)
for (k=N; k>0; k--)
sum += k+j;

 Start with outer loop:

 How many iterations? N
 How much time per iteration? Need to evaluate inner loop

 Inner loop uses O(N) time

 Total time is N * O(N) = O(N*N) = O(N2) 134

 ANALYZING NESTED LOOPS[2]
 What if the number of iterations of one loop depends on the
counter of the other?

int j,k;
for (j=0; j < N; j++)
for (k=0; k < j; k++)
sum += k+j;

 Analyze inner and outer loop together:

 Number of iterations of the inner loop is:

 0 + 1 + 2 + ... + (N-1) = O(N2)
135
 HOW DID WE GET THIS ANSWER?
 When doing Big-O analysis, we sometimes have to compute a
series like: 1 + 2 + 3 + ... + (n-1) + n

 i.e. Sum of first n numbers. What is the complexity of this?

 Gauss figured out that the sum of the first n numbers is always:

136
 SEQUENCE OF STATEMENTS
 For a sequence of statements, compute their complexity
functions individually and add them up

 Total cost is O(n2) + O(n) +O(1) = O(n2)

137
 CONDITIONAL STATEMENTS
 What about conditional statements such as

if (condition)
statement1;
else
statement2;

 where statement1 runs in O(n) time and statement2 runs in O(n2)

time?

 We use "worst case" complexity: among all inputs of size n, what is

the maximum running time?
138

 The analysis for the example above is O(n2)

 DERIVING A RECURRENCE EQUATION
 So far, all algorithms that we have been analyzing have been non
recursive

 Example : Recursive power method

 If N = 1, then running time T(N) is 2

 However if N ≥ 2, then running time T(N) is the cost of each step taken plus time
required to compute power(x,n-1). (i.e. T(N) = 2+T(N-1) for N ≥ 2)
139

 How do we solve this? One way is to use the iteration method.

 ITERATION METHOD
 This is sometimes known as “Back Substituting”.

 Involves expanding the recurrence in order to see a pattern.

 Solving formula from previous example using the iteration method

:

 Solution : Expand and apply to itself :

Let T(1) = n0 = 2
T(N) = 2 + T(N-1)
= 2 + 2 + T(N-2)
= 2 + 2 + 2 + T(N-3)
= 2 + 2 + 2 + ……+ 2 + T(1)
= 2N + 2 remember that T(1) = n0 = 2 for N = 1
140

 So T(N) = 2N+2 is O(N) for last example.

Let’s call it a day!