0% found this document useful (0 votes)

43 views374 pages

20 Merged

10.e^logex grows faster than xlogex/10. To see this, take the limit of their ratio: 10.e^logex lim = ∞ x→∞ xlogex/10 Since the limit goes to infinity, 10.e^logex grows faster than xlogex/10.

Uploaded by

Psycho Mind

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

43 views374 pages

20 Merged

10.e^logex grows faster than xlogex/10. To see this, take the limit of their ratio: 10.e^logex lim = ∞ x→∞ xlogex/10 Since the limit goes to infinity, 10.e^logex grows faster than xlogex/10.

Uploaded by

Psycho Mind

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 374

Basics of Algorithm Analysis

History
Al-khwarizmi (Persian) introduced the
concept of algorithm.
Also founded the discipline of algebra.
Around 800 CE

Analysis of Algorithms 2
What is an algorithm?
Definition:

Example: Suppose you have 8 balls …

Analysis of Algorithms 3
We analyze an algorithm on the basis of
its following resource requirements.

1. Time required.
2. Space required.

Analysis of Algorithms 4
Theoretical Analysis
Describe the algorithm in pseudo code.
Count the number of pseudo code
steps.
Characterize the running time as a
function of the input size, n.

Analysis of Algorithms 5
Example
The algorithm below finds the maximum element in an
array of size n.

Algorithm arrayMax(A, n) # operations

currentMax ← A[0] 1
for i ← 1 to n − 1 do (n – 1)
if A[i] > currentMax then (n − 1)
currentMax ← A[i] (n − 1)

return currentMax 1
Total 3n − 1

Analysis of Algorithms 6
Best case vs worst case
What input array will lead to best case performance.
What input array will lead to worst case performance.

Algorithm arrayMax(A, n) # operations

currentMax ← A[0] 1
for i ← 1 to n − 1 do (n – 1)
if A[i] > currentMax then (n − 1)
currentMax ← A[i] (n − 1)

return currentMax 1
Total 3n − 1

Analysis of Algorithms 7
Big-Oh Notation
Given functions f(n) and g(n), we say that f(n) is
O(g(n)) if there are positive constants
c > 0 and n0 >=0 such that
f(n) ≤ cg(n) for n ≥ n0

Example: 2n + 10 is O(n)

How ?

Analysis of Algorithms 8
Big-Oh Notation
10,000
3n
Example: 2n + 10 is O(n)
1,000 2n+10
 2n + 10 ≤ cn
n
 (c − 2) n ≥ 10
 n ≥ 10/(c − 2) 100

 Pick c = 3 and n0 = 10
10

1
1 10 100 1,000
n

Analysis of Algorithms 9
Big-Oh Example
Example: the function n2 is not O(n)

Why ?

Analysis of Algorithms 10
Big-Oh Example
1,000,000
n^2
Example: the function 100n
n2 is not O(n)
100,000
10n
 n2 ≤ cn 10,000 n

 n≤c
 The above inequality 1,000
cannot be satisfied
since c must be a 100
constant
10

1
1 10 100 1,000
n

Analysis of Algorithms 11
More Big-Oh Examples
7n-2
7n-2 is O(n)
need c > 0 and n0 ≥ 1 such that 7n-2 ≤ c•n for n ≥ n0
this is true for c = 7 and n0 = 1

 3n3 + 20n2 + 5
3n3 + 20n2 + 5 is O(n3)
need c > 0 and n0 ≥ 1 such that 3n3 + 20n2 + 5 ≤ c•n3 for n ≥ n0
this is true for c = 4 and n0 = 21

 3 log n + log log n

3 log n + log log n is O(log n)
need c > 0 and n0 ≥ 1 such that 3 log n + log log n ≤ c•log n for n ≥ n0
this is true for c = 4 and n0 = 2
Analysis of Algorithms 12
Big-Oh and Growth Rate
The big-Oh notation gives an upper bound on the
growth rate of a function
The statement “f(n) is O(g(n))” means that the growth
rate of f(n) is no more than the growth rate of g(n)
We can use the big-Oh notation to rank functions
according to their growth rate,

e.g. log n, log2n, √n, n, n3 , 5n

Analysis of Algorithms 13
Big-Oh Rules

If f(n) is a polynomial of degree d, then f(n) is

O(nd), i.e.,
1. Drop lower-order terms
2. Drop constant factors
Use the smallest possible class of functions
 Say “2n is O(n)” instead of “2n is O(n2)”
Use the simplest expression of the class
 Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”

Analysis of Algorithms 14
Example revisited
 We say that algorithm arrayMax “runs in O(n) time”

Algorithm arrayMax(A, n) # operations

currentMax ← A[0] 1
for i ← 1 to n − 1 do (n – 1)
if A[i] > currentMax then (n − 1)
currentMax ← A[i] (n − 1)

return currentMax 1
Total 3n − 1

Analysis of Algorithms 15
What constitutes a fast
algorithm ?
O(nx) is considered fast. (x > 0)

O(xn) is considered slow. (x > 1)

Analysis of Algorithms 16
What constitutes a fast
algorithm ?
O(nx) is considered fast. (x > 1)

O(xn) is considered slow.

How about f(n) = n500 ?

Analysis of Algorithms 17
What constitutes a fast
algorithm ?
O(nx) is considered fast. (x > 1)

O(xn) is considered slow.

How about f(n) = n500 ?

Yes, it is also fast !!

Analysis of Algorithms 18
Relatives of Big-Oh
big-Omega

 f(n) is Ω(g(n)) if there is a constant c > 0

and an integer constant n0 ≥ 1 such that
f(n) ≥ c•g(n) for n ≥ n0

Analysis of Algorithms 19
Relatives of Big-Oh

big-Theta

 f(n) is Θ(g(n)) if there are constants

c’ > 0 and c’’ > 0 and an integer
constant n0 ≥ 1 such that c’•g(n) ≤
f(n) ≤ c’’•g(n) for n ≥ n0

Analysis of Algorithms 20
Relatives of Big-Oh

big-Theta

 f(n) is Θ(g(n)) if there are constants

c’ > 0 and c’’ > 0 and an integer
constant n0 ≥ 1 such that c’•g(n) ≤
f(n) ≤ c’’•g(n) for n ≥ n0

We say, f(n) and g(n) have same growth rate

Analysis of Algorithms 21
Remember

f(x) is Θ(g(x)) if and only if

f ( x)
lim =c
x →∞ g ( x)

Analysis of Algorithms 22
Relatives of Big-Oh

little-oh

 f(n) is o(g(n)) if, for any

constant c > 0, there is an
integer constant n0 ≥ 0 such
that f(n) ≤ c•g(n) for n ≥ n0

Analysis of Algorithms 23
Relatives of Big-Oh

little-oh

 f(n) is o(g(n)) if, for any

constant c > 0, there is an
integer constant n0 ≥ 0 such
that f(n) ≤ c•g(n) for n ≥ n0

We say, g(n) is faster than f(n)

Analysis of Algorithms 24
Remember

f(x) if o(g(x)) if and only if

f ( x)
lim =0
x →∞ g ( x)

Analysis of Algorithms 25
Relatives of Big-Oh

little-omega
 f(n) is ω(g(n)) if, for any
constant c > 0, there is an
integer constant n0 ≥ 0 such
that f(n) ≥ c•g(n) for n ≥ n0

Analysis of Algorithms 26
Relatives of Big-Oh

little-omega
 f(n) is ω(g(n)) if, for any
constant c > 0, there is an
integer constant n0 ≥ 0 such
that f(n) ≥ c•g(n) for n ≥ n0

We say, g(n) is slower than f(n)

Analysis of Algorithms 27
Remember

f(x) if ω(g(x)) if and only if

f ( x)
lim =∞
x →∞ g ( x)

Analysis of Algorithms 28
Example Uses of the
Relatives of Big-Oh
 5n2 is Ω(n2)
f(n) is Ω(g(n)) if there is a constant c > 0 and an integer constant n0 ≥ 1
such that f(n) ≥ c•g(n) for n ≥ n0
let c = 5 and n0 = 1
 5n2 is Ω(n)
f(n) is Ω(g(n)) if there is a constant c > 0 and an integer constant n0 ≥ 1
such that f(n) ≥ c•g(n) for n ≥ n0
let c = 1 and n0 = 1
 5n2 is ω(n)
f(n) is ω(g(n)) if, for any constant c > 0, there is an integer constant n0 ≥
0 such that f(n) ≥ c•g(n) for n ≥ n0
need 5n02 ≥ c•n0 → given c, the n0 that satifies this is n0 ≥ c/5 ≥ 0

Analysis of Algorithms 29
Some Examples
Example 1

•Which of the two functions y = e x

and y = (100).2x grows faster?
Example 1

•Which of the two functions y = e x

and y = (100).2x grows faster?

ex 1 e x
lim = lim ( ) =∞
x→∞ (100).2x 100 x→∞ 2
Example 2

•Which of the two functions log2 x

and loge x grows faster?
Example 2

•Which of the two functions log2 x

and loge x grows faster?

log2 x
loge x log2 e 1
lim = lim = lim =
x→∞ log2 x x→∞ log2 x x→∞ log2 e
Example 3

•Which of the two functions loge x

and x 0.01 grows faster?
Example 3

•Which of the two functions loge x

and x 0.01 grows faster?

x 0.01 (0.01)x −0.99

lim = lim =
x→∞ loge x x→∞ 1/x
(0.01)x 0.01
lim =∞
x→∞ 1
Example 4

2
•Which of the two functions 10.e loge x
and x loge x /10 grows faster?
Example 4

2
•Which of the two functions 10.e loge x
and x loge x /10 grows faster?

They grow at the same rate because,

log2e x
the terms e and x loge x are equal.
Recurrence Relations

Aug-14 1
Tower of Hanoi

 Invented by French mathematician

Edouard Lucas in 1883.

8/7/2014 2
Towers of Hanoi

 There are three pegs: A, B and C

 Peg A has stack of disks
 Disks are stored in decreasing size.
Tower of Hanoi

Problem : How will you move all the disks from

peg A to peg C ?

8/7/2014 4
Tower of Hanoi

 Rule 1: You can only move one disk at a time

 Rule 2: You can make use of all the pegs

 Rule 3: At no point should a larger disk be

placed on a smaller one

8/7/2014 5
Solution :

 Transfer the top n-1 disks from A to B

 Transfer the largest disk from A to C

 Transfer n-1 disks from B to C

Aug-14 6
What is the cost ?

Let T(n) denote the number of moves

required to transfer a stack of size n

What is T(0), T(1), and T(2)

8/7/2014 7
Tower of Hanoi

Let T(n) denote the number of moves

required to transfer a stack of size n

What is T(0), T(1), and T(2)

T(0)=0, T(1)=1, and T(2)= 3

8/7/2014 8
Let T(n) denote the number of moves
required to transfer a stack of size n

Solution :

 Transfer the top n-1 disks from A to B

 Transfer the largest disk from A to C

 Transfer n-1 disks from B to C

Aug-14 9
Let T(n) denote the number of moves
required to transfer a stack of size n

Solution :

 Transfer the top n-1 disks from A to B----T(n-1)

 Transfer the largest disk from A to C--- 1

 Transfer n-1 disks from B to C ----- T(n-1)

Aug-14 10
Upper bound on the cost

T(n) <= T(n-1) + 1 + T(n-1)

T(n) <= 2. T(n-1) + 1

8/7/2014 11
What about the lower bound ?

T(n) >= ??

Is it possible to accomplish the task

In less than 2.T(n-1) + 1 moves ?

8/7/2014 12
What about the lower bound ?

T(n) >= ??

Is it possible to accomplish the task

In less than 2.T(n-1) + 1 moves ?

No it is not possible. But why ?

8/7/2014 13
Tower of Hanoi

 Given T(0)=0 and T(1)=1 we have:

 T(n) <= 2. T(n-1) + 1

and

 T(n) >= 2. T(n-1) + 1

Aug-14 14
Given T(0)=0 and T(1)=1 we have:
T(n) = 2. T(n-1) + 1

 There are multiple ways to solve it

 First one is make a guess and then

verify the guess.

Aug-14 15
Tower of Hanoi

 T(0)=0
 T(1)=2. 0 + 1 = 1
 T(2)= 2.1 + 1 = 3
 T(3)= 2.3 + 1 = 7

 It might be that T(n) = 2n – 1

 Verify it !
Aug-14 16
Tower of Hanoi

How to verify: Remember that

recurrences are ideally suited for
Mathematical Induction.

8/7/2014 17
Pizza cutting problem

 The problem was solved by Jacob

Steiner in 1826.

Aug-14 18
Problem Statement:

 How many slices of pizza can a person

obtain by making n straight cuts with a
pizza knife ?

Aug-14 19
Let’s rephrase it mathematically !

 What is the maximum number of regions

defined by n lines in a plane ?

Aug-14 20
 Notation: Let Ln be the number of
regions obtained by drawing n lines.

 Base Case: We know what is L0

Aug-14 21
What is L1 , L2 and L3
 Case 1: We cut through the center each time
 In Case 1, each cut adds two regions.

 Thus, Ln =2.n (n > 0)

Aug-14 23
Case 2: You don’t need to cut through the
center each time
A Better Slicing Method ...

 What is common that you see between these

two figures ?
An Important Observation

The nth line increases the number of

regions by k iff it splits k of the old regions
iff it intersects previous lines in k-1 different
places.

Aug-14 26
Upper bound

Ln <= Ln-1 + n

Aug-14 27
Is it possible that Ln = Ln-1 + n

 It’s possible to attain upper bound always,

provided you draw the new line such that
it intersects with all the previous lines.

Aug-14 28
7-Aug-14 29
Pizza Cutting Problem: A Variation
What is the variation ?
• It is the same problem, except for the fact that
instead of straight lines we use bent lines.
Bent lines
• Each line has exactly one bend, as shown
below:
Statement
What is the number of regions determined by n
bent lines.
Notation
• Let Zn be the number of regions determined
by the bent lines.

• What is the value of Z1 and Z2 ?

Z1 = 2

1
Z2 = 7

2 3

6 7 4
1 5
What is Zn

Hint: Try to relate it with the classical pizza

cutting problem.
4
1
3 2

• A bent line is like two straight lines.

• Regions 2, 3, and 4 are merged.

Zn = L2n – 2n
Note : Here Ln is the regions produced by the
n-lines in the original pizza cutting problem.
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15

L3.5
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15

L3.6
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15

L3.7
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15

L3.8
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15

L3.9
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15

L3.10
Recurrence for binary search
T(n) = 1 T(n/2) + Q(1)

# subproblems work dividing

and combining
subproblem size

nlogba = nlog21 = n0 = 1  CASE 2 (k = 0)

 T(n) = Q(lg n) .

L3.11
Powering a number
Problem: Compute a n, where n N.
Naive algorithm: Q(n).
Divide-and-conquer algorithm:
a n/2  a n/2 if n is even;
an =
a (n–1)/2  a (n–1)/2  a if n is odd.

T(n) = T(n/2) + Q(1)  T(n) = Q(lg n) .

L3.12
Fibonacci numbers
Recursive definition:
0 if n = 0;
Fn = 1 if n = 1;
Fn–1 + Fn–2 if n  2.

0 1 1 2 3 5 8 13 21 34 L
Naive recursive algorithm: W(fn)
(exponential time), where f = (1  5) / 2
is the golden ratio.
L3.13
Matrix multiplication
Input: A = [aij], B = [bij].
i, j = 1, 2,… , n.
Output: C = [cij] = AB.
 c11 c12 L c1n   a11 a12 L a1n   b11 b12 L b1n 
c c L c2n  a21 a22 L a2 n  b21 b22 L b2n 
 21 22   
                 
c c L cnn   an1 an 2 L ann  bn1 bn 2 L bnn 
 n1 n 2

n
cij   aik  bkj
k 1
L3.17
Standard algorithm
for i  1 to n
do for j  1 to n
do cij  0
for k  1 to n
do cij  cij + aik bkj

Running time = Q(n3)

L3.18
Divide-and-conquer algorithm
IDEA:
nn matrix = 22 matrix of (n/2)(n/2) submatrices:
r s  a b   e f 
 
 t u  c d   g h 
     
C = A  B
r = ae + bg
s = af + bh 8 mults of (n/2)(n/2) submatrices
t = ce + dh 4 adds of (n/2)(n/2) submatrices
u = cf + dh
L3.19
Analysis of D&C algorithm
T(n) = 8 T(n/2) + Q(n2)

# submatrices work adding

submatrices
submatrix size

nlogba = nlog28 = n3  CASE 1  T(n) = Q(n3).

No better than the ordinary algorithm.

L3.20
Strassen’s idea
• Multiply 22 matrices with only 7 recursive mults.
P1 = a  ( f – h) r = P 5 + P 4 – P2 + P 6
P2 = (a + b)  h s = P1 + P2
P3 = (c + d)  e t = P3 + P4
P4 = d  (g – e) u = P5 + P1 – P3 – P7
P5 = (a + d)  (e + h)
P6 = (b – d)  (g + h) 7 mults, 18 adds/subs.
P7 = (a – c)  (e + f ) Note: No reliance on
commutativity of mult!
L3.21
Strassen’s idea
• Multiply 22 matrices with only 7 recursive mults.
P1 = a  ( f – h) r = P 5 + P 4 – P2 + P 6
P2 = (a + b)  h = (a + d) (e + h)
P3 = (c + d)  e + d (g – e) – (a + b) h
P4 = d  (g – e) + (b – d) (g + h)
P5 = (a + d)  (e + h) = ae + ah + de + dh
P6 = (b – d)  (g + h) + dg –de – ah – bh
P7 = (a – c)  (e + f ) + bg + bh – dg – dh
= ae + bg
L3.22
Strassen’s algorithm
1. Divide: Partition A and B into
(n/2)(n/2) submatrices. Form terms
to be multiplied using + and – .
2. Conquer: Perform 7 multiplications of
(n/2)(n/2) submatrices recursively.
3. Combine: Form C using + and – on
(n/2)(n/2) submatrices.

T(n) = 7 T(n/2) + Q(n2)

L3.23
Analysis of Strassen
T(n) = 7 T(n/2) + Q(n2)
nlogba = nlog27  n2.81  CASE 1  T(n) = Q(nlg 7).
The number 2.81 may not seem much smaller than
3, but because the difference is in the exponent, the
impact on running time is significant. In fact,
Strassen’s algorithm beats the ordinary algorithm
on today’s machines for n  30 or so.
Best to date (of theoretical interest only): Q(n2.376L).
L3.24
Conclusion

• Divide and conquer is just one of several

powerful techniques for algorithm design.
• Divide-and-conquer algorithms can be
analyzed using recurrences and the master
method (so practice this math).
• Can lead to more efficient algorithms

L3.25
Divide-and-Conquer
7 29 4  2 4 7 9

72  2 7 94  4 9

77 22 99 44

Divide-and-Conquer 1
Outline and Reading
Divide-and-conquer paradigm
Review Merge-sort
Recurrence Equations
 Iterative substitution
 Recursion trees
 Guess-and-test
 The master method

Divide-and-Conquer 2
Divide-and-Conquer
Divide-and conquer is a
general algorithm design
paradigm:
 Divide: divide the input data S in
two or more disjoint subsets S1,
S2 , …
 Recur: solve the subproblems
recursively
 Conquer: combine the solutions
for S1, S2, …, into a solution for S
The base case for the
recursion are subproblems of
constant size
Analysis can be done using
recurrence equations
Divide-and-Conquer 3
Merge-Sort Review
Merge-sort on an input
sequence S with n Algorithm mergeSort(S)
elements consists of Input sequence S with n
three steps: elements
 Divide: partition S into Output sequence S sorted
two sequences S1 and S2
of about n/2 elements if S.size() > 1
each
(S1, S2)  partition(S, n/2)
 Recur: recursively sort S1
and S2 mergeSort(S1,)
 Conquer: merge S1 and mergeSort(S2)
S2 into a unique sorted S  merge(S1, S2)
sequence

Divide-and-Conquer 4
Recurrence Equation
Analysis
The conquer step of merge-sort consists of merging two sorted
sequences, each with n/2 elements and implemented by means of
a doubly linked list, takes at most bn steps, for some constant b.
Likewise, the basis case (n < 2) will take at b most steps.
Therefore, if we let T(n) denote the running time of merge-sort:

 b if n  2
T (n)  
2T ( n / 2)  bn if n  2
We can therefore analyze the running time of merge-sort by
finding a closed form solution to the above equation.
 That is, a solution that has T(n) only on the left-hand side.

Divide-and-Conquer 5
Iterative Substitution
In the iterative substitution, or “plug-and-chug,” technique, we
iteratively apply the recurrence equation to itself and see if we can
find a pattern: T ( n )  2T ( n / 2)  bn
 2( 2T ( n / 2 2 ))  b( n / 2))  bn
 2 2 T ( n / 2 2 )  2bn
 23 T ( n / 23 )  3bn
 2 4 T ( n / 2 4 )  4bn
 ...
 2i T ( n / 2i )  ibn
Note that base, T(n)=b, case occurs when 2i=n. That is, i = log n.
So, T (n)  bn  bn log n
Thus, T(n) is O(n log n).
Divide-and-Conquer 6
The Recursion Tree
Draw the recursion tree for the recurrence relation and look for a
pattern:
 b if n  2
T (n)  
2T ( n / 2)  bn if n  2
time
depth T’s size
0 1 n bn

1 2 n/2 bn

i 2i n/2i bn

… … … …

Total time = bn + bn log n

(last level plus all previous levels)
Divide-and-Conquer 7
Guess-and-Test Method
In the guess-and-test method, we guess a closed form solution
and then try to prove it is true by induction:
 b if n  2
T (n)  
2T ( n / 2)  bn log n if n  2
Guess: T(n) < cn log n.

Divide-and-Conquer 8
Guess-and-Test Method
In the guess-and-test method, we guess a closed form solution
and then try to prove it is true by induction:
 b if n  2
T (n)  
2T ( n / 2)  bn log n if n  2
Guess: T(n) < cn log n.
T (n)  2T ( n / 2)  bn log n
 2(c(n / 2) log( n / 2))  bn log n
 cn (log n  log 2)  bn log n
 cn log n  cn  bn log n

The above guess is correct only if c.n is greater than equal to

b.n.(log n). This is not possible.

Divide-and-Conquer 9
Guess-and-Test Method,
Part 2
Recall the recurrence equation:
 b if n  2
T (n)  
2T ( n / 2)  bn log n if n  2
Guess #2: T(n) < cn log2 n.

Divide-and-Conquer 10
Guess-and-Test Method,
Part 2
Recall the recurrence equation:
 b if n  2
T (n)  
2T ( n / 2)  bn log n if n  2
Guess #2: T(n) < cn log2 n.
T (n)  2T (n / 2)  bn log n
 2(c(n / 2) log 2 (n / 2))  bn log n
 cn(log n  log 2) 2  bn log n
 cn log 2 n  2cn log n  cn  bn log n

 Here the above guess is correct if c > b. This is very much possible.

Divide-and-Conquer 11
Guess-and-Test Method,
Part 2
Recall the recurrence equation:
 b if n  2
T (n)  
2T ( n / 2)  bn log n if n  2
Guess #2: T(n) < cn log2 n.
T (n)  2T (n / 2)  bn log n
 2(c(n / 2) log 2 (n / 2))  bn log n
 cn(log n  log 2) 2  bn log n
 cn log 2 n  2cn log n  cn  bn log n

 Here the above guess is correct if c > b. This is very much

possible. So, T(n) is O(n log2 n). In general, to use this
method, you need to have a good guess and you need to be
good at induction proofs.

Divide-and-Conquer 12
Master Method
Many divide-and-conquer recurrence equations have
the form:
 c if n  d
T (n)  
aT ( n / b)  f ( n ) if n  d

The Master Theorem:

1. if f (n) is O(n log b a  ), then T (n) is (n log b a )
2. if f (n) is (n log b a log k n), then T (n) is (n log b a log k 1 n)
3. if f (n) is (n log b a  ), then T (n) is ( f (n)),
provided af (n / b)  f (n) for some   1.

Divide-and-Conquer 13
Master Method, Example 1
The form: T (n )   c if n  d
aT ( n / b)  f ( n ) if n  d
The Master Theorem:
1. if f (n) is O(n log b a  ), then T (n) is (n log b a )
2. if f (n) is (n log b a log k n), then T (n) is (n log b a log k 1 n)
3. if f (n) is (n log b a  ), then T (n) is ( f (n)),
provided af (n / b)  f (n) for some   1.
Example:
T (n)  4T (n / 2)  n
Solution: logba=2, so case 1 says T(n) is Θ(n2).

Divide-and-Conquer 14
Master Method, Example 2
The form: T (n )   c if n  d
aT ( n / b)  f ( n ) if n  d
The Master Theorem:
1. if f (n) is O(n log b a  ), then T (n) is (n log b a )
2. if f (n) is (n log b a log k n), then T (n) is (n log b a log k 1 n)
3. if f (n) is (n log b a  ), then T (n) is ( f (n)),
provided af (n / b)  f (n) for some   1.
Example:
T (n)  2T (n / 2)  n log n
Solution: logba=1, so case 2 says T(n) is Θ(n log2 n).

Divide-and-Conquer 15
Master Method, Example 3
The form: T (n )   c if n  d
aT ( n / b)  f ( n ) if n  d
The Master Theorem:
1. if f (n) is O(n log b a  ), then T (n) is (n log b a )
2. if f (n) is (n log b a log k n), then T (n) is (n log b a log k 1 n)
3. if f (n) is (n log b a  ), then T (n) is ( f (n)),
provided af (n / b)  f (n) for some   1.
Example:
T (n)  T (n / 3)  n log n
Solution: logba=0, so case 3 says T(n) is Θ(n log n).

Divide-and-Conquer 16
Master Method, Example 4
The form: T (n )   c if n  d
aT ( n / b)  f ( n ) if n  d
The Master Theorem:
1. if f (n) is O(n log b a  ), then T (n) is (n log b a )
2. if f (n) is (n log b a log k n), then T (n) is (n log b a log k 1 n)
3. if f (n) is (n log b a  ), then T (n) is ( f (n)),
provided af (n / b)  f (n) for some   1.
Example:
T (n)  8T (n / 2)  n 2

Solution: logba=3, so case 1 says T(n) is Θ(n3).

Divide-and-Conquer 17
Master Method, Example 5
The form: T (n )   c if n  d
aT ( n / b)  f ( n ) if n  d
The Master Theorem:
1. if f (n) is O(n log b a  ), then T (n) is (n log b a )
2. if f (n) is (n log b a log k n), then T (n) is (n log b a log k 1 n)
3. if f (n) is (n log b a  ), then T (n) is ( f (n)),
provided af (n / b)  f (n) for some   1.
Example:
T (n)  9T (n / 3)  n 3

Solution: logba=2, so case 3 says T(n) is Θ(n3).

Divide-and-Conquer 18
Master Method, Example 6
The form: T (n )   c if n  d
aT ( n / b)  f ( n ) if n  d
The Master Theorem:
1. if f (n) is O(n log b a  ), then T (n) is (n log b a )
2. if f (n) is (n log b a log k n), then T (n) is (n log b a log k 1 n)
3. if f (n) is (n log b a  ), then T (n) is ( f (n)),
provided af (n / b)  f (n) for some   1.
Example:
T (n)  T (n / 2)  1 (binary search)

Solution: logba=0, so case 2 says T(n) is Θ(log n).

Divide-and-Conquer 19
Master Method, Example 7
The form: T (n )   c if n  d
aT ( n / b)  f ( n ) if n  d
The Master Theorem:
1. if f (n) is O(n log b a  ), then T (n) is (n log b a )
2. if f (n) is (n log b a log k n), then T (n) is (n log b a log k 1 n)
3. if f (n) is (n log b a  ), then T (n) is ( f (n)),
provided af (n / b)  f (n) for some   1.
Example:
T (n)  2T (n / 2)  log n (heap construction)
Solution: logba=1, so case 1 says T(n) is Θ(n).

Divide-and-Conquer 20
Iterative “Proof” of the
Master Theorem
Using iterative substitution, let us see if we can find a pattern:
T (n)  aT (n / b)  f (n)
 a (aT (n / b 2 ))  f (n / b))  bn
 a 2T (n / b 2 )  af (n / b)  f (n)
 a 3T (n / b 3 )  a 2 f (n / b 2 )  af (n / b)  f (n)
 ...
(log b n ) 1
a log b n
T (1)  a
i 0
i
f (n / b i )
(log b n ) 1
n log b a
T (1)  a
i 0
i
f (n / b i )
We then distinguish the three cases as
 The first term is dominant
 Each part of the summation is equally dominant
 The summation is a geometric series
Divide-and-Conquer 21
Integer Multiplication
Algorithm: Multiply two n-bit integers I and J.
 Divide step: Split I and J into high-order and low-order bits
I  I h 2n / 2  I l
J  J h 2n / 2  J l
 We can then define I*J by multiplying the parts and adding:
I * J  ( I h 2n / 2  I l ) * ( J h 2n / 2  J l )
 I h J h 2n  I h J l 2n / 2  I l J h 2n / 2  I l J l
 So, T(n) = 4T(n/2) + n, which implies T(n) is O(n2).
 But that is no better than the algorithm we learned in grade
school.

Divide-and-Conquer 22
An Improved Integer
Multiplication Algorithm
Algorithm: Multiply two n-bit integers I and J.
 Divide step: Split I and J into high-order and low-order bits
I  I h 2n / 2  I l
J  J h 2n / 2  J l
 Observe that there is a different way to multiply parts:
I * J  I h J h 2 n  ( I h J l  I l J h )2 n / 2  I l J l
 I h J h 2n  [(I h J l  I l J l  I h J h  I l J h )  I h J h  I l J l ]2n / 2  I l J l
I h J h 2n  [(I h  I l )( J l  J h )  I h J h  I l J l ]2n / 2  I l J l

 So, T(n) = 3T(n/2) + n, which implies T(n) is O(nlog23), by

the Master Theorem.
 Thus, T(n) is O(n1.585).
Divide-and-Conquer 23
Median Finding Algorithm

Submitted By:
Arjun Saraswat
Nishant Kapoor
Problem Definition

 Given a set of "n" unordered numbers we

want to find the "k th" smallest number. (k is
an integer between 1 and n).
A Simple Solution
 A simple sorting algorithm like heapsort will take
Order of O(nlg2n) time.

Step Running Time

Sort n elements using heapsort O(nlog2n)
Return the kth smallest element O(1)
Total running time O(nlog2n)
Linear Time selection
algorithm
 Also called Median Finding Algorithm.
 Find k th smallest element in O (n) time
in worst case.
 Uses Divide and Conquer strategy.
 Uses elimination in order to cut down
the running time substantially.
Steps to solve the problem
 Step 1: If n is small, for example n<6,
just sort and return the k th smallest
number in constant time i.e; O(1) time.

 Step 2: Group the given number in

subsets of 5 in O(n) time.
 Step3: Sort each of the group in O (n)
time. Find median of each group.
 Given a set
(……..2,5,9,19,24,54,5,87,9,10,44,32,21
,13,24,18,26,16,19,25,39,47,56,71,91,6
1,44,28………) having n elements.
Arrange the numbers in groups of five

……………….. 2 54 44 4 25 ………………..

………………..
……………….. 5 5 32 18 39

………………..
21 26 47
……………….. 9 87

……………….. 19 9 13 16 56 ………………..

2 19 71 ………………..
……………….. 24 10
Find median of N/5 groups

……………….. 2 5 2 4 25 ………………..

………………..
……………….. 5 9 13 16 39

………………..
……………….. 9 10 21 18 47

……………….. 19 54 32 19 56 ………………..

44 26 71 ………………..
……………….. 24 87
Median of each group
Find the Median of each group

……………….. 2 5 2 4 25 ………………..

3.n/10 ………………..
……………….. 5 9 13 16 39

………………..
21 18 47
……………….. 9 10

……………….. 19 54 32 19 56 ………………..

44 26 71 ………………..
……………….. 24 87

Find m ,the median of medians

Find the sets L and R

 Compare each n-1 elements with the median m and find two
sets L and R such that every element in L is smaller than M and
every element in R is greater than m.

m
L R

3n/10<L<7n/10 3n/10<R<7n/10
Description of the Algorithm step

 If n is small, for example n<6, just sort and return the k the smallest
number.( Bound time- 7)
 If n>5, then partition the numbers into groups of 5.(Bound time n/5)
 Sort the numbers within each group. Select the middle elements (the
medians). (Bound time- 7n/5)
 Call your "Selection" routine recursively to find the median of n/5
medians and call it m. (Bound time-Tn/5)
 Compare all n-1 elements with the median of medians m and
determine the sets L and R, where L contains all elements <m, and R
contains all elements >m. Clearly, the rank of m is r=|L|+1 (|L| is the
size or cardinality of L). (Bound time- n)
Contd….

 If k=r, then return m

 If k<r, then return k th smallest of the set L .(Bound time T7n/10)

 If k>r, then return k-r th smallest of the set R.

Recursive formula

 T (n)=O (n) + T (n/5) +T (7n/10)

We will solve this equation in order to get the complexity.
We assume that T (n)< C*n
T (n) = a*n + T (n/5) + T (7n/10)
C*n >= T(n/5) +T(7n/10) + a*n
C*n >= C*n/5+ C*7*n/10 + a*n
C >= 9*C/10 +a
C/10 >= a
C >= 10*a
There is such a constant that exists….so T (n) = O (n)
Why group of 5 why not some other term??

 If we divide elements into groups of 3 then we will have

T (n) = O (n) + T (n/3) + T (2n/3) so T (n) > O (n)…..
 If we divide elements into groups of more than 5, the value of
constant 5 will be more, so grouping elements in to 5 is the
optimal situation.
Quick-Sort
7 4 9 6 2  2 4 6 7 9

4 2  2 4 7 9  7 9

22 99

Quick-Sort 1
Outline and Reading
Quick-sort
 Algorithm
 Partition step
 Quick-sort tree
 Execution example
Analysis of quick-sort
In-place quick-sort
Summary of sorting algorithms

Quick-Sort 2
Quick-Sort
Quick-sort is a randomized
sorting algorithm based x
on the divide-and-conquer
paradigm:
 Divide: pick a random
element x (called pivot) and
x
partition S into
 L elements less than x
 E elements equal x L E G
 G elements greater than x
 Recur: sort L and G
 Conquer: join L, E and G x

Quick-Sort 3
Partition
We partition an input Algorithm partition(S, p)
sequence as follows: Input sequence S, position p of pivot
 We remove, in turn, each Output subsequences L, E, G of the
elements of S less than, equal to,
element y from S and
or greater than the pivot, resp.
 We insert y into L, E or G, L, E, G  empty sequences
depending on the result of
x  S.remove(p)
the comparison with the
while S.isEmpty()
pivot x
y  S.remove(S.first())
Each insertion and removal if y < x
is at the beginning or at the L.insertLast(y)
end of a sequence, and else if y = x
hence takes O(1) time E.insertLast(y)
Thus, the partition step of else { y > x }
quick-sort takes O(n) time G.insertLast(y)
return L, E, G
Quick-Sort 4
Quick-Sort Tree
An execution of quick-sort is depicted by a binary tree
 Each node represents a recursive call of quick-sort and stores
 Unsorted sequence before the execution and its pivot
 Sorted sequence at the end of the execution
 The root is the initial call
 The leaves are calls on subsequences of size 0 or 1

7 4 9 6 2  2 4 6 7 9

4 2  2 4 7 9  7 9

22 99
Quick-Sort 5
Execution Example
Pivot selection
7 2 9 43 7 6 1  1 2 3 4 6 7 8 9

7 2 9 4  2 4 7 9 3 8 6 1  1 3 8 6

22 9 4  4 9 33 88

99 44

Quick-Sort 6
Execution Example (cont.)
Partition, recursive call, pivot selection
7 2 9 4 3 7 6 1 1 2 3 4 6 7 8 9

2 4 3 1 2 4 7 9 3 8 6 1  1 3 8 6

22 9 4  4 9 33 88

99 44

Quick-Sort 7
Execution Example (cont.)
Partition, recursive call, base case
7 2 9 43 7 6 1 1 2 3 4 6 7 8 9

2 4 3 1  2 4 7 3 8 6 1  1 3 8 6

11 9 4  4 9 33 88

99 44

Quick-Sort 8
Execution Example (cont.)
Recursive call, …, base case, join
7 2 9 43 7 6 1 1 2 3 4 6 7 8 9

2 4 3 1  1 2 3 4 3 8 6 1  1 3 8 6

11 4 3  3 4 33 88

99 44

Quick-Sort 9
Execution Example (cont.)
Recursive call, pivot selection
7 2 9 43 7 6 1 1 2 3 4 6 7 8 9

2 4 3 1  1 2 3 4 7 9 7 1  1 3 8 6

11 4 3  3 4 88 99

99 44

Quick-Sort 10
Execution Example (cont.)
Partition, …, recursive call, base case
7 2 9 43 7 6 1 1 2 3 4 6 7 8 9

2 4 3 1  1 2 3 4 7 9 7 1  1 3 8 6

11 4 3  3 4 88 99

99 44

Quick-Sort 11
Execution Example (cont.)
Join, join
7 2 9 4 3 7 6 1 1 2 3 4 6 7 7 9

2 4 3 1  1 2 3 4 7 9 7  17 7 9

11 4 3  3 4 88 99

99 44

Quick-Sort 12
Worst-case Running Time
The worst case for quick-sort occurs when the pivot is the unique
minimum or maximum element
One of L and G has size n - 1 and the other has size 0
The running time is proportional to the sum
n + (n - 1) + … + 2 + 1
Thus, the worst-case running time of quick-sort is O(n2)
depth time
0 n

1 n-1

… …

n-1 1
Quick-Sort 13
Expected Running Time
Consider a recursive call of quick-sort on a sequence of size s
 Good call: the sizes of L and G are each less than 3s/4
 Bad call: one of L and G has size greater than 3s/4
7 2 9 43 7 6 19 7 2 9 43 7 6 1

2 4 3 1 7 9 7 1  1 1 7294376

Good call Bad call

A call is good with probability ?????

Quick-Sort 14
Expected Running Time
Consider a recursive call of quick-sort on a sequence of size s
 Good call: the sizes of L and G are each less than 3s/4
 Bad call: one of L and G has size greater than 3s/4
7 2 9 43 7 6 19 7 2 9 43 7 6 1

2 4 3 1 7 9 7 1  1 1 7294376

Good call Bad call

A call is good with probability 1/2

 1/2 of the possible pivots cause good calls:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Bad pivots Good pivots Bad pivots

Quick-Sort 15
Expected Running Time, Part 2
 For a node of depth i,
The size of the input sequence for the current call is at most (3/4)i.n

 Probabilistic Fact: The expected number of coin tosses required in order

to get k heads is 2k
 Therefore, we have expected height time per level
s(r) O(n)
 The expected height of the
quick-sort tree is O(log n)
 The amount or work done at the s(a) s(b) O(n)
nodes of the same depth is O(n)
O(log n)
 Thus, the expected running time s(c) s(d) s(e) s(f) O(n)
of quick-sort is O(n log n)

total expected time: O(n log n)

Quick-Sort 16
In-Place Quick-Sort
Quick-sort can be implemented
to run in-place
In the partition step, we use Algorithm inPlaceQuickSort(S, l, r)
replace operations to rearrange Input sequence S, ranks l and r
the elements of the input Output sequence S with the
sequence such that elements of rank between l and r
rearranged in increasing order
 the elements less than the
pivot have rank less than h if l  r
 the elements equal to the pivot return
have rank between h and k i  a random integer between l and r
 the elements greater than the x  S.elemAtRank(i)
pivot have rank greater than k (h, k)  inPlacePartition(x)
The recursive calls consider inPlaceQuickSort(S, l, h - 1)
 elements with rank less than h inPlaceQuickSort(S, k + 1, r)
 elements with rank greater
than k

Quick-Sort 17
Summary of Sorting Algorithms
Algorithm Time Notes
in-place
selection-sort O(n2) slow (good for small inputs)

in-place
insertion-sort O(n2) slow (good for small inputs)

O(n log n) in-place, randomized

quick-sort fastest (good for large inputs)
expected
in-place
heap-sort O(n log n) fast (good for large inputs)

sequential data access

merge-sort O(n log n) fast (good for huge inputs)

Quick-Sort 18
Selection

Selection 1
The Selection Problem
Given an integer k and n elements x1, x2, …, xn,
taken from a total order, find the k-th smallest
element in this set.
Of course, we can sort the set in O(n log n) time
and then index the k-th element.
k=3 7 4 9 6 2  2 4 6 7 9

Can we solve the selection problem faster?

Selection 2
Quick-Select
Quick-select is a randomized
selection algorithm based on
x
the prune-and-search
paradigm:
 Prune: pick a random element x
(called pivot) and partition S into
 L elements less than x
x
 E elements equal x
 G elements greater than x L E G
 Search: depending on k, either k < |L| k > |L|+|E|
answer is in E, or we need to k’ = k - |L| - |E|
recurse in either L or G
|L| < k < |L|+|E|
(done)
Selection 3
Partition
We partition an input Algorithm partition(S, p)
sequence as in the quick-sort Input sequence S, position p of pivot
algorithm: Output subsequences L, E, G of the
elements of S less than, equal to,
 We remove, in turn, each or greater than the pivot, resp.
element y from S and
L, E, G  empty sequences
 We insert y into L, E or G, x  S.remove(p)
depending on the result of
while S.isEmpty()
the comparison with the
y  S.remove(S.first())
pivot x
if y < x
Each insertion and removal is L.insertLast(y)
at the beginning or at the else if y = x
end of a sequence, and E.insertLast(y)
hence takes O(1) time else { y > x }
Thus, the partition step of G.insertLast(y)
quick-select takes O(n) time return L, E, G
Selection 4
Quick-Select Visualization
An execution of quick-select can be visualized by a
recursion path
 Each node represents a recursive call of quick-select, and
stores k and the remaining sequence

k=5, S=(7 4 9 3 2 6 5 1 8)

k=2, S=(7 4 9 6 5 8)

k=2, S=(7 4 6 5)

k=1, S=(7 6 5)

5
Selection 5
Expected Running Time
Consider a recursive call of quick-select on a sequence of size s
 Good call: the sizes of L and G are each less than 3s/4
 Bad call: one of L and G has size greater than 3s/4
7 2 9 43 7 6 19 7 2 9 43 7 6 1

2 4 3 1 7 9 7 1  1 1 7294376

Good call Bad call

A call is good with probability 1/2

 1/2 of the possible pivots cause good calls:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Bad pivots Good pivots Bad pivots

Selection 6
Expected Running Time,
Part 2
Probabilistic Fact #1: The expected number of coin tosses required in
order to get one head is two
Probabilistic Fact #2: Expectation is a linear function:
 E(X + Y ) = E(X ) + E(Y )
 E(cX ) = cE(X )
Let T(n) denote the expected running time of quick-select.
By Fact #2,
 T(n) < T(3n/4) + bn*(expected # of calls before a good call)

By Fact #1,
 T(n) < T(3n/4) + 2bn

That is, T(n) is a geometric series:

 T(n) < 2bn + 2b(3/4)n + 2b(3/4)2n + 2b(3/4)3n + …

So T(n) is O(n).
We can solve the selection problem in O(n) expected
time.
Selection 7
Deterministic Selection
We can do selection in O(n) worst-case time.
Main idea: recursively use the selection algorithm
itself to find a good pivot for quick-select:
 Divide S into n/5 sets of 5 each

 Find a median in each set

 Recursively find the median of the “baby”

Min size medians.

1 1 1 1 1 1 1 1 1 1 1
for L 2 2 2 2 2 2 2 2 2 2 2
3
4
3
4
3
4
3
4
3
4
3
4
3
4
3
4
3
4
3
4
3
4
Min size
5 5 5 5 5 5 5 5 5 5 5 for G

See Exercise C-4.24 for details of analysis.

Selection 8
Probability theory
Sample Space: It is the set of all possible
outcomes of an experiment. It is denoted by S

Examples:

Flipping a coin, S = {T, H}

Rolling a dice, S = {1,2,..,6}

Probability Space (PS)
PS = (S, Pr), where :

S is the sample space

Pr is a function from 2S to [0, 1]

Event: Each subset of S is called an event. It is
denoted by A.

Example: Rolling a dice, S= {1,2, …,6}

Odd number appears, A={1,3,5}
Axioms of a PS = (S, Pr)
1. Pr(Φ)= 0
2. Pr(S)=1
3. 0<= Pr(A) <= 1, where A  S
4. If A, B  S and A Π B = Φ, then
Pr(A U B)= Pr(A) + Pr(B)
Independence
• Two events A and B are independent if

Pr(A Π B)= Pr(A). Pr(B)

Conditional Probability
The probability that A happens, given that B has
already happened is denoted by Pr(A|B).

Pr(A|B) = Pr(A Π B)/ Pr(B)

Random Variable X
It is a function X that maps each element of S to
a real number
X: S  R

Indicator Random Variable:

X: S  {0, 1}
Expected value
• The expected value of a random variable X is
denoted by E(X) where

E 𝑋 = σ𝑥 𝑥. Pr(𝑋 = 𝑥)
Example
• Let X be a random variable that assigns the
outcomes of the roll of two fair dice the sum
of the number on the two dices. What is the
expected value of X.

• E(X)= ?
Example
• Let X be a random variable that assigns the
outcomes of the roll of two fair dice the sum
of the number on the two dices. What is the
expected value of X.

• E(X)= 7
Linearity of Expectation
• Let X and Y be two random variables

• E(X + Y)= E(X) + E(Y)

• E(c.X)= c.E(X) where c is a real number.
Example
• What is the expected number of times a
person must toss a coin to get a head.
Example
• What is the expected number of times a
person must toss a coin to get a head.

• Answer is 2
The Greedy Method

The Greedy Method 1

Outline and Reading

The Greedy Method Technique

Fractional Knapsack Problem
Task Scheduling
Minimum Spanning Trees [future lecture]

The Greedy Method 2

The Greedy Method
Technique
The greedy method is a general algorithm
design paradigm, built on the following
elements:
 configurations: different choices or collections
 objective function: a score assigned to
configurations, which we want to either maximize or
minimize
It works best when applied to problems with the
greedy-choice property:
 a globally-optimal solution can always be found by a
series of local improvements from a starting
configuration.
The Greedy Method 3
The Fractional Knapsack
Problem
Given: A set S of n items, with each item i having
 bi - a positive benefit
 wi - a positive weight
Goal: Choose items with maximum total benefit but with
weight at most W.
If we are allowed to take fractional amounts, then this is
the fractional knapsack problem.
 In this case, we let xi denote the amount we take of item i

 Objective: maximize b ( x / w )
iS
i i i

 Constraint: x
iS
i W
The Greedy Method 4
Example
Given: A set S of n items, with each item i having
 bi - a positive benefit
 wi - a positive weight
Goal: Choose items with maximum total benefit but with
weight at most W.
“knapsack”

Solution:
• 1 ml of 5
Items:
1 2 3 4 5 • 2 ml of 3
• 6 ml of 4
Weight: 4 ml 8 ml 2 ml 6 ml 1 ml • 1 ml of 2
Benefit: $12 $32 $40 $30 $50 10 ml
Value: 3 4 20 5 50
($ per ml)
The Greedy Method 5
The Fractional Knapsack
Algorithm
Greedy choice: Keep taking
item with highest value Algorithm fractionalKnapsack(S, W)
(benefit to weight ratio) Input: set S of items w/ benefit bi
 Since  bi ( xi / wi )   (bi / wi ) xi and weight wi; max. weight W
iS iS Output: amount xi of each item i
 Run time: ? to maximize benefit with
Correctness: ? weight at most W
for each item i in S
xi  0
vi  bi / wi {value}
w0 {total weight}
while w < W
remove item i with highest vi
xi  min{wi , W - w}
w  w + min{wi , W - w}

The Greedy Method 6

Task Scheduling
Given: a set T of n tasks, each having:
 A start time, si
 A finish time, fi (where si < fi)
Goal: Perform all the tasks using a minimum number of
“machines.”

Machine 3
Machine 2
Machine 1

1 2 3 4 5 6 7 8 9

The Greedy Method 7

The Greedy Method 8

Task Scheduling
Algorithm
Greedy choice: consider tasks
by their start time and use as
few machines as possible with Algorithm taskSchedule(T)
this order.
Input: set T of tasks w/ start time si
 Run time: O(n log n). Why? and finish time fi
Correctness: Suppose there is a Output: non-conflicting schedule
better schedule. with minimum number of machines
 We can use k-1 machines m0 {no. of machines}
 The algorithm uses k
while T is not empty
remove task i w/ smallest si
 Let i be first task scheduled
on machine k if there’s a machine j for i then
schedule i on machine j
 Machine i must conflict with
k-1 other tasks else
mm+1
 But that means there is no
non-conflicting schedule schedule i on machine m
using k-1 machines
The Greedy Method 9
Example
Given: a set T of n tasks, each having:
 A start time, si
 A finish time, fi (where si < fi)
 [1,4], [1,3], [2,5], [3,7], [4,7], [6,9], [7,8] (ordered by start)
Goal: Perform all tasks on min. number of machines

Machine 3
Machine 2
Machine 1

1 2 3 4 5 6 7 8 9

The Greedy Method 10

Making Change
Problem: A dollar amount yet to return to a customer.
Objective function: Minimize number of coins returned.
Greedy solution: Always return the largest coin you can
Example 1: Coins are valued $.32, $.08, $.01
 Has the greedy-choice property, since no amount over $.32 can
be made with a minimum number of coins by omitting a $.32
coin (similarly for amounts over $.08, but under $.32).
Example 2: Coins are valued $.30, $.20, $.05, $.01
 Does not have greedy-choice property, since $.40 is best made
with two $.20’s, but the greedy solution will pick three coins
(which ones?)

The Greedy Method 11

Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15

L3.5
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15

L3.6
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15

L3.7
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15

L3.8
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15

L3.9
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3 5 7 8 9 12 15

L3.10
Recurrence for binary search
T(n) = 1 T(n/2) + Q(1)

# subproblems work dividing

and combining
subproblem size

nlogba = nlog21 = n0 = 1  CASE 2 (k = 0)

 T(n) = Q(lg n) .

L3.11
Powering a number
Problem: Compute a n, where n N.
Naive algorithm: Q(n).
Divide-and-conquer algorithm:
a n/2  a n/2 if n is even;
an =
a (n–1)/2  a (n–1)/2  a if n is odd.

T(n) = T(n/2) + Q(1)  T(n) = Q(lg n) .

L3.12
Fibonacci numbers
Recursive definition:
0 if n = 0;
Fn = 1 if n = 1;
Fn–1 + Fn–2 if n  2.

n
cij   aik  bkj
k 1
L3.17
Standard algorithm
for i  1 to n
do for j  1 to n
do cij  0
for k  1 to n
do cij  cij + aik bkj

Running time = Q(n3)

# submatrices work adding

submatrices
submatrix size

nlogba = nlog28 = n3  CASE 1  T(n) = Q(n3).

No better than the ordinary algorithm.

T(n) = 7 T(n/2) + Q(n2)

• Divide and conquer is just one of several

powerful techniques for algorithm design.
• Divide-and-conquer algorithms can be
analyzed using recurrences and the master
method (so practice this math).
• Can lead to more efficient algorithms

L3.25
David Huffman

• Build the tree (code) bottom-up in a greedy

fashion
Building the Encoding Tree
Building the Encoding Tree
Building the Encoding Tree
Building the Encoding Tree
Building the Encoding Tree
Minimum Spanning Tree
Spanning Trees

A spanning tree of a graph is just a subgraph that

contains all the vertices and is a tree.
A graph may have many spanning trees.

Graph A Some Spanning Trees from Graph A

or or or
Complete Graph All 16 of its Spanning Trees
Minimum Spanning Trees

The Minimum Spanning Tree for a given graph is the Spanning Tree of
minimum cost for that graph.

Complete Graph Minimum Spanning Tree

2 2
5 3 3

1 1
Algorithms for Obtaining the Minimum Spanning Tree

• Kruskal's Algorithm

• Prim's Algorithm

• Boruvka's Algorithm
Kruskal's Algorithm

This algorithm creates a forest of trees. Initially the forest consists of n

single node trees (and no edges). At each step, we add one edge (the
cheapest one) so that it joins two trees together. If it were to form a cycle,
it would simply link two nodes that were already part of a single
connected tree, so that this edge would not be needed.
The steps are:

1. The forest is constructed - with each node in a separate tree.

2. The edges are placed in a priority queue.
3. Until we've added n-1 edges,
1. Extract the cheapest edge from the queue,
2. If it forms a cycle, reject it,
3. Else add it to the forest. Adding it to the forest will join two
trees together.

Every step will have joined two trees in the forest together, so that at
the end, there will only be one tree in T.
Complete Graph

B 4 C
4
2 1

A 4 E
1 F

D 2 3
10
G 5

5 6 3

4
I
H

2 3
J
A 4 B A 1 D

B 4 C B 4 D

B 4 C B 10 J C 2 E
4
2 1
C 1 F D 5 H
A 4 E
1 F

2 D 6 J E 2 G
D 3
10
G 5
F 3 G F 5 I
5 6 3

4
I G 3 I G 4 J
H

2 3
J H 2 J I 3 J
Sort Edges A 1 D C 1 F