COMPSCI 311: Introduction to Algorithms

Lecture 10: Divide and Conquer

Marius Minea and Ghazaleh Parvini

University of Massachusetts Amherst

20 March 2023

slides credit: Dan Sheldon

Divide and Conquer: Recipe

I Divide problem into several parts

I Recursively solve each part

I Combine solutions to sub-problems into overall solution

Learning Goals

Greedy Divide and Conquer

Formulate problem
Design algorithm X
Prove correctness X
Analyze running time X
Specific algorithms Dijkstra, MST
A Classic Algorithm: Mergesort

MergeSort(List) Û complexity T(n) = ?

if List.length < 2 then Û Base case
return List Û sorted
else
split List in halves List1 and List2 Û (n)
Sort1 = MergeSort(List1) Û T(n/2)
Sort2 = MergeSort(List2) Û T(n/2)
return Merge(Sort1,Sort2) Û (n)
Mergesort Complexity

You probably know Mergesort is (n log n)

Informal argument:
I log n tree levels

I same array/list at every level, just split differently

I work at each level (split/merge) is linear: (n)
I total: (n log n)

How do we solve such recurrences in the general case?

Goal: rigorous recipes/rules, make them easy to use
Motivating Problem: Maximum Subsequence Sum (MSS)

I Input: array A of n numbers, e.g.

A = 4, ≠3, 5, ≠2, ≠1, 2, 6, ≠2

I Find: value of the largest subsequence sum

A[i] + A[i + 1] + . . . + A[j]

empty subsequence allowed and has sum zero

I MSS in example? 11 (first 7 elements)

I Can we extract any observations?

A Simple MSS Algorithm

MSS(A)
Initialize all entries of n ◊ n array B to zero
for i = 1 to n do
sum = 0
for j = i to n do
B[i, j] = sum of A[i] ... A[j]
Return maximum value among all B[i, j]

Running time? O(n3 ) with naive sum, O(n2 ) reusing sums.

Can we do better?
Divide-and-conquer for MSS

I Recursive solution for MSS

I Idea:
I Find MSS L in left half of array
I Find MSS R in right half of array
I Find MSS M for sequence that crosses the midpoint

M =11
˙ ˝¸ ˚
A = 4, ≠3, 5, ≠2, ≠1, 2, 6 , ≠2
¸ ˚˙ ˝ ¸˚˙˝
L=6 R=8

I Return max(L, R, M )
I How to find L, R? Recursively.
I Find M ? max. left + max. right from middle
max max
Ω≠≠≠≠≠≠|≠≠≠≠æ
4, -3, 5, -2, -1, 2, 6, -2
Divide and Conquer: Maximal Subsequence Sum

MSS(A, left, right)

if left = right then Û Base case
Solve directly and return MSS Û max(A[left], 0])
mid = Â left+right
2
Ê Û Recurse on left and right halves
L = MSS(A, left, mid)
R = MSS(A, mid+1, right)

Set sum = 0 and LÕ = 0 Û Compute LÕ (left part of M )

for i = mid downto left do
sum += A[i]
LÕ = max(LÕ , sum)
Set sum = 0 and RÕ = 0 Û Compute RÕ (right part of M )
for i = mid+1 to right do
sum += A[i]
RÕ = max(RÕ , sum)
M = LÕ + R Õ Û Compute M
return max(L, R, M ) Û Return max
MSS(A, left, right)
if right ≠ left Æ 2 then
Solve directly and return
MSS
Running time?
I Let T (n) be running time of MSS
mid = Â left+right
2
Ê on array of size n
L = MSS(A, left, mid)
R = MSS(A, mid+1, right) I Two recursive calls on arrays of size
n/2: 2T (n/2)
Set sum = 0 and LÕ = 0
for i = mid down to 1 do I Work outside of recursive calls:
sum += A[i]
O(n)
LÕ = max(LÕ , sum)
I Running time
Set sum = 0 and RÕ = 0
for i mid+1 to right do
T (n) = 2T (n/2) + O(n)
sum += A[i]
RÕ = max(RÕ , sum)
Same as mergesort, (n log n)
M = LÕ + R Õ

return max(L, R, M )
Recurrence

I Recurrence (with convenient base case)

T (n) = 2T (n/2) + O(n)

T (1) = O(1) (a constant)

can also have more base cases: T (1) and T (2), etc.

I Goal: solve the recurrence = find simple expression for T (n)

I First, let’s use definition of Big-O:

T (n) Æ 2T (n/2) + cn

I What next?
Solving the Recurrence

I In general, won’t worry whether n/2 is still an integer, etc.

we might actually have T (n) Æ T (Ân/2Ê) + T (Án/2Ë) + cn
can prove our approach is still correct in this case

I Three approaches to solve it

1. Unrolling the recurrence
2. Recursion tree (another version of unrolling)
3. Guess and verify (proof by induction)

Unrolling substitutes n æ n/2 to get T (n/2) Æ 2T (n/4) + cn/2 . . .

If substituting n with n/2 seems confusing:
I first, rewrite recurrence with another variable m:
T (m) Æ 2T (m/2) + cm
I then substitute m = n/2
Recurrence Solving (1): Unrolling

T (n) Æ 2T
Ë
(n/2) + cn È
equation in n
Æ 2 2T (n/22 ) + c(n/2) + cn n æ n/2
= 22 T (n/22 )
Ë
+ 2cn È
Æ 22 2T (n/23 ) + c(n/22 ) + 2cn n æ n/4
= 23 T (n/23 ) + 3cn
Æ ...

I Do you see a pattern? T (n) Æ 2k T (n/2k ) + k · cn

I When does this stop?

Base case: n/2k = 1 =∆ k = log n unrollings

T (n) Æ 2log n · T (1) + log n · cn = c1 n + cn log n = O(n log n)

Work done unrolling (cn log n) dominates work in base cases (c1 n).
Recurrence Solving (2): Recursion Tree

I Each level adds work to expand nodes

I red: extra terms in recurrence, i.e., outside T ()
I blue: total work done, from top until that level
T (n)

T ( n2 ) T ( n2 ) +cn cn

T ( n4 ) T ( n4 ) +c n2 T ( n4 ) T ( n4 ) +c n2 cn + cn = 2cn

T(n
8
)T ( n
8
) +c n
4
T(n
8
)T ( n
8
)+c n
4
T(n
8
)T ( n
8
)+c n
4
T(n
8
)T ( n
8
) +c n
4 2cn + cn = 3cn

At bottom level: n leaves with T (1) +d · cn, with depth d = log n

Conclusion: T (n) Æ c1 · n + cn log n = O(n log n)
Recurrence Solving (3): Guess and Verify

I Guess solution
I Prove by (strong) induction

T (n) Æ 2T (n/2) + cn

Guess solution T (n) Æ kn log n, find k so it works.

Choose base case n = 2 ∆ T (2) Æ 2k log 2, need k Ø T (2)/2.
n = 1 won’t work as base case, since log 1 = 0
(but small n does not matter for big-O)
Guess and Verify: Induction Step

Strong induction:
Assume T (m) Æ k · m log m for all m < n
Prove statement for n: T (n) Æ k · n log n

T (n) Æ 2 · T (n/2) + cn
Æ 2 · k(n/2) log(n/2) + cn apply inductive hypothesis
= kn(log n ≠ 1) + cn
= kn log n + (c ≠ k)n Æ kn log n if k Ø c

For base case, we needed k Ø T (2)

∆ choose k = max(c, T (2)/2).
The induction proof is complete, T (n) Æ kn log n = O(n log n)
A More General Recurrence

T (n) Æ q · T (n/2) + cn
I What does the algorithm look like?
I q recursive calls to itself on problems of half the size
I O(n) work outside of the recursive calls

I We’ve seen: q = 2, T (n) Æ 2T (n/2) + cn

I Now: q = 1, q > 2

I Use geometric sum: if r ”= 1 then

1 ≠ rd+1 rd+1 ≠ 1
1 + r + r2 + . . . + rd = =
1≠r r≠1
General Case

Unrolling or recursion tree for T (n) = qT (n/2) + cn

Level 0, unroll T (n): + cn work
Level 1, unroll q nodes of T (n/2): + q · cn/2 work, etc.
Level 2, unroll q 2 nodes of T (n/4): + q 2 · cn/4 work, etc.
Unrolling level j of recursion tree (substitutes n æ n/2j ):
1 q 2j
qj ◊ cn/2j = cn
¸˚˙˝ ¸ ˚˙ ˝ 2
number of subproblems work per subproblem

ÿ1
d≠1
q 2j
Work added at all levels: L(n) = cn (d = log2 n)
j=0
2
Work for base cases: B(n) = q d · T (1) = c1 q log2 n = c1 nlog2 q .
q = 1? Easy to see L(n) Æ 2cn = O(n) and B(n) = O(1).
General Case (q > 2)

ÿ1
d≠1
q 2j
Work added at all levels: L(n) = cn (d = log2 n)
j=0
2
Let r = q/2 > 1. Then
d≠1
ÿ rd ≠ 1 1 d 1 log2 n 1 log2 r
rj = Æ r≠1 r = r≠1 r = r≠1 n
j=0
r≠1

We’ve used: xlogb y = y logb x (take log: logb y logb x = logb x logb y)
d≠1
ÿ
log2 (q/2)
So rj = 1
r≠1 n = O(nlog2 q≠1 )
j=0

Thus, L(n) = cn · O(nlog2 q≠1 ) = O(nlog2 q ) work while unrolling.

Base cases: B(n) = q d T (1) = O(q log2 n ) = O(nlog2 q ) (same complexity).

Overall, T (n) = O(nlog2 q ). Example: q = 3, T (n) is O(n1.59 )

Summary

Useful general recurrence and its solutions:

T (n) Æ q · T (n/2) + cn

1. q = 1: T (n) = O(n)
2. q = 2: T (n) = O(n log n)
3. q > 2: T (n) = O(nlog2 q )
Next time: Master Theorem (more general)

Algorithms with these running times?

1. Mergesort, Maximum Subsequence Sum
2. Counting Inversions, Integer multiplication (next lectures)