Algorithms Illuminated Part 3 Greedy Algorithms and Dynamic Programming Tim Roughgarden 2024 scribd download
Algorithms Illuminated Part 3 Greedy Algorithms and Dynamic Programming Tim Roughgarden 2024 scribd download
com
https://textbookfull.com/product/algorithms-illuminated-
part-3-greedy-algorithms-and-dynamic-programming-tim-
roughgarden/
OR CLICK BUTTON
DOWNLOAD NOW
https://textbookfull.com/product/algorithms-illuminated-part-3-greedy-
algorithms-and-dynamic-programming-1st-edition-tim-roughgarden/
textboxfull.com
https://textbookfull.com/product/beyond-the-worst-case-analysis-of-
algorithms-1st-edition-tim-roughgarden/
textboxfull.com
https://textbookfull.com/product/algorithms-for-competitive-
programming-1st-edition-david-esparza-alba/
textboxfull.com
https://textbookfull.com/product/twenty-lectures-on-algorithmic-game-
theory-1st-edition-tim-roughgarden/
textboxfull.com
Algorithms and Programs of Dynamic Mixture Estimation
Unified Approach to Different Types of Components Ivan
Nagy
https://textbookfull.com/product/algorithms-and-programs-of-dynamic-
mixture-estimation-unified-approach-to-different-types-of-components-
ivan-nagy/
textboxfull.com
https://textbookfull.com/product/programming-quantum-computers-
essential-algorithms-and-code-samples-1st-edition-eric-r-johnston/
textboxfull.com
https://textbookfull.com/product/art-of-computer-programming-the-
combinatorial-algorithms-volume-4b-1st-edition-donald-knuth/
textboxfull.com
Algorithms Illuminated
Part 3: Greedy Algorithms and Dynamic
Programming
Tim Roughgarden
c 2019 by Tim Roughgarden
All rights reserved. No portion of this book may be reproduced in any form
without permission from the publisher, except as permitted by U. S. copyright
law.
First Edition
Preface vii
14 Huffman Codes 23
14.1 Codes 23
14.2 Codes as Trees 28
14.3 Huffman’s Greedy Algorithm 32
*14.4 Proof of Correctness 41
Problems 49
v
vi Contents
Index 211
Preface
vii
viii Preface
This series of books has only one goal: to teach the basics of algorithms
in the most accessible way possible. Think of them as a transcript
of what an expert algorithms tutor would say to you over a series of
one-on-one lessons.
There are a number of excellent more traditional and encyclopedic
textbooks about algorithms, any of which usefully complement this
book series with additional details, problems, and topics. I encourage
you to explore and find your own favorites. There are also several
books that, unlike these books, cater to programmers looking for
ready-made algorithm implementations in a specific programming
language. Many such implementations are freely available on the Web
as well.
x Preface
Additional Resources
These books are based on online courses that are currently running
on the Coursera and Stanford Lagunita platforms. I’ve made several
resources available to help you replicate as much of the online course
experience as you like.
Videos. If you’re more in the mood to watch and listen than
to read, check out the YouTube video playlists available from
www.algorithmsilluminated.org. These videos cover all the topics
in this book series, as well as additional advanced topics. I hope they
exude a contagious enthusiasm for algorithms that, alas, is impossible
to replicate fully on the printed page.
Quizzes. How can you know if you’re truly absorbing the concepts
in this book? Quizzes with solutions and explanations are scattered
throughout the text; when you encounter one, I encourage you to
pause and think about the answer before reading on.
End-of-chapter problems. At the end of each chapter you’ll find
several relatively straightforward questions for testing your under-
Preface xi
Acknowledgments
These books would not exist without the passion and hunger supplied
by the hundreds of thousands of participants in my algorithms courses
over the years. I am particularly grateful to those who supplied
detailed feedback on an earlier draft of this book: Tonya Blust, Yuan
Cao, Carlos Guia, Jim Humelsine, Vladimir Kokshenev, Bayram
Kuliyev, and Daniel Zingaro.
I always appreciate suggestions and corrections from readers.
These are best communicated through the discussion forums men-
tioned above.
Tim Roughgarden
New York, NY
April 2019
Chapter 13
1
2 Introduction to Greedy Algorithms
The first half of this book is about the greedy algorithm design
paradigm. What is a greedy algorithm, exactly? Much blood and ink
have been spilled over this question, so we’ll content ourselves with
an informal definition.1
The best way to get a feel for greedy algorithms is through exam-
ples. We’ll see several over the next few chapters.2
Warning
Most greedy algorithms are not always correct.
Our first case study concerns scheduling, in which the goal is to sched-
ule tasks on one or more shared resources to optimize some objective.
For example, a resource could represent a computer processor (with
tasks corresponding to jobs), a classroom (with tasks corresponding
to lectures), or your calendar for the day (with tasks corresponding
to meetings).
Completion Times
Quiz 13.1
Consider a problem instance that has three jobs with `1 = 1,
`2 = 2, and `3 = 3, and suppose they are scheduled in this
order (with job 1 first). What are the completion times
of the three jobs in this schedule? (The job weights are
irrelevant for this question, so we have not specified them.)
a) 1, 2, and 3
b) 3, 5, and 6
c) 1, 3, and 6
d) 1, 4, and 6
3 · 1 + |{z}
|{z} 2 · 3 + |{z}
1 · 6 = 15.
job #1 job #2 job #3
6 Introduction to Greedy Algorithms
Greedy algorithms seem like a good fit for the problem of scheduling
jobs to minimize the weighted sum of completion times. The output
has an iterative structure, with jobs processed one by one. Why not
6
For example, n! is bigger than 3.6 million when n = 10, bigger than 2.4
quintillion when n = 20, and bigger than the estimated number of atoms in the
known universe when n 60. Thus no conceivable improvement in computer
technology would transmute exhaustive search into a useful algorithm.
13.3 Developing a Greedy Algorithm 7
job #3
time
3
job #2
1
job #1
0
Figure 13.1: The completion times of the three jobs are 1, 3, and 6.
Quiz 13.2
a) larger/shorter
b) smaller/shorter
c) larger/longer
d) smaller/longer
In the general case, jobs can have different weights and different
lengths. Whenever our two rules-of-thumb—to prefer shorter jobs
and higher-weight jobs—luckily coincide for a pair of jobs, we know
which one to schedule first (the shorter, higher-weight one). But what
if the two rules give conflicting advice? What should we do with one
short low-weight job and one long high-weight job?
What’s the simplest greedy algorithm that might work? Each
job has two parameters, and the algorithm must look at both. The
best-case scenario would be to come up with a formula that compiles
each job’s length and weight into a single score, so that scheduling
jobs from highest to lowest score is guaranteed to minimize the sum of
weighted completion times. If such a formula exists, our two special
cases imply that it must have two properties: (i) holding the length
fixed, it should be increasing in the job’s weight; and (ii) holding the
weight fixed, it should be decreasing in the job’s length. (Remember,
higher scores are better.) Take a minute to brainstorm some formulas
that have both of these properties.
* * * * * * * * * * *
There are plenty of other options. For example, the ratio of the
two parameters is another candidate:
wj
proposal #2 for score of job j: .
`j
These two scoring functions lead to two different greedy algo-
rithms.
GreedyDiff
Schedule the jobs in decreasing order of wj `j
(breaking ties arbitrarily).
GreedyRatio
wj
Schedule the jobs in decreasing order of `j
(breaking ties arbitrarily).
Thus, already, our first case study illustrates the first theme of the
greedy paradigm (Section 13.1.2): It is often easy to propose multiple
competing greedy algorithms for a problem.
Which of the two algorithms, if any, is correct? A quick way to
rule out one of them is to find an instance in which the two algorithms
output different schedules, with different objective function values.
For whichever algorithm fares worse in this example, we can conclude
that it is not always optimal.
Both algorithms do the right thing in our two special cases, with
equal-weight or equal-length jobs. The simplest possible example for
ruling out one of them would be a problem instance with two jobs,
having different weights and lengths, such that the two algorithms
schedule the jobs in opposite orders. That is, we seek two jobs whose
ordering by difference is the opposite of their ordering by ratio. One
simple example is:
Job #1 Job #2
Length `1 = 5 `2 = 2
Weight w1 = 3 w2 = 1.
The first job has the larger ratio ( 35 vs. 12 ) but the smaller (more
negative) difference ( 2 vs. 1). Thus the GreedyDiff algorithm
schedules the second job first, while GreedyRatio does the opposite.
10 Introduction to Greedy Algorithms
Quiz 13.3
What is the sum of weighted completion times in the sched-
ules output by the GreedyDiff and GreedyRatio algorithms,
respectively?
a) 22 and 23
b) 23 and 22
c) 17 and 17
d) 17 and 11
Correct answer: (a). First suppose that all n jobs have the same
length, say length 1. Then, every schedule has exactly the same
set of completion times—{1, 2, 3, . . . , n}—and the only question is
which job gets which completion time. Our semantics for job weights
certainly suggests that the higher-weight jobs should receive the
smaller completion times, and this is in fact the case. For example,
you wouldn’t want to schedule a job with weight 10 third (with
completion time 3) and one with weight 20 fifth (with completion
time 5); you’d be better off exchanging the positions of these two jobs,
which would decrease the sum of weighted completion times by 20 (as
you should check).
The second case, in which all jobs have equal weights, is a little
more subtle. Here, you want to favor shorter jobs. For example,
consider two unit-weight jobs with lengths 1 and 2. If you schedule
the shorter job first, the completion times are 1 and 3, for a total
of 4. In the opposite order, the completion times are 2 and 3, for
an inferior total of 5. In general, the job scheduled first contributes
to the completion times of all the jobs, as all jobs must wait for
the first one to finish. All else being equal, scheduling the shortest
job first minimizes this negative impact. The second job contributes
12 Introduction to Greedy Algorithms
to all the completion times other than that of the first job, so the
second-shortest job should be scheduled next, and so on.
w1 · C1 + w2 · C2 = 3 · 7 + 1 · 2 = 23.
3 · 5 + 1 · 7 = 22.
Two Assumptions
wi wj
(2) There are no ties between ratios: `i 6= `j whenever
i 6= j.
job #n
..........
job #3
time
job #2
job #1
σ
more more
stuff stuff
j i
exchange!
i j
time
time
stuff stuff
σ* σ’
(a) Before exchange (b) After exchange
Figure 13.3: Obtaining the new schedule 0 from the allegedly optimal
schedule ⇤ by exchanging the jobs in a consecutive inversion (with i > j).
Quiz 13.4
What effect does the exchange have on the completion time
of: (i) a job other than i or j; (ii) the job i; and (iii) the
job j?
Now is the time to use the fact that ⇤ scheduled i and j in the “wrong
order,” with i > j. Our standing assumptions (1) and (2) imply that
jobs are indexed in strictly decreasing order of weight-length ratio, so
wi wj
< .
`i `j
w i `j < wj `i .
|{z} |{z}
cost of exchange benefit of exchange
Because the benefit of the exchange exceeds the cost, equation (13.3)
tells us that
0 ⇤
objective function value of < objective function value of .
“Did you ever see any one so changed as that pretty Mrs. Beacham? Still
more like a lady even than she used to be—I mean, she hasn’t the same
fresh country look that everyone admired so before. Mr. Delmaine thinks
her both out of health and out of spirits; he said so yesterday afternoon to
me when mamma and I went to the church to see the decorations. She
attends to her class, though, at the school just the same as ever, and gets the
children on wonderfully with their singing. How well they sang the anthem
to-day! and all, Mr. Delmaine says, thanks to Mrs. John Beacham.”
Rather to Kate’s disappointment, her brother did not enter with much
apparent interest into the question of Honor’s illness or her merits. And yet
he was interested in John’s pretty, pensive-looking wife, more interested
than could well be explained to the young girl walking by his side. There
were many circumstances in Honor’s short life which were perforce
unknown to the high-born and carefully-reared Katherine, whose secluded
life kept her very little au fait of the doings and sayings of the outer world.
Of the former intimacy of her brother at Updown Paddocks she had heard
little or nothing; nor, though it was more than probable that the “ower true
tale” of Honor’s birth had reached Miss Vavasour’s ears, was the subject
one which could well be touched upon with a discreetly-brought-up young
lady. Under these circumstances, it is only natural that Horace Vavasour
should have manifested some unwillingness to pursue the subject touched
upon by his sister. Concerning one cause, amongst others, of Mrs. John
Beacham’s lowness of spirits he might have entertained his own ideas, but
those ideas he, for the moment, wisely kept to himself.
CHAPTER XIV.
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
textbookfull.com