Get Introduction to Probability for Computing 1st Edition Harchol-Balter PDF ebook with Full Chapters Now
Get Introduction to Probability for Computing 1st Edition Harchol-Balter PDF ebook with Full Chapters Now
Get Introduction to Probability for Computing 1st Edition Harchol-Balter PDF ebook with Full Chapters Now
com
https://textbookfull.com/product/introduction-to-
probability-for-computing-1st-edition-harchol-balter/
OR CLICK BUTTON
DOWNLOAD NOW
https://textbookfull.com/product/introduction-to-probability-david-f-
anderson/
textboxfull.com
https://textbookfull.com/product/introduction-to-probability-second-
edition-joseph-k-blitzstein/
textboxfull.com
https://textbookfull.com/product/introduction-to-probability-and-
statistics-for-engineers-and-scientists-6th-edition-sheldon-m-ross/
textboxfull.com
https://textbookfull.com/product/introduction-to-probability-and-
statistics-metric-edition-1925-2009-mendenhall/
textboxfull.com
Introduction to Probability and Statistics (Metric
Version) 15th Edition William Mendenhall
https://textbookfull.com/product/introduction-to-probability-and-
statistics-metric-version-15th-edition-william-mendenhall/
textboxfull.com
https://textbookfull.com/product/introduction-to-probability-2nd-
edition-dimitri-p-bertsekas-john-n-tsitsiklis/
textboxfull.com
https://textbookfull.com/product/introduction-to-probability-and-
statistics-metric-version-15e-william-mendenhall/
textboxfull.com
https://textbookfull.com/product/introduction-to-the-history-of-
computing-a-computing-history-primer-gerard-oregan/
textboxfull.com
https://textbookfull.com/product/introduction-to-computing-and-
problem-solving-using-python-1st-edition-e-balaguruswamy/
textboxfull.com
Introduction to Probability
for Computing
MOR HARCHOL-BALTER
Cambridge University Press
Illustrated by Elin Zhou
v
Acknowledgments xxiii
2 Probability on Events 21
2.1 Sample Space and Events 21
2.2 Probability Defined on Events 22
2.3 Conditional Probabilities on Events 24
2.4 Independent Events 27
2.5 Law of Total Probability 30
2.6 Bayes’ Law 32
2.7 Exercises 34
4 Expectation 58
4.1 Expectation of a Discrete Random Variable 58
4.2 Linearity of Expectation 63
4.3 Conditional Expectation 67
4.4 Computing Expectations via Conditioning 72
4.5 Simpson’s Paradox 74
4.6 Exercises 76
6 z-Transforms 116
6.1 Motivating Examples 116
6.2 The Transform as an Onion 117
6.3 Creating the Transform: Onion Building 118
6.4 Getting Moments: Onion Peeling 120
6.5 Linearity of Transforms 121
6.6 Conditioning 123
6.7 Using z-Transforms to Solve Recurrence Relations 124
6.8 Exercises 128
References 539
Index 544
Preface
Every topic is driven by what computer science students need to know. For ex-
ample, the book covers distributions that come up in computer science, such as
heavy-tailed distributions. There is a large emphasis on variability and higher
moments, which are very important in empirical computing distributions. Com-
puter systems modeling and simulation are also discussed, as well as statistical
inference for estimating parameters of distributions. Much attention is devoted to
tail bounds, such as Chernoff bounds. Chernoff bounds are used for confidence
intervals and also play a big role in the analysis of randomized algorithms, which
themselves comprise a large part of the book. Finally, the book covers Markov
chains, as well as a bit of queueing theory, both with an emphasis on their use in
computer systems analysis.
Intended Audience
The material is presented at the advanced undergraduate level. The book is based
on an undergraduate class, Probability and Computing (PnC), which I have been
teaching at Carnegie Mellon University (CMU) for almost 20 years. While PnC
is primarily taken by undergraduates, several Masters and PhD students choose
to take the class. Thus we imagine that instructors can use the book for different
levels of classes, perhaps spanning multiple semesters.
Preface xvii
Exercises
The exercises in this book are an integral part of learning the material. They also
introduce many of the computer science and statistics applications. Very few of
the exercises are rote. Every problem has important insights, and the insights
often build on each other. Exercises are (very roughly) organized from easier to
harder. Several of the exercises in the book were contributed by students in the
class!
Before we describe the parts in more detail, it is worth looking at the dependency
structure for the book, given in Figure P1. Aside from Parts I, II, and III, most
of the parts can be taught in any order.
Part I
Probability
on Events
Part II
Discrete
Random
Variables
Part III
Continuous
Random
Variables
Part VII
Randomized
Algorithms
Figure P1 The dependency structure between the parts of this book. Most parts are
independent of other parts and can be taught in any order.
Part II: Discrete Random Variables: Part II introduces the most common dis-
crete random variables (Bernoulli, Binomial, Geometric, and Poisson), and then
Preface xix
Part IV Part V
Part V Part VI Part IV
Systems Statistical
Statistical Tail Systems
Modeling and Inference Bounds Modeling and
Simulation Simulation
Figure P2 Four different courses that one can teach out of this book.
Part III: Continuous Random Variables: Part III repeats the material in Part II,
but this time with continuous random variables. We introduce the Uniform,
Exponential, and Normal distributions, as well as the Central Limit Theorem.
In addition, we introduce the Pareto heavy-tailed distribution, which is most
relevant for empirical computing workloads, and discuss its relevance to today’s
data center workloads. We cover failure rate functions and the heavy-tail property
and their relevance to computing workloads. We again cover moment-generating
functions, but this time via Laplace transforms, which are more commonly used
with continuous random variables. Some applications covered in Part II are:
classifying jobs in a supercomputing center, learning the bias of a coin, dart
throwing, distributions whose parameters are random variables, relating laptop
quality to lifetime, modeling disk delays, modeling web file sizes, modeling
compute usage, modeling IP flow durations, and Internet node degrees.
Part IV: Computer Systems Modeling and Simulation: Part IV covers the
basics of what is needed to run simulations of computer systems. We start by
defining and analyzing the Poisson process, which is the most commonly used
model for the arrival process of jobs into computer systems. We then study how
to generate random variables for simulation, using the inverse transform method
and the accept–reject method. Finally, we discuss how one would program a
simple event-driven or trace-driven simulator. Some applications that we cover
include: Malware detection of infected hosts, population modeling, reliability
theory, generating a Normal random variable, generating Pareto and Bounded
Pareto random variables, generating a Poisson random variable, simulation of
heavy-tailed distributions, simulation of high-variance distributions, simulation
of jointly distributed random variables, simulation of queues, and simulation of
networks of queues.
Part VI: Tail Bounds and Applications: Part VI starts with a discussion of tail
bounds and concentration inequalities (Markov, Chebyshev, Chernoff), for which
we provide full derivations. We provide several immediate applications for these
tail bounds, including a variety of classic balls-and-bins applications. The balls
and bins framework has immediate application to dispatching tasks to servers in
a server farm, as well as immediate application to hashing algorithms, which we
also study extensively. We cover applications of tail bounds to defining confidence
intervals in statistical estimation, and well as bias estimation, polling schemes,
crowd sourcing, and other common settings from computing and statistics.
Part VII: Randomized Algorithms: Part VII introduces a wide range of ran-
domized algorithms. The randomized algorithms include Las Vegas algorithms,
such as randomized algorithms for sorting and median finding, as well as Monte
Carlo randomized algorithms such as MinCut, MaxCut, matrix multiplication
checking, polynomial multiplication, and primality testing. The exercises in this
part are particularly relevant because they introduce many additional randomized
algorithms such as randomized dominating set, approximate median finding, in-
dependent set, AND/OR tree evaluation, knockout tournaments, addition of 𝑛-bit
numbers, randomized string exchange, path-finding in graphs, and more. We use
the tail bounds that we derived earlier in Part VI to analyze the runtimes and
accuracy of our randomized algorithms.
Part VIII: Markov Chains with a Side of Queueing Theory: Part VIII provides
an introduction to stochastic processes as they come up in computer science.
Here we delve deeply into discrete-time Markov chains (both finite and infinite).
We discuss not only how to solve for limiting distributions, but also when
they exist and why. Ergodicity, positive-recurrence and null-recurrence, passage
times, and renewal theory are all covered. We also cover time averages versus
ensemble averages and the impact of these different types of averages on running
simulations. Queueing theory is integral to Part VIII. We define the performance
metrics that computer scientists care about: throughput, response time, and load.
We cover Little’s Law, stability, busy periods, and capacity provisioning. A
huge number of applications are covered in Part VIII, including, for example,
the classic PageRank algorithm for ranking web pages, modeling of epidemic
spread, modeling of caches, modeling processors with failures, Brownian motion,
estimating the spread of malware, reliability theory applications, population
modeling, server farm and data center modeling, admission control, and capacity
provisioning.
Acknowledgments
Most textbooks begin with a class, and this book is no exception. I created the
Probability and Computing (called “PnC” for short) class 20 years ago, with the
aim of teaching computer science undergraduates the probability that they need
to know to be great computer scientists. Since then I have had a few opportunities
to co-teach PnC with different colleagues, and each such opportunity has led to
my own learning. I would like to thank my fantastic co-instructors: John Lafferty,
Klaus Sutner, Rashmi Vinayak, Ryan O’Donnell, Victor Adamchik, and Weina
Wang. I’m particularly grateful to Weina, who collaborated with me on three of
the chapters of the book and who is a kindred spirit in Socratic teaching. The
book has also benefited greatly from many spirited TAs and students in the class,
who proposed fun exercises for the book, many referencing CMU or Pittsburgh.
I would also like to thank my illustrator, Elin Zhou, who painstakingly created
every image and figure in the book, while simultaneously managing her under-
graduate classes at CMU. I chose Elin as my illustrator because her artwork
embodies the spirit of fun and inclusiveness that permeates the PnC class. One
of the themes of PnC is chocolate, which is tossed out throughout the class to
students who answer questions. This chocolate would not be possible if it weren’t
for our class sponsor, Citadel, who even paid to have chocolate mailed directly
to student homes throughout the pandemic, while classes were online.
Fundamentals and
Probability on Events
In this first part of the book we focus on some basic tools that we will need
throughout the book.
This book assumes some mathematical skills. The reader should be comfortable
with high school algebra, including logarithms. Basic calculus (integration, dif-
ferentiation, limits, and series evaluation) is also assumed, including nested (3D)
integrals and sums. We also assume that the reader is comfortable with sets and
with simple combinatorics and counting (as covered in a discrete math class).
Finally, we assume versatility with “big-O” and “little-o” notation. To help the
reader, in this chapter we review a few basic concepts that come up repeatedly
throughout the book. Taking the time to understand these now will make it much
easier to work through the book.
There are several series that come up repeatedly in the book, starting in Chapter 3.
Question: Try evaluating the following in closed form. (Don’t peek at the answers
until you’ve tried these yourself.) We provide the full derivations below.
(a) 𝑆 = 1 + 𝑥 + 𝑥 2 + 𝑥 3 + · · · + 𝑥 𝑛.
(b) 𝑆 = 1 + 𝑥 + 𝑥 2 + 𝑥 3 + · · · , where |𝑥 | < 1.
(c) 𝑆 = 1 + 2𝑥 + 3𝑥 2 + 4𝑥 3 + · · · + 𝑛𝑥 𝑛−1 .
(d) 𝑆 = 1 + 2𝑥 + 3𝑥 2 + 4𝑥 3 + · · · , where |𝑥 | < 1.
Solution: The trick here is to multiply both sides by the quantity (1 − 𝑥):
(1 − 𝑥)𝑆 = 𝑆 − 𝑥𝑆
= 1 + 𝑥 + 𝑥2 + 𝑥3 + · · · + 𝑥𝑛
−𝑥 − 𝑥 2 − 𝑥 3 − · · · − 𝑥 𝑛+1
= 1 − 𝑥 𝑛+1 .
1.1 Review of Simple Series 3
Hence,
1 − 𝑥 𝑛+1
𝑆= . (1.1)
1−𝑥
Note that (1.1) assumes that 𝑥 ≠ 1. If 𝑥 = 1, then the answer is clearly 𝑆 = 𝑛 + 1.
Solution: This is the same as series (a) except that we need to take the limit as
𝑛 → ∞:
1 − 𝑥 𝑛+1 1
𝑆 = lim 1 + 𝑥 + 𝑥 2 + · · · + 𝑥 𝑛 = lim = . (1.2)
𝑛→∞ 𝑛→∞ 1 − 𝑥 1−𝑥
Integrals, nested integrals, and nested sums come up throughout the book, starting
in Chapter 7. When evaluating these, it is important to pay attention to the area
over which you’re integrating and also to remember tricks like integration by
parts.
Question: Try deriving the following three expressions (again, no peeking at the
answers).
∫∞
(a) 0 𝑦𝑒 − 𝑦 𝑑𝑦.
∫∞∫ 𝑦
(b) 0 0 𝑒 − 𝑦 𝑑𝑥𝑑𝑦. Do this both with and without changing the order of inte-
gration.
∫ 𝑒 ∫ ln 𝑥
(c) 1 0 1𝑑𝑦𝑑𝑥. Do this both with and without changing the order of inte-
gration.
To change the order of integration, we first need to understand the space over
which we’re integrating. The original region of integration is drawn in Fig-
ure 1.1(a), where 𝑦 ranges from 0 to ∞, and, for each particular value of 𝑦, we
let 𝑥 range from 0 to 𝑦.
y y
x x
(a) Original integration space (b) Equivalent integration space
We can visualize this instead as shown in Figure 1.1(b), where 𝑥 now ranges
from 0 to ∞, and, for each particular value of 𝑥, we let 𝑦 range from 𝑥 to ∞:
∫ 𝑥=∞ ∫ 𝑦=∞ ∫ 𝑥=∞ 𝑦=∞
−𝑦
𝑒 𝑑𝑦𝑑𝑥 = −𝑒 − 𝑦 𝑑𝑥
𝑥=0 𝑦=𝑥 𝑥=0 𝑦=𝑥
∫ 𝑥=∞
= (0 + 𝑒 − 𝑥 )𝑑𝑥
𝑥=0
𝑥=∞
= −𝑒 − 𝑥
𝑥=0
= 1.
6 1 Before We Start ... Some Mathematical Basics
∫ 𝑒 ∫ ln 𝑥
Example 1.7 Derive: 1 0
1𝑑𝑦𝑑𝑥.
To change the order of integration, we first need to understand the space over
which we’re integrating. This is drawn in Figure 1.2(a).
y y
1 1
x x
1 e 1 e
(a) Original integration space (b) Equivalent integration space
We can visualize this instead as shown in Figure 1.2(b), which leads to the nested
integrals:
∫ 𝑦=1 ∫ 𝑥=𝑒 ∫ 𝑦=1 𝑥=𝑒
1𝑑𝑥𝑑𝑦 = 𝑥 𝑦
𝑑𝑦
𝑦=0 𝑥=𝑒 𝑦 𝑦=0 𝑥=𝑒
∫ 𝑦=1
= (𝑒 − 𝑒 𝑦 )𝑑𝑦
𝑦=0
𝑦=1
= (𝑒𝑦 − 𝑒 𝑦 )
𝑦=0
= (𝑒 − 𝑒) − (0 − 𝑒 0 )
= 1.
1.3 Fundamental Theorem of Calculus 7
The Fundamental Theorem of Calculus (FTC) will come up in the book starting
in Chapter 7. We state it here and provide some intuition for why it holds.
Theorem 1.8 (FTC and extension) Let 𝑓 (𝑡) be a continuous function defined
on the interval [𝑎, 𝑏]. Then, for any 𝑥, where 𝑎 < 𝑥 < 𝑏,
∫ 𝑥
𝑑
𝑓 (𝑡)𝑑𝑡 = 𝑓 (𝑥). (1.6)
𝑑𝑥 𝑎
Furthermore, for any differentiable function 𝑔(𝑥),
∫ 𝑔( 𝑥)
𝑑
𝑓 (𝑡)𝑑𝑡 = 𝑓 (𝑔(𝑥)) · 𝑔 ′ (𝑥). (1.7)
𝑑𝑥 𝑎
∫ 𝑥
𝑑 𝑑 Box(𝑥 + Δ) − Box(𝑥)
𝑓 (𝑡)𝑑𝑡 = Box(𝑥) = lim
𝑑𝑥 𝑎 𝑑𝑥 Δ→0 Δ
∫ 𝑥+Δ ∫𝑥
𝑎
𝑓 (𝑡)𝑑𝑡 − 𝑎 𝑓 (𝑡)𝑑𝑡
= lim
Δ→0 Δ
∫ 𝑥+Δ
𝑓 (𝑡)𝑑𝑡
= lim 𝑥
Δ→0 Δ
𝑓 (𝑥) · ✓
Δ
≈ lim 𝑓 (𝑥) ≈ 𝑓 (𝑥 + Δ) for tiny Δ
Δ→0 ✓
Δ
= 𝑓 (𝑥).
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
textbookfull.com