CS 109 NOTES

ARUN DEBRAY
JUNE 11, 2013

These notes were taken in Stanford’s CS 109 class in Spring 2013, taught by Professor Mehran Sahami. I live-TEXed
them using vim, and as such there may be typos; please send questions, comments, complaints, and corrections to
[email protected]. Thanks to Rebecca Wang, Shivaal Roy, and an anonymous student for catching a few errors.

Contents
1. Counting: 4/1/13 1
2. Permutations and Combinations: 4/3/13 3
3. Probability: 4/5/13 5
4. Conditional Probability: 4/8/13 8
5. More Conditional Probability: 4/10/13 10
6. Conditional Independence and Random Variables: 4/12/13 13
7. To Vegas: 4/15/13 16
8. The Poisson Distribution: 4/17/13 18
9. From Discrete to Continuous: 4/19/13 21
10. The Normal Distribution: 4/22/13 23
11. The Exponential Distribution: 4/24/13 26
12. The Multinomial Distribution and Independent Variables: 4/26/13 28
13. Adding Random Variables: 4/29/13 31
14. Continuous Conditional Distributions: 5/1/13 33
15. More Expectation: 5/3/13 35
16. Covariance: 5/6/13 38
17. Predictions: 5/8/13 41
18. Moment-Generating Functions: 5/10/13 42
19. Laws of Large Numbers and the Central Limit Theorem: 5/13/13 44
20. Parameters: 5/15/13 46
21. Likelihood: 5/17/13 47
22. More Bayesian Probability: 5/20/13 49
23. Machine Learning: 5/22/13 51
24. Logistic Regression: 5/24/13 53
25. Bayesian Networks: 5/29/13 55
26. Generating Random Numbers: 5/31/13 58
27. Some Review for the Final: 6/3/13 61

1. Counting: 4/1/13
“One, ah, ah, ah! Two, ah ah ah! Three, ah ah ah!” – Count von Count, Sesame Street
Though that average CS student might wonder why he is learning probability, it is in fact an incredibly useful
way of understanding principles and techniques in CS. The traditional view of probability (placing balls in urns)
isn’t as helpful, but there are a lot of problems in CS which are just abstractions of the conventional view of
probability.
For example, how does Amazon determine recommendations of similar orders? How does a spam filter classify
messages? Both of these applications rely on probability.
1
 Figure 1. Source: https://xkcd.com/552/.

Here’s a list of other applications:

(1) In terms of algorithms, there is a whole class called randomized algorithms. Additionally, analysis of
deterministic algorithms can be aided by probabilistic analysis and learning the expected value.
(2) Hashing and sorting (especially Quicksort) both deeply rely on randomness.
(3) Systems design also depends on randomness.
(4) Machine learning depends heavily on statistical models. And of course machine learning has huge
applications, such as financial modeling, DNA analysis, etc.
But to get to all of these applications, it will be necessary to start from the bottom. The very bottom. Let’s learn
how to count.
An experiment is some action that has a set of possible outcomes. Conducting the experiment indicates one of
the outcomes, but it’s not possible to predict the outcome beforehand (e.g. flipping a coin). An event is some
subset of outcomes we might care about (e.g. rolling a D6 and getting an even number). How can we count the
number of outcomes in an event?
The first rule to learn is the sum rule of counting: if the outcomes of some experiment can come from two sets
A and B, such that |A| = m, |B| = n, and A ∩ B = ∅, then the number of outcomes is m + n.
Example 1.1. This is as easy as it sounds: suppose Facebook has 100 machines in San Jose and 250 machines
in Denver. Then, if someone sends a login request, there are 350 possibilities for which machine that request
could go to. (
The next rule is the product rule of counting: if an experiment has two parts that don’t depend on each other
(e.g. rolling two dice), and the outcome of the first part is from a set A, where |A| = m again, and the outcome
of the second part comes from a set B, with |B| = n, then the total number of outcomes is mn.1
There are a couple of functions you should know because they come up a lot in probability:
Definition 1.2.
• The floor function of x is bxc, which is the largest integer less than or equal to x. This is just rounding
down: b1/2c = 0, b2.9c = 2, and b8c = 8. Be careful with negative numbers: b−1/2c = −1.
• The ceiling function of x is dxe, which is the smallest integer greater than or equal to x. Thus, d1/2e = 1,
d8e = 8, and d2.9e = 3.
These functions are used, for example, when it is necessary to fit an integer value into something given
by a real number, such as when an optimization calls for 2.9 people or something similarly problematic.
The inclusion-exclusion principle is slightly more general than the sum rule: if the outcome of an experiment
is given from sets A and B (and we don’t know anything about the intersection of A and B), then the number of
outcomes is |A ∪ B| = |A| + |B| − |A ∩ B|.
Example 1.3. As a CS major, you are familiar with binary numbers.2 How many 4-bit binary sequences start or
end with a 1? It turns out there are 12 such possibilities: there are 8 strings that start with a 1, 8 that end with
a 1, and 4 strings that start and end with a 1. Thus, the total number is 8 + 8 − 4 = 12 such strings. (
1Hey, it’s Theorem 13.27 from Math 121!
2But it is possible to drink binary numbers, too, thanks to the existence of Pepsi One and Coke Zero. This would make for a very slow,
but very refreshing, Turing machine.
2
 Another useful thing to know is the General Pigeonhole Principle: if m, n ∈ N and m objects are placed in n
buckets, then at least one bucket contains at least dm/ne objects.3 For example, if a hash table has 100 buckets
and 950 strings hashed into it, then it can be guaranteed that at least one bucket has 10 or more strings in it.
We cannot guarantee that there would be a bucket with 11 strings in it, and an explicit counterexample can be
given: 50 buckets have 10 strings each in them, and 50 buckets have 100 strings each in them. Of course, the
individual buckets could have anything in them, or could be empty.
Time for yet another principle, called the General Principle of Counting. This says that if r experiments are
performed such that part 1 of the experiment has n1 outcomes, part 2 has n2 outcomes, and so on until part r
has nr outcomes, then the total number of outcomes is n1 n2 . . . nr .
Example 1.4. California license plates were once given by 3 uppercase letters, followed by 3 digits. Thus, the
total number of distinct license places was 263 · 103 = 17, 576, 000.
Of course, this is less than the number of cars in California, so this was recently changed to add another
digit, so there are now enough letters. And if you think this is bad, consider IP addresses. (
Definition 1.5. A binary function f is a function f : {1, . . . , n} → {0, 1}.
Thus, there are 2n possible binary functions on {1, . . . , n}. This can be shown by realizing that each binary
function corresponds to a unique binary string of length n, and vice versa, and there are 2n such strings.
This illustrates a common theme of this class: the math isn’t all that difficult, but what’s tricky is understanding
how to turn the problem into a mathematical question.
Definition 1.6. A permutation is an ordered arrangement of distinct objects.
There are n! possible permutations of n distinct objects. This is because the first slot can be filled by all n
objects, then the second has n − 1 possibilities (since one is already in a spot), then n − 2, etc.
This can be used to make statements in which orders are restricted. If one has 4 PCs, 3 Macs, and 2 Linux
boxes, one could try to order them in a way such that all computers of a given OS are adjacent. The total
number of possibilities is 4!3!3!2!: within each group, all machines of a given OS can be permuted, giving 4!, 3!,
and 2!, and these are unrelated events, so they are multiplied together. Then, these groups can be permuted,
giving another 3!

2. Permutations and Combinations: 4/3/13

“Should array indices start at 0 or 1? My compromise of 0.5 was rejected without, I thought,
proper consideration.” – Stan Kelly-Bootle
We’re encouraged to use LATEX in this class, which is exciting!
Permutations, discussed in the previous lecture, can be applied to binary search trees.
Definition 2.1. A binary search tree (BST) is a binary tree such that for every node n:
• n ≥ `, where ` is the value of its left child;
• n ≤ r, where r is the value of its right child;
• The left and right subtrees are both binary search trees.
You might remember that the insertion time is about O(log n).
How many degenerate BSTs of length 3 (for 3 distinct elements) are there? These are the BSTs in which
each node has at most one child. There are 3! ways to order the elements to be inserted. Note that depending
on order, the placement of left or right children will change: in 1 − 2 − 3, there will be two right children, but in
order 2 − 1 − 3, one obtains both left and right children of the element 2. Thus, 2 − 1 − 3 and 2 − 3 − 1 aren’t
degenerate, but the rest are, so there are four such degenerate trees. When the size of the tree increases, the
probability that a randomly selected tree is degenerate becomes vanishingly small.
Here’s another problem: given 2 0s and 3 1s, how many different strings of length 5 can be made? This is
more interesting than the previous problems, as some of the permutations are identical. Thus, distinguish the
zeros and ones, so that each option is distinct, calling them 01 , 02 , etc. Thus, there are 5! = 120 orderings of the
distinct elements. However, any permutation of the zeroes is unimportant in the final answer, and similarly for
the ones, so the final result is 5!/2!3! = 10.
3The name denotes pigeons nesting in their pigeonholes, but there’s an alternate formulation: if one drills m holes in n pigeons, with
m > n, then at least one pigeon has at least one hole in it!
3
 For another example, consider the RCC who works on 4 PCs, 3 Macs, and 2 Linux boxes, and we consider two
machines to be identical iff they have the same OS. Then, the number of distinct orderings is 9!/2!3!4! = 1260.
Okay, now for some security: consider iPhone passwords. These are 4-digit passwords, so there are 10000
possibilities for passwords. However, it’s possible to obtain the four digits of the password based on looking at
where grease smudges exist on the screen. This reduces the search space to 4! = 24, which is much less secure.
What if there are only 3 smudges? Then, it’s not obvious which one is repeated. If the digits are a, b, and c,
without loss of generality assume c is repeated (so we have to multiply the final answer by 3). Then, this is a
problem we know how to solve: 4!/2! = 12, and then factoring in the chance that a or b might be doubled gives
12 · 3 = 36 options. Slightly more secure, interestingly.
But then, two digits must be better, right? Then, suppose the two digits are each given twice: the computation
is 4!/2!2! = 6, but there are no other possibilties. If one digit is used three times, the computation is 4!/3! = 4,
but there are two possibilities, so there are in total 6 + 8 = 14 options. This isn’t as secure.
The general formula is: if one has n objects and there are distinct groupsQof objects that are considered
r
identical with sizes n1 , . . . , nr , then the number of distinct permutations is n!/ i=1 ni !.
Next is (of course) combinations. This is an unordered selection of r objects from n objects. This can be
modelled by ordering all n objects, and
then treating the first r as identical and the last n − r to be identical in
n
the above formula.
n

Thus,
n

we obtain r = n!/(r!(n − r)!), sometimes also written C(n, r). It has the important
property that r = n−r .
Example 2.2. Mehran has six big books on probability (including the modestly titled All of Statistics). If you
wanted to borrow three of them, there are 63 = 20 options. However, supposing two of them are different

editions of the same book, one might not want to get both of them. Under this constraint, break the problem into
three subproblems:
(1) Suppose the 8th edition and two other books are chosen. Then, there are 42 options, since the 9th

edition can’t be picked.

(2) Symetrically, getting the 9th edition yields another 42 choices.

(3) Neither edition could be selected, which gives 43 choices.

Thus, adding them together gives the total number: 16. Dividing this into subproblems is a good strategy, but
it’s important to make sure they cover the whole space and don’t overlap.
Alternatively, you could count all the options that you don’t want: there are four such options, given by both
editions and another book. Thus, there are 20 − 4 = 16 options, again. (
The formula for nk has a nice recursive definition which can be illustrated with some code: pick some

special point. Either it is in the combination, or it isn’t, and these are all of the distinct cases. Thus, there are
C(n − 1, k − 1) options if it is included, and C(n − 1, k) if it isn’t included. This is nice, but recursion usually
needs the base cases: C(n, 0) = C(n, n) = 1.
Here’s how this looks in code:
int C ( int n , int k ) {
if ( k == 0 || n == k ) return 1;
return C (n -1 , k ) + C (n -1 ,k -1);
}
n


k is what is called a binomial coefficient: it is named so because it appears in the formula given below. There
are two proofs in the textbook, but they’re not all that important in the class. However, the equation is very
important.
Theorem 2.3 (Binomial).
n  
X n
(x + y)n = xk y n−k .
k
k=0

For an application, consider
a set with n elements. Then, what is the size of the power set? If one chooses a
subset of size k, there are nk such options, so the total number of subsets is
n   n  
X n X n k n−k
= 1 1 = (1 + 1)n = 2n .
k k
k=0 k=0
4
 n


More generally, one has the multinomial coefficient n1 ,...,n r
= n!/n1 ! · · · nr !, where n1 + · · · + nr = n.
Technically, this is overdetermined, and sometimes the nr -term is dropped (since it can be recovered).
Example 2.4. Suppose Google has 13 machines and 3 data centers and wants to allocate machines to data

13
centers A, B, and C which can hold 6, 4, and 3 machines, respectively. Then, there are 13!/6!4!3! = 6,4,3
possibilities. (
Returning to the ball-and-urn formalism, how many ways are there to place n distinct balls into r urns?
There are rn possibilities, since any ball can go into any urn.
If they are taken to be indistinct, but the urns aren’t, then the question becomes a bit more nuanced. There
are n balls and r urns, but once everything is placed the indistinctness of the balls means there are n + r − 1
distinct objects: in some sense, the dividers
between the urns are what we care about. Thus, the number of
options is (n + r − 1)!/n!(r − 1)! = n+r−1 r−1 . The total number of options is divided by the number of ways
the balls can be permuted and the dividers can be permuted. This is because the urns can be represented as
dividers.
In CS, there is a slightly different example: consider a one-indexed array x of length r such that 0 ≤ x[i] ≤ n
for each i, and such that the sum of the elements is n. This is a reformulation of the balls problem: each index
of the array is an urn, and its value represents the number of the balls in the urn. The book calls this 6.2, and
has a related proposition 6.1:
Proposition 2.5. Consider the slightly related problem in which every urn has at least one ball in it.4 This can
be solved by giving each child one piece of candy, which gives a slightly different spin on the previous problem:
there are n − r candies to put in n baskets. Thus, the total number of options is (n−r)+(r−1) n−1



r−1 = r−1 .

Consider some venture capital firm which is required to distribute $10 million amongst 4 distinct companies
A, B, C, and D, in $1-million increments.
the goal is to put 10 balls in 4 urns, and some company might
Thus,
be left out in the cold, giving 10+4−1 13


4−1 = 3 = 286.

But if you want to keep some of it, the answer is 14
4 , since you can be thought of as adding on more company:
your own back account. This is much nicer than all the casework that is the obvious approach.
Lastly, suppose that A only will accept payments of at least $3 million. This reduces the search space:
there are 10


choices, because after allocating $3 million to A, there is $7 million left to put in 4 companies,
10

3
giving 3 . Add this to the case where A is given no money, which is an exercise in allocating $10 million to 3
companies.
This final question used to be used at company A (i.e. Google). Consider an n × n grid with a robot in the
lower left-hand corner of the grid and a destination in the right-hand corner. If the robot can only move up or
right, it will reach the destination, but the path isn’t unique. How many distinct paths are there? The robot
needs to make n − 1 moves up and n − 1 moves right, so the goal is to find the number of permutations
of these
moves given that the moves in the same direction are identical: (2n − 2)!/(n − 1)!(n − 1)! = 2n−2 n−1 .

3. Probability: 4/5/13
“I like to think of the size of a set as how much Stanford spirit it has: its cardinality.”
Today’s lecture will help with the material in the second half of the first problem set. However, in order to
understand probability some of the theory behind it will be necessary.
Definition 3.1. A sample space is the set of all possible outcomes of an experiment.
Example 3.2. Some examples of sample spaces:
• When flipping a coin, the sample space is S = {H, T }.
• When flipping two distinct coins, the sample space is S × S = {(H, H), (H, T ), (T, H), (T, T )}.
• When rolling a six-sided die, the sample space is {1, 2, 3, 4, 5, 6}.
• Sample spaces need not be finite: the number of emails sent in a day is {0, 1, 2, 3, . . . }.
• They can also be dense sets: the number of hours spent watching videos on Youtube in a day instead of
working is S = [0, 24]. (

4Real-world example: trick-or-treating. You wouldn’t want some kid to go home candyless, would you?

5
Definition 3.3. An event, usually denoted E, is a subset of the sample space.5 This is in some sense the outcome
we care about.
Example 3.4. Here are some events corresponding to the sample spaces in Example 3.2:
• If a coin flip is heads, E = {H}.
• If there is at least one head in two flips, E = {(H, H), (H, T ), (T, H)}. (
Now let’s take some set operations. If E, F ⊆ S are events in the sample space S, then an event that is in E
or F is represented by E ∪ F , and an event that is in E and F is given by E ∩ F .6 For example, if E represents
rolling a 1 or a 2 on a die and F represents rolling an even number, then E ∩ F is rolling a 2.
Definition 3.5. Two events E, F are called mutually exclusive if E ∩ F = ∅.
One can take the complement of an event E, ∼E = S \ E, or everything that isn’t in E. The book uses
E C . This leads to De Morgan’s Laws: ∼(E ∪ F ) = (∼E) ∩ (∼F ) and ∼(E ∩ F ) = (∼E) ∪ (∼F ). This can be
inductively generalized to n events in the unsurprising manner.
Probability is given in several different ways. The first will be given now, and another will be given later. The
frequentist interpretation of probability interprets the probability as the relative frequency of the event. Let
n(E) be the number of times event E happens when the experiment is done n times; then, the probability is
P (E) = limn→∞ n(E)/n. With this definition comes some axioms:
(1) Probabilities must lie between 0 and 1.
(2) P (S) = 1.
(3) If E and F are mutually exclusive events, then P (E) + P (F ) = P (E ∪ F ).
From axiom 3 most of the interesting mathematics comes up. For just one example, the multivariate case is a
straightforward generalization: if E1 , E2 , . . . is a sequence of mutually exclusive events, then
∞ ∞
!
[ X
P Ei = P (Ei ).
i=1 i=1
This can be generalized to the uncountable case, but things get weird, so take care.
There are several other elementary consequences of the axioms. They are easy to prove by doing a bit of set
chasing or drawing Venn diagrams.
• P (∼E) = 1 − P (E).
• If E ⊆ F , then P (E) ≤ P (F ).
• P (E ∪ F ) = P (E) + P (F ) − P (E ∩ F ).
The latter formula can generalize to n variables as
!  
[n Xn X \ r
P Ei = (−1)r+1 P Eij .
i=1 r=1 i1 <···<ir j=1

This seems pretty scary, but all that it does is take the intersections of subsets of size r for all r in 1 to n,
and adds and subtracts things based on the number intersected (so nothing is double-counted). All of the odd
intersections have a positive coefficient, and the even ones have a negative coefficient. The i1 < · · · < ir means
that every combination is considered, and order is used to ensure that such each combination is taken only once.
For example, if n = 3,
P (A ∪ B ∪ C) = P (A) + P (B) + P (C) − P (A ∩ B) − P (B ∩ C) − P (A ∩ C) + P (A ∩ B ∩ C).
By drawing a Venn diagram, it should be possible to see why this works.
Some sample spaces have a property called equally likely outcomes. This means that each point in the
sample space has the same probability: if x ∈ S, then P (x) = 1/|S|. Examples of these include fair coins and
fair dice. Thus, if E ⊆ S, then P (E) = |E|/|S|.
Example 3.6. If one rolls two six-sided dice, the sample space is S = {1, . . . , 6}2 , and if one wants to calculate
the probability that the sum is 7, the event is E = {(1, 6), (2, 5), (3, 4), (4, 3), (5, 2), (6, 1)}, so the probability is
|E|/|S| = 6/36 = 1/6. (
5The textbook writes that E ⊂ S, but E = S is allowed too, so be careful.
6More questionable textbook notation: the textbook uses EF to denote E ∩ F .

6
Example 3.7. Suppose one has four Twinkies and three Ding Dongs.7 What is the probability that one Twinkie
and 2 Ding Dongs are drawn if three are picked? There are two ways to look at this:
• One can consider it to be ordered: the Twinkie is either the first, second, or third item, giving (4)(3)(2),
(3)(4)(2), and (3)(2)(4) combinations respectively. Thus, the probability is ((4)(3)(2) + (3)(4)(2) +
(3)(2)(4))/210 = 12/35.
• Alternatively, it can be unordered: |S| = 73 = 35, and |E| = 41 32 = 12 (picking the Twinkies and

Ding Dongs respectively).

The takeaway is that either method works, but be careful not to accidentally mix them. (

Example 3.8. Suppose someone is manufacturing chips, and has n chips, one of which is defective. If n is large,
it’s impractical to test them all, so only k will be examined. Thus, the sample space has nk possible choices,

and the event has a size of 11 n−1

k−1 (since the bad chip is picked, and the rest are fine). Thus, after cancelling
some terms, the probability is n/k.
Alternatively, here’s a more intuitive formulation: pick k chips out of the n, and then randomly make a chip
defective. Then, the chance that the chip was part of the original k is k/n. Intuition and cleverness isn’t always
necessary, but it’s a great way to save time. (

Example 3.9. Let’s play poker! Suppose one has a 5-card poker hand drawn from a standard deck and
wants a straight. This is a set of 5 cards of consecutive rank, e.g. {2, 3, 4, 5, 6}. Again, this is slightly
different from the textbook’s definition. The probability is a little weird, because there are lots of cases:
4 5
{A, 2, 3, 4, 5}, {2, 3, 4, 5, 6}, . . . , {10, J, Q, K, A}. The sample space has size 52



5 , and each of these has 1

5
options. Thus, |E| = 10 41 , giving a probability of just under one percent.
But an “official” straight in poker is slightly different: a straight flush (i.e. all five cards are the same suit) is

5
not considered to be a straight. There are 10 41 possible straight flushes, so |E| = 10 41 − 10 41 now. (

Example 3.10. Consider flipping cards until an ace comes up, and take the next card. Is it more likely that the
ace of spades is this next card, or the two of clubs? Here, |S| = 52!, since the deck is shuffled. The first case, in
which the ace of spades is removed, and then reinserted after the first ace, there are 51! · 1 ways for this to
happen (since the other cards can be anywhere). Thus, the probability is 51!/52!.
The two of clubs has the same thing going on: there’s one place to put it, so the probability is the same. This
is counterintuitive. (

Example 3.11. Suppose 28% of students at a university program in Java, 7% program in C++, and 5% program
in both. Then, what percentage of students program in neither?
When answering a problem, define an event very precisely, as this will make errors less likely. So let A be the
probability that a randomly chosen student programs in Java and B be the probability that a student programs in
C++. Thus, the probability that someone programs in neither is 1−P (A∪B) = 1−(P (A)+P (B)−P (A∩B)) =
1 − (0.28 + 0.07 − 0.05) = 0.7, so the answer is 70%.
What percentage of programmers use C++, but not Java? This is P ((∼A) ∩ B) = P (B) − P (A ∩ B) =
0.07 − 0.05 = 0.02, or 2%.
Notice how similar this looks to a problem on this week’s homework. (

One important consequence is the birthday problem. What is the probability that in a room of n people, no
two share the same birthday? Then, |S| = 365n and |E| = (365)(364) · · · (365 − n + 1). Clearly bad things
happen if n ≥ 365. More interestingly, at 23 people, the probability that two people share a birthday is at least
1/2. This is much less than intuition would expect. At 75 people, it’s over 99%.
Now consider the probability that of n other people, none of them share a birthday with you? Here, |S| = 365n
again, and |E| = 364n . Thus, the probability is (364/365)n , and at n = 23, the probability that nobody’s
birthday matches yours is about 94%. Even at 160 people it’s only about 64%. In order to get up to a 50%
chance, you need about 253 people. That’s again quite different from intuition! The interaction of many-to-many
and one-to-many effects is tricky.

7Dr. Sahami actually had these with him. How long has it been since Hostess stopped producing them again?

7
 4. Conditional Probability: 4/8/13
First, it will be helpful to understand the notion of distinctness and indistinctness and ordered and unordered
sets. In many problems, it helps to temporarily view indistinct objects as distinct ones for the purposes of a
problem. However, one must be careful, because a common mistake is to assume that all options are equally
likely in a system.
Example 4.1. Suppose there are n balls to be placed in m urns (e.g. strings in a hash table, server requests to
m machines in a computer cluster, etc.). The counts of the balls in the urns aren’t equally likely. For example, if
there are 2 balls A and B and two urns 1 and 2, then all possibilities are equally likely, since each ball has a
1/2 probability of ending up in each particular urn.
But once A and B are taken to be indistinguishable, then the counts differ: there is one configuration in
which both balls end up in urn 1, but two in which one ball ends up in each urn. Be careful! (
Now, consider two fair, six-sided dice, yielding values D1 and D2 . Let E be the event D1 + D2 = 4. Then,
P (E) = 1/12, since there are 3 possibilities out of 36 total. Then, let F be the probability that D1 = 2. What is
P (E) if F has already been observed? Not all 36 rolls are possible, and S is reduced to {(2, 1), (2, 2), . . . , (2, 6)},
and the event space is {(2, 2)}. Thus, the probability P (E) given F is 1/6 — which is different. This basic idea
is one of the most important concepts of probability.
Definition 4.2. A conditional probability is the probability of an event E occurs given that some other event F
has already occured. This is written P (E | F ).
In this case, the sample space is reduced to those options consistent with F , or S ∩ F , and the event space is
reduced in the same way to E ∩ F . Thus, in the case of equally likely outcomes, P (E | F ) = |E ∩ F |/|S ∩ F | =
|E ∩ F |/|F |, because F ⊆ S.
This can be generalized to the general case, even when there aren’t equally likely outcomes: P (E | F ) =
P (E ∩ F )/P (F ), where P (F ) > 0. Probabilities are used here because counts aren’t as valid. This implies
something known as the “chain rule,” which shows that P (E ∩ F ) = P (E | F )P (F ). Intuitively, the probability of
both E and F happening is the probability that F happens, multiplied by the probability that one happens, then
the other happens given the first. This also happens to be commutative, so this is also equal to P (F | E)P (E).
If P (F ) = 0, then the conditional probability is undefined, because the statement “P (E) given that F happened”
makes no sense when F is impossible.
The chain rule is also known as the multiplication rule. The generalized version is
!  
\n Yn  i−1
\
P Ei = P Ei  Ej  = P (E1 )P (E2 | E1 )P (E3 | E1 ∩ E2 ) · · · P (En | E1 ∩ · · · ∩ En−1 ).

i=1 i=1 j=1

Note that this doesn’t require the events to be mutually exclusive, which makes it quite powerful.
Example 4.3. Suppose 24 emails are sent to four users, and are evenly distributed. If 10 of the emails are spam,
what is the probability user 1 receives 3 pieces of spam and user 2 receives 6 pieces of spam? Call the event for
user 1 E and for user 2 F . Then,
4 14



3
P (E | F ) = 18

3 ≈ 0.0784.
6
Here, user 2 can be thought of as a honeypot, which makes it less likely that spam reaches a legitimate user
assuming the spammer is rate-limited.
If the above also has G, which is the event that user 3 receives 5 spam emails, P (G | F ) = 0, since there
aren’t that many spam emails. (
Example 4.4. Suppose a bit string with m 0s and n 1s is sent over a network, so all arrangements are equally
likely. Then, if E is the probability that the first bit received is a 1, and F is the probability that k of the first r
bits is a 1, then
P (E ∩ F ) P (F | E)P (E)
P (E | F ) = =
P (F ) P (F )
n−1

m

k−1 r−k
P (F | E) = m+n−1

.
r−1
8
Notice that the bits are treated as distinct objects, which makes life
easier.
Then, P (E) = n/(m + n), which is
just counting how many of the bits are there, and P (F ) = nk r−k

m
/ m+n
r . Thankfully, things cancel out, and
in the end P (E | F ) = k/r.
Another way to think of this is that once we know there are k bits out of the first r are 1, then that’s all that
needs to be worried about: given this much smaller set, there are k possibilities for E among r choices given
F. (
Example 4.5. Suppose a deck of 52 cards is distributed into four piles, with 13 cards per pile. What is the
probability that each pile contains exactly one ace? Let
• E1 be the probability that the ace of spades is in any one pile,
• E2 be the probability that the ace of spades and ace of hearts are in different piles,
• E3 be the probability that the aces of spades, hearts, and clubs are in different piles, and
• E4 the solution: that every ace is in a different pile.
Then, P (E1 ) = 1, and P (E2 | E1 ) = 39/51, since there are 39 cards not in the pile with the ace of
spaces. Then, there are 26 cards left in the other piles, so P (E3 | E1 ∩ E2 ) = 26/50, and by the same logic
P (E4 | E1 ∩ E2 ∩ E3 ) = 13/49. Thus, the overall probability is
39 · 26 · 13
P (E4 ) = P (E1 ∩ E2 ∩ E3 ∩ E4 ) = ≈ 0.105.
51 · 50 · 49
Notice that a seemingly-convoluted problem becomes much easier from this viewpoint. (
One of the most influential figures in the history of probability was Thomas Bayes,8 who formulated an
extremely important theorem named after him. It has several fomulations: P (F | E) = P (E ∩ F )/P (E) =
(P (E | F )P (F ))/P (E). (Yes, it really is that simple.)
The textbook writes Bayes’ theorem as
P (E) = P (E ∩ F ) + P (E ∩ (∼F )) = P (E | F )P (F ) + P (E | ∼F )P (∼F ).
These can be combined into the most computationally useful one:
P (E | F )P (F )
P (F | E) = .
P (E | F )P (F ) + P (E | ∼F )P (∼F )
In the fully general form:
Theorem 4.6. Let F1 , . . . , Fn be a set of mutually exclusive and exhaustive events (i.e. the sample space is the
union of the Fi ). Then,
P (E ∩ Fj ) P (E | Fj )P (Fj )
P (Fj | E) − = Pn .
P (E) i=1 P (E | Fi )P (Fi )

This is useful if given that an event E has occured, one wishes to know whether one of the Fj also occured.
Example 4.7. Consider a test for HIV, which is 98% effective and has a false positive rate of 1%. It is known that
about 1 person in 200 has HIV in the United States, so let E be the probability that someone tests positive for
HIV with this test, and F be the probability that person actually has HIV. Then, P (E | F ) is the efficacy of the
test, and the probability that a positive result is true is
P (E | F )P (F )
P (F | E) =
P (E | F )P (F ) + P (E | ∼F )P (∼F )
(0, 98)(0, 005)
= ≈ 0.330.
(0, 98)(0.005) + (0.01)(1 − 0.005)
Oddly enough, that the test that has such a high effectiveness still doesn’t mean it returns the expected results!
This is because only a very small number of people actually have HIV relative to the false positive rate, so
a false positive is more likely than a real positive.9 Consider the four possibilities: Table 1 contains a lot of
information, and can be used to aid in calculations using Bayes’ Theorem. (

8. . . who according to Dr. Sahami looks very much like Charlie Sheen. I don’t see it.
9Relevant xkcd.

9
 Table 1

HIV+ HIV-
Test positive 0.98 = P (E | F ) 0.01 = P (E | ∼F )
Test negative 0.02 = P (∼E | F ) 0.99 = P (∼E | ∼F )

Example 4.8. Turning again to spam detection, suppose 60% of all email is spam, and 90% of spam has a
forged header, but 20% of non-spam has a forged header. One can check for a forged header, and let E be the
event that a message has a forged header, and F be the event that it is spam. Then, using Bayes’ Theorem,
P (F | E) = (0.9)(0.6)/((0.9)(0.6) + (0.2)(0.4)) ≈ 0.871, so this allows for a really good characterization of
whether email is spam. (
Example 4.9. Consider the Monty Hall problem, in which there are 3 doors A, B, and C. Behind one door is a
prize, and behind the other two is absolutely nothing. A contestant chooses a door, and then the game show
host opens one of the other doors to show it is empty. Then, the contestant can switch the door it chooses, but
is this a good idea?
• If the contestant doesn’t switch, then the probability of winning is 1/3, since opening the door doesn’t
make a difference.
• If it does switch, then without loss of generality assume the contestant chose A. If A was the winner,
then the probability of winning after switching is zero. If one of the other two cases occurs (each of
which happens with equal probability 1/3), the contestant wins. Thus, the overall probability of winning
is 2/3, so it is advantageous to switch.
This is generally confusing or hard to understand, so tread carefully. Suppose instead of three doors there were
1000. Then, the host opens 998 empty doors. What is the chance the winner is in the remaining door? Awfully
high, since the chance that you picked correctly initially is very small. (

5. More Conditional Probability: 4/10/13

“A bad statistician, afraid of terrorists when flying, brought a bomb on every flight, under the
reasoning that it was extremely unlikely that there would be two bombs on a flight!”10
Definition 5.1. The odds of an event A is defined as P (A)/P (∼A) = P (A)/(1 − P (A)). This is a positive
value, but is not bounded above.
Thus, one can reason about conditional odds. Given some event E, the odds of some hypothesis H is
P (H | E) P (H)P (E | H)/P (E) P (H)P (E | H)
= = .
P (∼H | E) P (∼H)P (∼H | E)/P (E) P (∼H)P (E | ∼H)
When E is observed, the odds of H are multiplied by P (E | H)/P (E | ∼H), in some sense the odds that H is
related to E being observed.
In Bayes’ Theorem, one has P (H | E) = P (E | H)P (H)/P (E). Here, P (H) is called the prior, because it
is the probability before any conditional expectation, P (E | H) is called the likelihood, and P (H | E) is called
the posterior (after the condition). This is a form of probabilistic inference, which uses probability to determine
whether a hypothesis is true after some evidence.
Example 5.2. An urn contains two coins A and B, where coin A comes up heads with probability 1/4, and B
returns heads with probability 3/4. If a random coin was chosen from the urn and comes up heads when flipped,
what are the odds that A was picked?
P (A | H) P (A)P (H | A) P (A)P (H | A) (1/2)(1/4) 1
= = = = .
P (∼A | H) P (∼A)P (H | ∼A) P (B)P (H | B) (1/2)(3/4) 3
This is odds, not probability; be careful! It means that there is a 1/3 ratio for A to B, so the probability becomes
1/4.
A priori, before the coin was flipped, the odds of picking A was 1 : 1, and the probability was 1/2. (
Here are some useful formulas for probability:
10On the subject of abuses of conditional probability, here is a relevant xkcd.

10
 • Commutativity: P (A ∩ B) = P (B ∩ A).
• The chain rule (multiplication rule), as seen above: P (A ∩ B) = P (A | B)P (B) = P (B | A)P (A).
• The intersection rule: P (A ∩ (∼B)) = P (A) − P (A ∩ B).
• Bonferroni’s inequality: P (A ∩ B) ≥ P (A) + P (B) − 1. This will be proved in the next problem set, but
it’s a four-line proof if you do it right.
Another notion is the generality of conditional probability: for any events A, B, and E, these all still hold in the
presence of event E:
• P (A ∩ B | E) = P (B ∩ A | E), and P (E | A ∩ B) = P (E | B ∩ A).
• The chain rule becomes P (A ∩ B | E) = P (A | B ∩ E)P (B | E).
• Bayes’ Theorem: P (A | B ∩ E) = P (B | A ∩ E)P (A | E)/P (B | E). This is the same as before, but
every probability has been conditioned on E.
There’s actually some philosophy behind this, but this can be seen as E as a summary of everything you already
know, and in some sense, every probability is a conditional probability, and without any such event E, there’s
no way to know any probability. How would you know the probabilities of a fair coin without any context?
Formally, this means that P ( | E) always satisfies the axioms of probability.
Example 5.3. For a simple example, take two six-sided dice D1 and D2 . Let E be the probability that D1 rolls a
1, and F be that D2 rolls a 1. Then, P (E) = P (F ) = 1/6, and P (E ∩ F ) = 1/36. Here, P (E ∩ F ) = P (E)P (F ).
Define a new event G, which is the probability that D1 + D2 = 5. Here, P (G) = 1/9, but P (E ∩ G) =
1/36 6= P (E)P (G). (
Definition 5.4. If E and F are two events such that P (E ∩ F ) = P (E)P (F ), then E and F are called
independent events; otherwise, they are called dependent events.
Independent events can be thought of as events which aren’t affected by each other. They also have the
property (via the chain rule) that P (E | F ) = P (E). Observing F doesn’t change the probability of E.
Claim. If E and F are independent events, then P (E | F ) = P (E | ∼F ). (
Proof.
P (E ∩ (∼F )) = P (E) − P (E ∩ F )
= P (E) − P (E)P (F )
= P (E)(1 − P (F ))
= P (E)P (∼F ),
so E and ∼F are indepedent events, so P (E | ∼F ) = P (E) = P (E | F ). The independence of E and F are
necessary in the second line, so that P (E ∩ F ) = P (E)P (F ). 

Definition 5.5. More generally, the events E1 , . . . , En are independent if for every subset E1 , . . . , Er (r ≤ n) of
E1 , . . . , En , it holds that !
\r Yr
P E1 = P (Ei ).
i=1 i=1
This just means that the product formula holds for all possible subsets. This sounds hairy, but isn’t all that
bad in practice.
Example 5.6. Roll two dice D1 and D2 , and let E be the event that D1 = 1, and F be that D2 = 6. Then,
let G be the event that D1 + D2 = 7. E and F are clearly independent. P (E) = 1/6 and P (G) = 1/6, but
P (E ∩ G) = 1/36, so they have to be independent. This is counterintuitive, but 7 is the exception, since there’s
always exactly one possibility for getting a 7 on D2 no matter the roll on D1 . Thus, it is important to be careful
when mathematics and intuition meet. Note also that F and G are independent by the same reasoning.
For more weirdness with intuiton, E and F are independent, E and G are independent, and F and G
are independent, but E, F , and G are not independent: P (E ∩ F ∩ G) = 1/36, but P (E)P (F )P (G) =
(1/6)(1/6)(1/6) = 1/216. (
Example 5.7. Suppose a computer produces a sequence of random bits, where p is the probability that a bit is a
1. Then, each bit is generated in an independent trial. Let E be the probability of getting n 1s followed by a 0.
The probability of n 1s is pn , and that of getting a 0 next is 1 − p, so P (E) = pn (1 − p). (
11
Example 5.8. Imagine a coin that comes up heads with probability p (really the same thing as the previous
example). Then, the probability of n heads in n flips is pn , and the probability of n tails in n flips is (1 − p)n .
The probability of k heads first, then all tails is pk (1 −
p)n−k . This requires order, so if the order is ignored,
then it’s a matter of combination, so the probability is nk pk (1 − p)n−k . There are k heads to place in n slots
if order is ignored. One could also place the n − k tails rather than the heads, and this is the same, since
n n
(



k = n−k .

Example 5.9. Consider a hash table in which m strings are hashed into n buckets with equal, independent
probability. Let E be the probability that at leat one string was hashed into the first bucket. In general, defining
one’s own events makes solving these problems much easier by breaking them into subproblems. Additionally,
the phrase “at least one” suggests solving by taking the complement, which is that no strings are hashed into
the first bucket.
Tm that string i is not hashed into the first bucket, so P (Fi ) = 1 − 1/n = (n − 1)/n for
Let Fi be the probability
every i. Then, E = ∼ i=1 Fi , so
m
! m  m
\ Y n−1
P (E) = 1 − P Fi = 1 − P (Fi ) = 1 − .
i=1 i=1
n

This is very similar to the birthday problem, which attempts to find collisions in a calendar rather than a hash
table.
Now, take m strings placed into a hash table with unequal probability: let pi be the probability that a string
is hashed into bucket i. This is a more accurate model for the real world, and has lots of applications. Then, let
E be the probability that at least one of buckets 1 through k has at least one string hashed into it. Let Fi be
the probability that at least one string is hashed into the ith bucket; then, using De Morgan’s Law,
k
! k
! k
!
[ [ \
P (E) = P Fi = 1 − P ∼ Fi = 1 − P ∼Fi .
i=1 i=1 i=1

This is a problem, because these events aren’t independent! If only one string is hashed, for example, knowing
where it isn’t adds some information about where else it might be. The strings are hashed independently, but
the events Fi are about buckets, not strings.
It looks like we’re stuck, but thinking about strings, the probability that it’ll not be hashed into the first bucket
is 1 − p1 , and the probability that it won’t be in the first two is 1 − p1 − p2 . This illustrates the technique of
stepping back and figuring out what it all means. Thus, the probability that no strings are hashed into buckets 1
to k is
k
! k
!m
\ X
P ∼Fi = 1 − pi ,
i=1 i=1

so the probability of the original event is

k
!m
X
P (E) = 1 − 1− pi .
i=1

Considering one final question about this hash table, let E be the probability that each of the buckets 1 to k
has a string hashed into it. Then, if Fi is the probability that at least one string is hashed into the ith bucket.
These events aren’t independent, so this becomes
P (E) = P (F1 ∩ · · · ∩ Fk ) = 1 − P (∼(F1 ∩ F2 ∩ · · · ∩ Fk ))
k
!
[
=1−P ∼Fi
i=1
k
X X
=1− (−1)r+1 P (∼(Fi1 ) ∩ · · · ∩ (∼Fir )),
r=1 i1 <···<ir

where P (∼Fi1 ∩ · · · ∩ ∼Fir ) = (1 − pi1 − · · · − pir )r as before, using the union formula. (
12
 Before the next example recall the geometric series
n
X 1 − xn+1
xi = .
i=0
1−x
As n → ∞, if |x| < 1, then the infinite sum reduces to 1/(1 − x) in the limit.
Example 5.10. Suppose two six-sided dice are rolled repeatedly. What is the probability that a 5 is rolled
before a 7? This is a simplified version of a game called craps. Define the series of events Fn , in which neither
a 5 nor a 7 is rolled in the first n − 1 trials, but a 5 was rolled on the nth trial. Thus, the probability of rolling a
5 on any trial is 4/36, and of a 7 on any trial is 6/36. Thus, the overall probability is
∞ ∞ ∞  ∞  ∞ 
!  n−1   X n−1   n
[ X X 10 4 26 4 4 X 26 2
P (E) = P Fn = P (Fn ) = 1− = = =
i=1 n=1 n=1
36 36 n=1
46 36 36 n=0 36 5
after using the geometric series. Notice that the events Fi are all mutually exclusive, so the probability of their
union is just the sum of their probabilities. (

6. Conditional Independence and Random Variables: 4/12/13

“There are three kinds of lies: lies, damned lies, and statistics.” – misattributed to Mark Twain
Returning to Example 5.10, there is a simpler way to look at it: the goal is to roll a 5 before rolling a 7, which
is P (5)/P (5 or 7) = (4/36)/(10/36) = 2/5. This is a much easier way to solve the problem, but it requires a
little more thought.
Example 6.1. Consider DNA paternity testing, in which a child is born with a pair of genes (A, a), which
will be event BA,a . The mother has a gene pair (A, A), and there are two possible fathers, M1 = (a, a) and
M2 = (a, A). Suppose the probability that M1 is the father is p (ignoring the child’s genes), so the probability
that M2 is the father is 1 − p. Then,
P (M1 ∩ BA,a ) P (BA,a | M1 )P (M1 ) 1·p 2p
P (M1 | BA,a ) = = = = > p.
P (BA,a ) P (BA,a | M1 )P (M1 ) + P (BA,a | M2 )P (M2 ) 1 · p + (1 − p)/2 1+p
Thus, this genetic test makes it more likely that M1 is the father. A priori, it makes sense that M1 is the father,
since there are more ways for M1 to contribute an a to the child than M2 to. (
One standard class of problems in probability is called coupon collecting: there exist N different sorts of
coupons, and each is collected with some probability pi . Then, one can ask questions such as:
• After collecting m coupons, what is the probability that there are k different kinds?
• What is the probability that one has at least one of each coupon after collecting m coupons?
These problems come up a lot in computer science, and have in fact come up before in this course, such as in
Example 5.9.
Recall that the laws of probability work when conditioning was thrown in, but this doesn’t hold true for
indepedence (a property): it is not necessarily true that if E and F are independent events, that P (E | G)P (F |
G) = P (E ∩ F | G). However, if all of the events are independent, then equality does hold.
Example 6.2. Roll two six-sided dice, giving values D1 and D2 . Then, let E be the event D1 = 1, F be the
event D2 = 6, and G be D1 + D2 = 7. E and F are independent, as was shown in Example 5.6. However,
P (E | G) = 1/6, P (F | G) = 1/6, and P (E ∩ F | G) = 1/6. Introducing new information can do weird things to
independence. (
Example 6.3. Suppose a dorm has 100 students, and 10 of them are CS majors. 30 students get straight As,
and there are 3 CS majors who get straight As. Let C be the probability that a randomly chosen student is a
CS major, and A be that a randomly chosen student gets straight As. Thus, P (C ∩ A) = 0.03 = P (C)P (A), so
C and A are independent.
Now, it’s faculty night, which means that only CS majors and A students show up, so 37 students arrive. 10
of them are CS majors, and 30 are A students, so P (C | C ∪ A) = 10/37. It looks like being a CS major lowers
the probability of getting good grades. . . which isn’t true. Once Faculty Night, a new event, is introduced, they
are dependent.
13
 This can be explained away (no, really; this line of reasoning is called explaining away). Everyone at Faculty
Night needs a reason for showing up. If a lawn is watered by either rain or sprinklers in independent events. If
the grass is wet, and then someone notices that the sprinklers are on, does that make it less likely that it has
rained? In general, if there are multiple things that can cause an observation and one of them is observed, that
makes the other causes less likely, since the event is already accounted for. (
Alternatively, if you go to the doctor with a cough, and the doctor says you might have a cold or dengue fever,
and the doctor says that a test result indicates you have a cold, then do you think you have dengue fever?
Not only can conditioning create dependence, it can destroy it.
Example 6.4. Let A be the event that it is not Monday on a randomly chosen day of the week, let B be that it
is Saturday, and let C be that it is the weekend.
Intuitively, A and B are dependent, and mathematically P (A) = 6/7, P (B) = 1/7 and P (A ∩ B) = 1/7.
However, after conditioning on C, P (A | C) = 1, P (B | C) = 1/2, and P (A∩B | C) = 1/2 = P (A | C)P (B | C).
Set-theoretically, this depends on how exactly the events are dependent. A field of research called Bayesian
networks investigates this; take CS 228 for more details. (
Definition 6.5. Two events E and F are called conditionally independent given an event G if P (E ∩ F | G) =
P (E | G)P (F | G), or, equivalently, P (E | F ∩ G) = P (E | G).
Exploiting conditional independence to make probabilistic computations faster was a major application of
computer science on probability theory (as opposed to the other way around).
Definition 6.6. A random variable is a real-valued function on a sample space.
This sounds kind of unclear, but one simple example is the number of heads obtained when three coins
are flipped. Intuitively, a random variable is something that can be measured as a result of observing some
probabilistic experiment. If Y is the random variable indicating the number of heads, then P (Y = 0) = 1/8,
P (Y = 2) = 3/8, and P (Y = 5) = 0.
Definition 6.7. A binary random variable is a random variable with two possible values.
This is the simplest possible random variable (other than the constant one, which is silly). Flipping a coin is
a good example of this, but keep in mind the results aren’t H and T , but real values corresponding to heads and
tails.
Example 6.8. Consider an urn with 11 balls: 3 are red, 3 are blue, and 5 are black. Then, suppose one draws 3
balls, and is awarded $1 for a blue ball, −$1 for a red ball, and 0 for the black ones. Then,
5 3 3 5





3 + 1 1 1 55
P (Y = 0) = 11

= ,
3
165
by considering all of the ways it can happen. Similarly,
3

5
3

3

1 2 + 2 1 39
P (Y = 1) = P (Y = −1) = 11

= .
3
165
P (Y = k) = P (Y = −k) by symmetry. (
This can be generalized:
Definition 6.9. A random variable is discrete if it can take on countably many values.
Definition 6.10. A probability mass function p (PMF) of a discrete random variable X is p(a) = P (X = a).
P∞
Thus, if X can take on values x1 , x2 , . . . , then i=1 p(xi ) = 1, so in general

p(xi ) ≥ 0, a = xi
P (X = a) =
0, a 6= xi for any i
Another way to understand a PMF is as a histogram of the potential values.
Definition 6.11. For a random variable X, the Cumulative Distribution Function (CDF) is defined as F (a) =
P (X ≤ a), with a ∈ R.
14
 Figure 2. The CDF for a fair six-sided die. Source

This isn’t terribly helpful in the discrete case, but will be much more essential in the continuous
P case, which
will pop up in a couple weeks. That said, the CDF of a discrete random variable is F (a) = x≤a p(x).
Definition 6.12. The expected value for a discrete random value X is
X
E[x] = xp(x).
{x|p(x)>0}

It’s necessary to specify that p(x) > 0 for various tricky analytic reasons. This is also known as the mean, the
center of mass, the average value, etc.
P6
Example 6.13. The expected value of rolling a six-sided die is E[X] = i=1 i(1/6) = 7/2. Notice that you
can’t actually roll a 7/2, but this is the average over a large number of rolls. (
Expected value is one of the most useful notions in probability.
Definition 6.14. A variable I is called an indicator variable for an event A if its value is 1 if A occurs, and 0 if
∼A occurs.
Then, P (I = 1) = P (A), so E[I] = (1)(P (A)) + (0)(P (∼A)) = P (A). This seems pretty tautological, but
it’s a huge concept: the expected value of the indicator is the probability of the underlying event.
Now, we have the mechanics to lie with statistics. Suppose a class has 3 classes with 5, 10, and 150 students.
If a class is randomly chosen with equal probability, then let X be the number of students in the class. Then,
E[X] = 5(1/3) + 10(1/3) + 150(1/3) = 55. The average value is 55, which seems reasonable.
But why do so many classes seem so large? If a student is randomly chosen and Y is the number of people
in that class, then E[Y ] = 5(5/165) + 10(10/165) + 150(150/165) ≈ 137, which is about twice as much! E[Y ]
is the student’s perception, and E[X] is what is usually reported.
This example illustrates that a lot of statistics involves interpretation, so the upshot is to be careful when
reading statistics.
If g is some real-valued function, then let Y = g(X). Then, the expectation is
X X X
E[g(X)] = E[Y ] = yj p(yj ) = yj p(xi ).
j j xi :g(xi )=yi

What this means is that one can obtain the probability for Y = k by looking at the cases of X that lead to
Y = k.
One consequence of this is linearity, in which g is a linear function. This means that E[aX + b] = aE[X] + b.
For example, if X is the value of a 6-sided die and Y = 2X − 1, then E[X] = 7/2, so E[Y ] = 6.
Definition 6.15. The nth moment of a random variable X is
X
E[X n ] = xn p(x).
x:p(x)>0
15
 This will be useful later; remember it!
The last concept for today is utility. If one has two choices with various consequences c1 , . . . , cn , where one
of the ci happens with probability pi , and has a utility (cost) U (ci ). For example. the probability of winning $1
million in a $1 lottery ticket is 10−7 . The utility of winning is $99999 (since you have to buy the ticket), and of
winning is −1. If the ticket isn’t brought, you can’t win, so the utility is zero. Then, the expected value of buying
a ticket can be calculated to be E ≈ −0.9. The takeaway lesson is that you can’t lose if you don’t play.

7. To Vegas: 4/15/13
“About the Binomial Theorem I am teeming with a lot o’ news. . . ” – The Major-General
There’s an interesting puzzle of probability called the St. Petersburg Paradox. Suppose one has a fair coin
(i.e. it comes up heads exactly half of the time). Let n be the number of coin flips before the first tails, and the
winning value is $2n .
How much would you pay to play? In expectation, the payoff is infinite over a large number of trials, but that
seems like a little much to pay. Formally, if X is the amount of winnings as a random variable,

∞  i+1 ∞
X 1 X 1
E[X] = 2i = ,
i=0
2 i=0
2

which is infinite. This seems weird, but mathematically, this is perfectly fine. Of course, for playing exactly once
one wouldn’t want to pay a million dollars. . .
Okay, now on to Vegas. Take some even-money game (e.g. betting red on a roulette table). Here, there is a
probability p = 18/38 that one wins $Y , and (1 − p) that you lose $Y . Make bets according to the following
algorithm:
(1) Let Y = $1.
(2) Bet Y .
(3) If this bet wins, then halt. Otherwise, multiply Y by 2 and go to step 2
Let Z be the amount of winnings upon stopping. Then,
 
∞  i   i−1  X ∞  i
X 20 18  i X j  18 20
E[Z] = 2 − 2 = (1) = 1
i=0
38 38 j=0
38 i=0 38

using the geometric series formula. The expected value of the game is positive, which is interesting. Thus, it can
be used to generate infinite money, right?
Well, maybe not:
• Real games have maximum betting amounts to prevent strategies such as this one (called Martingale
strategies).
• You have finite money, and are psychologically disinclined to spend it.
• Casinos like kicking people out.
Thus, in practice, in order to win infinite money, you need to have infinite money.
There are probability distributions with the same expected value, but in which the probabilities are “spread
out” over a larger area (e.g. a uniform distribution from 0 to 10 versus a large spike at 5). Thus, some techniques
have been introduced:

Definition 7.1. If X is a random variable with mean µ, then the variance of X, denoted Var(X), is Var(X) =
E[(X − µ)2 ].

Notice that the variance is always nonnnegative. The variance is also known as the second central moment,
or the square of the standard deviation.
16
 In practice, the variance isn’t computed by that formula:
Var(X) = E[(X − µ)2 ]
X
= (x − µ)2 p(x)
x
X
= (x2 − 2µx + mu2 )p(x)
x
X X X
= x2 p(x) − 2µ xp(x) + µ2 p(x)
x x x
= E[X 2 ] − 2µE[X] + µ2
= E[X 2 ] − E[X]2 , (7.2)
since E[X] = µ. This last formula is easier to use, and the quantity E[X 2 ] in particular is called the second
moment (different from the second central moment).
Variance is preserved under linear transformation: Var(aX + b) = a2 Var(X).

Proof.
Var(aX + b) = E[(aX + b)2 ] − (E[aX + b])2
= E[a2 X 2 + 2abX + b2 ] − (aE[X] + b)2
= a2 E[X 2 ] + 2abE[X] + b2 − (a2 (E[X])2 + 2abE[X] + b2 )
= a2 E[X 2 ] − a2 E[X]2 = a2 Var(X).
This depends on noticing that the expected value of a constant is just the constant itself, which makes sense. 
p
Definition 7.3. The standard deviation of X is SD(X) = Var(X).
This is convenient because it has the same units as X does, and is therefore easier to understand.
Definition 7.4. The Bernoulli random variable is an experiment that can result in success (defined as 1) or
failure (defined as 0). The probability of success is denoted p, and of failure 1 − p. The phrase “X is a Bernoulli
random variable with probability p of success” is shortened to X ∼ Ber(p).
The expectation of such a variable is E[X] = p, since the expectation of an indicator variable is just its
probability, and Var(X) = p(1 − p), which is a little harder to show.
Definition 7.5. A binomial random variable X denotes the number of successes in n independent trials of some
Bernoulli random variable Ber(p). Then, one writes X ∼ Bin(n, p), and P (X = i) = ni pi (1 − p)n−i .

By the Binomial Theorem, as n → ∞, the probability eventually becomes 1. For example, if one flips three
fair coins and takes X to be the number of heads, then X ∼ Bin(3, 0.5).
Example 7.6. This has a direct application to error-correcting codes (also called Hamming codes) for sending
messages over unreliable networks. Take some 4-bit message, and append three “parity” bits. The goal is to
take the message and choose parity bits in a way such that there are three sets with four bits, and each set has
an even number of 1s.
Then, if some bit is lost in transmission, it can be detected, and it can even be fixed, by taking all the
intersection of the sets with odd numbers of 1s and the complements of those with an even number (i.e. the
correct ones).
Suppose each bit in the example is flipped with probability 0.1 (which is much more than in the real world),
and if X is the number of bits corrupted, then X ∼ Bin(7, 0.1). Thus, the probability that a correct message is
received is P (X = 0) + P (X = 1). This can be calculated to be 0.8503, but without the error-correcting codes,
X ∼ Bin(4, 0.1) and P (X = 0) ≈ 0.6561. This is interesting because the reliability is significantly improved
despite only being a software update. (
Example 7.7. Consider a setup in which a person has two genes for eye color, and brown is dominant over blue
(i.e. if the child has a brown gene, then it has brown eyes, and if it has two blue genes, then it has blue eyes).
17
 Suppose each parent has one brown and one blue eye. Then, what is the probability that 3 children out of
four have brown eyes? The probability that a single child has blue eyes is 1/4, and that it has brown eyes is
thus 3/4.
Each child is an independent trial (in more ways that one), so X ∼ (4, 0.75), so P (X = 3) = 43 (0.75)3 (0.25) ≈

0.4219. This is considerably easier than modeling all of the combinatioons. (

Here are some properties of X ∼ Bin(n, p): consider the k th moment
n   n  
k n n i
X X
k i n−i
E[X ] = i p (1 − p) = p (1 − p)n−i .
i=0
i i=1
i
n n−1



Then, one can use i i = n i−1 (which can be verified by chasing factorials in the expansions of the formulas),
yielding
n  
k−1 n − 1
X
k
E[X ] = np i pi−1 (1 − p)n−i
i=1
i − 1
n−1  
X n−1 j
= np (j − 1)k−1 p (1 − p)n−(j+1) , where j = i − 1
j=0
j

= npE[(Y + 1)k−1 ],

where Y ∼ Bin(n − 1, p). Then, setting k = 1, E[X] = np, and if k = 2, then E[X 2 ] = npE[Y + 1] =
np((n − 1)p + 1), so the variance is Var(X) = E[X 2 ] − (np)2 = np(1 − p). After all this math, think about what
these mean: the expected value is what intuition would suggest. This also offers a proof for the Bernoulli case,
since Ber(p) = Bin(1, p).
Example 7.8. This can be used to ask how powerful one’s vote is. Is it better to live in a small state, where it
makes it more likely that the vote will change the outcome in that state, or a larger one, where there is a larger
outcome if the state does swing?
Adding some (not quite realistic) numbers: Suppose there are a = 2n voters equally likely to vote for either
candidate11 and the voter in question will be the deciding a + 1st vote. The probability that there is a tie is
  n  n
2n 1 1 (2n!)
P (tie) = = ,
n 2 2 n!n!22n
√ √
which simplifies to 1/ nπ after Stirling’s Approximation n! ≈ nn+1/2 e−n 2π. This is exceptionally accurate for
n ≥ 70, which is reasonable for this situation.
its probability multiplied by the number of electoral votes: since a is the size of
Then, the power of a tie is p
the state, then this becomes c 2a/π, which would means that living in a larger state means a more powerful
vote. (

8. The Poisson Distribution: 4/17/13

“Life is good for only two things: doing mathematics, and teaching mathematics.” – Simeon-Denis
Poisson
It’s possible to analyze the behavior of the binomial distribution in the limit: if X ∼ Bin(n, p), then P (X = i) =
n!(pi (1 − p)n−i )/(i!(n − i)!) Letting λ = np, this can be rewritten as
 i  n−i
n! λ λ n(n − 1) · · · (n − i + 1) λi (1 − λ/n)n
P (X = i) = 1− = .
i!(n − i)! n n ni i! (1 − λ/n)i
If n is large, p is small, and λ is “moderate” (the exact meaning of which will be clarified in a second), then the
first fraction is about 1, and (1 − λ/n)n ≈ e−λ and (1 − λ/n)i ≈ 1. In the limit, these all approach equality.
This can be used to approximate the binomial distribution for large n: P (X = i) = λi e−λ /i!.

11so they’re independent in at least two ways. . .

18
Definition 8.1. X is a Poisson12 random variable, denoted X ∼ Poi(λ), if X takes on nonnegative integer
values, and for the given parameter λ > 0,13 it has a PMF distribution that P (X = i) = e−λ λi /i!.
P∞ i
It’s necessary to check that this is in fact a probability distribution: using a Taylor series, eλ = i=0 λi! , so
∞ ∞
X X λi
P (X = i) = e−λ = e−λ eλ = 1.
i=0 i=0
i!

Example 8.2. Suppose a string of length 104 is sent over a network and the probability of a single bit
being corrupted is 10−6 . Using the Poisson distribution, λ = (104 )(10−6 ), P (X = 0) = e−0.01 (0.01)0 /0! ≈
0.990049834. Using a conventional binomial distribtuion, this is accurate to about nine decimal places, which is
incredibly useful. See Figure 3 for a graph of the accuracy of the Poisson distribution as an approximation. (
So what does “moderate” mean for λ, anyways? There are different interpretations, such as n > 20 and
p < 0.05, or n > 100 and p < 0.01. Moderate really just means that the approximation is reasonably accurate.

Figure 3. An illustration of the differences between binomial and Poisson distributions. Source

If X ∼ Poi(λ), where λ = np, then E[X] = np = λ, and Var(X) = np(1 − p) = λ(1 − 0) = λ as n → ∞ and
p → 0. Thus, λ represents the number of trials that are expected to be true.
Example 8.3. Intuitively, imagine that you are baking an extremely large raisin cake, which is cut into slices of
moderate size (with respect to the number of raisins in the slice of cake). The probability p that any particular
raisin is in any particular slice of cake is vanishingly small; if there are n cake slices, p = 1/n, assuming an
even distribution. If X is the number of raisins in a given cake slice, X ∼ Poi(λ), where λ = R/n, where R is
the total number of raisins. (
There are lots of concrete applications of this to computer science: one could sort strings into buckets in a
hash table, determine the list of crashed machines in some data center, or Facebook login requests distributed
across some set of servers.
Example 8.4. If computer chips are produced, so that p = 0.1 is the probability that a chip is defective. In
a sample of n = 10 chips, what is the probability that a sample contains at most one defective chip? n and
p aren’t extreme, so using the binomial distribution, P (Y ≤ 1) ≈ 0.7361, and with the Poisson distribution,
P (Y ≤ 1) ≈ 0.7358. This is still really close. (
Computing P (X ≤ a) when X ∼ Poi(λ) straightforwardly is computationally expensive, but it turns out that
λ
P (X = i + 1)/P (X = i) = λ+1 , so one can just do one approximation and then multiply, which is pretty fast.

12Poisson literally means “fish” in French, though this is not the reason that this distribution has its name.
13Relevant xkcd.

19
 A Poisson distribution is an approximation, so it still approximately works for things that aren’t really binomial
distributions. This is known as the Poisson paradigm. If the dependency is mild, then the approximation is
still pretty close. For example, in a large hash table, the number of entries in a bucket are dependent events,
but it grows weaker with the size of the hash table. Additionally, if the probability of success in each trial
varies slightly, the Poisson distribution still works pretty well (e.g. when load on a network varies slightly with
time). In each of these cases, the binomial random variable is also a good approximation, but it comes with the
connotation that the probability is exact, whereas the Poisson distribution doesn’t.
Returning to the birthday problem, take m people and let Ex,y be the event that people x and y have the
same birthday. Thus, P (Ex,y ) = 1/365 = p, but not all of these events are independent (e.g. if Alice and Bob
have the same birthday and Bob and Eve have the same birthday, do Alice
and Eve have the same birthday?).
Thus, this isn’t a binomial distribution, so let X ∼ Poi(λ), where λ = p m 2 = m(m − 1)/730. Then,

m(m−1) (m(m − 1)/730)0 m(m−1)

P (X = 0) = e− 730 = e− 730 .
0!
Then, the smallest m for which this probability is less than 0.5 can be solved to m ≥ 23, which we know to be
correct from before.
A Poisson process is a rare event that occurs over some time (e.g. radioactive decay, earthquakes, hits to a
web server on the small scale). If there is some time interval over which these events are rare, then the events
arrive at λ events per that interval of time. Then, split the time into n subintervals, where n is large. Events
occuring in subintervals are independent, and within any particular subinterval, the probability of the event
happening is small. Thus, a Poisson distribution can be used.
Example 8.5. Suppose a server averages two hits per second. Let X be the number of hits the server receives
in a second. If X ≥ 5, then the server crashes. How often does this happen?
Suppose the sever can process at most one request per millisecond, so n = 1000. In a particular millisecond,
the chance of getting a hit is p = 2/1000 = 1/500. Then, λ = np = 2, X ∼ Poi(2), and P (X = 5) can be
calculated. (
Definition 8.6. X is a geometric random variable, denoted X ∼ Geo(p), if X is a random variable denoting the
number of independent trials with probability p of success on each trial s.t. all but the last trial are failures, and
the last is success. Then, P (X = n) = (1 − p)n−1 p.
Then, E[X] = 1/p and Var(p) = (1 − p)/p2 .
Example 8.7.
• Flipping a fair coin until the first heads appears.
• Drawing balls from an urn that contains both white and black with replacement until a white ball is
drawn.
• Generating random bits until a 1 is gnerated. (
This seems a bit easier than the binomial distribution, so let’s try another one:
Definition 8.8. X is a negative binomial random variable, denoted X ∼ NegBin(r, p), if X is the number of
independent trials until r successes, where p is the probability of success on each trial. X takes on values
X ≥ r with probability
 
n−1 r
P (X = n) = p (1 − p)n−r ,
r−1
where n ≥ r. Notice that there is no upper bound on n, though the probability decreases after some point.

The n−1


r−1 term isn’t obvious, but the last trial must end in success, so the number of choices is restricted.
Notice additionally that Geo(p) ∼ NegBin(1, p), so the geometric distribution is analogous to the Bernoulli
distribution for the negative binomial distribution.
Definition 8.9. X is a hypergeometric random variable, denoted X ∼ HypG(n, N, m), if it varies as in the
following experiment: consider an urn with N balls, in which N − m are black and m are white. If n balls are
drawn without replacement, then X is the number of white balls drawn.
20

−m
N

This is just combinatorics: P (X = i) = mi Nn−i / m , where i ≥ 0. Then, E[X] = n(m/N ) and
Var(X) = (nm(N − n)(N − m))/(N 2 (N − 1)).
Let p = m/N , the probability of drawing a white ball on the first draw. Note that as N → ∞,
HypG(n, N, m) → Bin(n, m/N ). The binomial is the case with replacement, and the hypergeometric is
the case without replacement. As mentioned above, one can be made to approximate the other.
Example 8.10. Suppose N is the number of individuals of some endangered species that remain. If m of them
are tagged, and then allowed to mix randomly, then randomly observe another n of them. Let X be the number
of those n that were tagged. Then, X ∼ HypG(n, N, m). Then, using something called a maximum likelihood
estimate, N̂ = mn/i is the value that maximizes P (X = i). (

9. From Discrete to Continuous: 4/19/13

The stuff we have been discussing has come up in the real world. The Supreme Court case Berghuis v. Smith
dealt with determining how to detect whether a group is underrepresented in a jury pool.
Justice Breyer opened the questioning by invoking the binomial theorem. He hypothesized a
scenario involving “an urn w/ 1000 balls, 60 of which are red, 940 are black, and you select them
randomly twelve at a time.” He then estimated that if the red balls represented black jurors,
then something like one-third to one-half of the juries would have a black person on them.
Justice Breyer was wrong, though, since the balls are drawn without replacement. Thus, this is
a hypergeometric
distribution, rather than binomial, and the probability that twelve black balls are drawn is 940 1000


12 / 12 ≈ 0.4739,
so the probability that at least one red ball is drawn is one minus that, or about 0.5261. Not quite what Justice
Breyer got.
But this can be approximated with the binomial distribution, since in a large pool replacement doesn’t make
much of a difference. The probability doesn’t change: the number of red balls drawn, X is X ∼ Bin(12, 60/1000),
so P (X ≥ 1) = 1 − P (X = 0) ≈ 1 − 0.4759 = 0.5240, which isn’t much different.14 The upshot is that people
make mistakes all the time in probability, and it’s worth fact-checking. There’s actually some nuances depending
on how the jury selection is modeled, but the numbers don’t change very much.
So far, the random variables seen in this class have been discrete. However, continuous random variables
exist: these can take on an uncountable range of values, typically representing measurement (height, weight,
time). The distinction is one between “how much?” and “how many?” Mathematically, this means replacing
sums with integrals.
Definition 9.1. X is a continuous random variable if there is a nonnegative function f (x) on R such that
Rb
P (a ≤ X ≤ b) = a f (x) dx.
R∞
f is a probablity density function (PDF) if −∞ f (x) = 1.
An important distinction to be made is that f is not a probability. The unitsR don’t match, and f (x) can
a
take on values greater than 1. There is the additional quirk that P (X = a) = a f (x) dx = 0. Continuous
random variables are useful for ranges, not exact numbers. Contrast this with a probability mass function: in the
countable case, individual values can have nonzero probabilities.
Definition 9.2. For a continuous
Ra random variable X, the cumulative distribution function (CDF) is F (X) =
P (X ≤ a) = P (X < a) = −∞ f (x) dx.

Thus, the density f is just the derivative of the CDF: f (a) = dF

da . Then, if f is continuous and ε is small,
Z a+ε/2
 ε ε 
P a− ≤X ≤a+ = f (x) dx ≈ εf (a).
2 2 a−ε/2

This is important, because it allows ratios of probabilities to be meaningfully defined:

P (X = 1) εf (1) f (1)
≈ = ,
P (X = 2) εf (2) f (2)
since ε → 0 is perfectly fine in the limit. Thus, things such as conditional probability are meaningful.

14The cause of misestimation may have been that CS 109 didn’t exist when Justice Breyer was at Stanford. . .

21
Example 9.3. Suppose X is a continuous random variable (CRV) with PDF
C(4x − 2x2 ), 0 < x < 2,

f (x) =
0, otherwise.
First, C can be calculated:
2 2
2x3
Z 
C(4x − 2x2 ) dx = C 2x2 − = 1,
0 3 0
so plugging in, C((8 − 16/3) − 0) = 1, so C = 3/8. Then,
Z ∞ Z 2 2
2x3

3 3 1
P (X > 1) = f (x) dx = (4x − 2x2 ) dx = 2x2 − = ,
1 1 8 8 3 1 2
which is what makes sense (since the function is symmetric). (
Example 9.4. Suppose X is the amount of time (in days) before a disk crashes. It is given by the PDF
x
λe− 100 , x ≥ 0

f (x) =
0, otherwise.
First, λ will be calculated:
Z ∞ ∞
e−x/100 ∞
Z
1= λe−x/100 dx = −100λ − dx = −100λe−x/100  = 100λ,

0 0 100 0

so λ = 1/100. Technically, an improper integral was taken here and sort of glossed over.
Then, what is P (50 < X < 150)?
Z 100 150
1 −x/100
F (150) − F (50) = e dx = −e−x/100  ≈ 0.383.

50 100 50

We can also calculate P (X < 10):

Z 10 10
1 −x/100
F (10) = e dx = −e−x/100  ≈ 0095. (

0 100 0

Expectation and variance can also be cacluated: if X is a continuous random variable,

Z ∞ Z ∞ Z ∞
n
E[X] = xf (x) dx E[g(X)] = g(x)f (x) dx E[X ] = xn f (x) dx.
−∞ −∞ −∞

Linearity of expectation is still a thing: E[aX + b] = aE[X] + b, and the same proof can be given. Similarly, the
formulae for variance still hold.
Example 9.5. For example, if X has linearly increasing density, given by f (x) = 2x on [0, 1] and f (x) = 0
otherwise, the expectation isn’t obvious from the graph, but can be found as
Z ∞ Z 1
2
E[X] = xf (x) dx = 2x2 dx =
−∞ 0 3
and the variance is given by
Z ∞ Z 1
1
E[X 2 ] = x2 f (x) dx = 2x3 dx = ,
−∞ 0 2
2 2
so Var(X) = E[X ] − E[X] = 1/2 − (2/3) = 1/18. 2
(
These are arguably easier than in the discrete case.
Definition 9.6. X is a uniform random variable, denoted X ∼ Uni(α, β), if its probability density function is
 1
f (x) = β−α , α ≤ x ≤ β
0, otherwise.
This is just a horizontal line.
22
 Sometimes, this is defined only on (α, β), but this causes issues later on. The probability is given by
Z b
b−a
P (a ≤ x ≤ b) = f (x) dx = ,
a β −α
which is valid when α ≤ a ≤ b ≤ β. The expectation is even easier: it can be formally calculated to be (α + β)/2,
which is no surprise, and the variance is (after a bunch of math) Var(X) = (β − α)2 /12.
Example 9.7. Suppose X ∼ Uni(0, 20). Then. f (x) = 1/20 for 0 ≤ x ≤ 20 and f (x) = 0 otherwise. Thus,
R6 R 17
P (X < 6) = 0 dx/20 = 6/20, and P (4 < X < 17) = 4 dx/20 = (17 − 4)/20 = 13/20. (
Example 9.8. Suppose the Marguerite bus arrives at 15-minute intervals (on the quarter hour), and someone
arrives at a bus stop uniformly sometime between 2 and 2:30. Thus, X ∼ Uni(0, 30). If the passenger waits less
than 5 minutes for the bus, then it must arrive between 2:10 and 2:15, or 2:25 to 2:30. This is 10 minutes out of
the 30, so the probability is 1/3. (Since the distribution is uniform, this sort of calculation can be done.) The
passenger waits for more than 14 minutes if it arrive between 2 and 2:01 or 2:15 to 2:16. These are 2 out of the
30 minutes, so the probability is 1/15. (
Example 9.9. Suppose a student bikes to class, and leaves t minutes before class starts, which is (usually) a
choice by the student. If X is the travel time in minutes, given by a PDF of f (x). Then, there is a cost to be
early or late to class: C(X, t) = c(t − X) if early, and C(X, t) = k(t − X) if late (which is a different cost).
Thus, the cost function is discontinuous. Then, choose t that minimizes E[C(X, t)]:
Z ∞ Z t Z ∞
E[C(X, t)] = C(X, t)f (x) dx = c(t − x)f (x) dx + k(x − t)f (x) dx.
0 0 t

Now we need to minimize it, which involves some more calculus. Specifically, it requires the Leibniz integral
rule:
Z f2 (t) Z f2 (t)
d df2 (t) df1 (t) ∂ g(x, t)
g(x, t) dx = g(f2 (t), t) − g(f1 (t), t) + dx.
dt f1 (t) dt dt f1 (t) ∂t
Let t∗ be the optimal time. Then,
Z t Z ∞
d
E[C(X, t)] = c(t − t)f (t) + cf (x) dx − k(t − t)f (t) kf (x) dx
dt 0 t
0 = cF (t∗ ) − k(1 − F (t∗ ))
k
=⇒ F (t∗ ) = .
c+k
The question comes down to whether being late is much worse than being early. If so, this becomes closer to
1, so t should be earlier. Similarly, if being early is a problem, then this becomes close to 0, so t should be
later. (

10. The Normal Distribution: 4/22/13

“Welcome to week four of CS 106A!”
Definition 10.1. X is a normal random variable (sometimes called Gaussian15), denoted X ∼ N(µ, σ 2 ), if it has
2 2 √
a PDF of f (x) = e−(x−µ) /2σ /(σ 2π), where µ represents the mean and σ the standard deviation.
This takes on a nonzero value for all x ∈ R. It has very nice moments: E[X] = µ and Var(X) = σ 2 . This is a
very common distribution for natural phenomena because it results from the sum of multiple roughly independent
variables.
Here are some properties of a normal random variable:
• If X ∼ N(µ, σ 2 ) and Y = aX +b, then by linearity of expectation Y ∼ N(aµ+b, a2 σ 2 ), so E[Y ] = aµ+b
and Var(Y ) = a2 σ 2 .

15Apparently C.F. Gauss, who was responsible for popularizing this distribution, looks like Martin Sheen.

23
 Figure 4. A normal distribution. Source

• Looking at the CDF, with X and Y as above,

   
x−b x−b
FY (x) = P (Y ≤ X) = P (aX + B ≤ x) = P X ≤ = FX .
a a
Obviously, this causes problems if a = 0, but that’s a silly case. After differentiating, one obtains the
PDF:    
d d x−b 1 x−b
fY (x) = FY (x) = FX = fX .
dx dx a a a
• In the special case where Z = (X − µ)/σ, this is just a = 1/σ and b = −µ/σ, so
µ µ σ2
 
2 2
Z ∼ N(aµ + b, a σ ) = N − , = N(0, 1).
σ σ σ2
This transformation will be useful.
A random variable Z ∼ N(0, 1) is called a standard (unit) normal random variable, and has E[Z] = 0, Var(Z) = 1,
and SD(Z) = 1. However, the CDF of Z doesn’t have a closed form, and must be solved numerically: denote the
CDF of a unit normal FZ (z) = Φ(z), which satisfies
Z z −(x−µ)2 /2σ2 Z z −x2 /2
e dx e
Φ(z) = P (Z ≤ z) = √ = √ dx.
−∞ σ 2π −∞ 2π
By symmetry, Φ(−z) = P (Z ≤ −z) = P (Z > z) = 1 − Φ(z). Geometrically, flipping Φ about the y-axis converts
the region Z ≤ −z into Z > z, which has probability 1 − Φ(z), and the transformation is area-preserving. This
is important to understand because tables of the value of Φ don’t store negative values, since they can be so
easily calculated from the positive ones. Thus, Z can be used to compute the CDF for an arbitrary normal
random variable X ∼ N(µ, σ 2 ), for σ > 0. Thus,
FX (x) = P (X ≤ x)
 
X −µ x−µ
=P ≤
σ σ
   
x−µ x−µ
=P Z≤ =Φ ,
σ σ
with Z ∼ N(aµ + b, a2 σ 2 ). Then, one can use a table of values for Φ(z) to calculate the required answer.
Example 10.2. Suppose X ∼ N(3, 16). Thus, µ = 3 and σ = 4. Then, let Z be the unit normal, so that
     
X −3 0−3 3 3
P (X > 0) = P > =P Z>− =1−P Z ≤−
4 4 4 4
      
3 3 3
=1−Φ − =1− 1−Φ =Φ ≈ 0.7734
4 4 4
24
and
   
2−3 X −3 5−3 1 2
P (2 < X < 5) = P < < =P − <Z<
4 4 4 4 4
   
2 1
=Φ −Φ − ≈ 0.6915 − (1 − 0.5987) = 0.2902.
4 4
More interestingly,
   
3 3
P (|X − 3| > 6) = P (X < −3) + P (X > 9) = P Z < − +P Z >
2 2
   
3 3
=Φ − +1−Φ ≈ 1(1 − 0.9332) = 0.1336.
2 2
Other transformations might exist, but they can be solved in roughly the same way. (
Example 10.3. Suppose a bit is sent on a wire with a voltage of either 2 V or −2 V to represents bits of 1 and
0, respectively. Let X be the voltage sent and R be the voltage received. R = X + Y , where Y ∼ N(0, 1) is the
error, called white noise. Then, R is decoded as 1 if R ≥ 0.5, and 0 otherwise. What is the probability of an
error in decoding given that X = 1? This is P (2 + Y < 0.5) = P (Y < −1.5) = Φ(−1.5) = 1 − Φ(1.5) ≈ 0.0668,
since Y is a standard normal random variable, making the computation a bit simpler. If X = 0, the probability of
an error is instead P (−2 + Y ≥ 0.5) = P (Y ≥ 2.5) = 1 − Φ(2.5) ≈ 0.0062. These two are different because
the decoding isn’t symmetric involving the distribution of Y , which seems kind of odd, but happens all the time,
in which the two cases are of different importance. For an analogy, suppose the bit represents a signal to launch
the nuclear missles, or not; clearly, one kind of error is much worse than the other. (
The normal approximation can be used to approximate the binomial distribution. Suppose X ∼ Bin(n, p), so
that E[X] = np and Var(X) = np(1 − p). The Poisson approximation is good for n large (about n > 20) and p
small (about 0.05 or less). For large n, we also have X ≈ Y ∼ N(E[X], Var(X)) = N(np, np(1 − p)), which is
good when Var(X) ≥ 10. The deciding value is how spread the data is. Using this approximation,
 
1 1
P (X = k) ≈ P k − < Y < k +
2 2
! !
k − np + 0.5 k − np − 0.5
=Φ p −Φ p .
np(1 − p) np(1 − p)
Here, something that is discrete is approximated by something continuous. This relies on the continuity correction,
which integrates the continuous quantity over a step of size 1.16
Theorem 10.4 (DeMoivre-Laplace Limit Theorem). Let Sn be the number of successes in n independent trials,
where success is given by probability p, then
!
Sn − np
lim P a ≤ p ≤ b = Φ(b) − Φ(a).
n→∞ np(1 − p)
This theorem guarantees the approximation we have just taken. It provides the foundation for the Central
Limit Theorem, which will be seen later, and illustrates why the normal distribution is so common in nature.
Example 10.5. Suppose 100 people are placed on a special diet, and X is the number of people on the diet
whose cholesterol increases. Then, a doctor will endorse the diet of X ≥ 65. What is the probability that the
doctor endorses the diet despite it having no effect? This is represented by X ∼ Bin(100, 0.5), so np = 50 and
the standard deviation is 5. Then, we need the case P (X ≥ 65), where ≥ is distinct from >, because X is a
discrete random variable.
Use a normal approximation: P (X ≥ 65) ≈ P (Y > 64.5). (If it were necessary to calculate P (X > 65), then
one would use P (Y > 65.5).) This is
 
Y − 50 64.5 − 50
P (Y ≥ 64.5) = P > = 1 − Φ(2.9) = 0.0019.
5 5
Here, the doctor is right. (
16One could use a different step size, but then some other constant for the step size would have to be multiplied in, which is unpleasant.

25
Example 10.6. Suppose Stanford admits 2480 students and each student has a 68% chance of attending. If
Stanford has room for 1745 students to matriculate, then let X be the number of students who attend, so that
X ∼ Bin(2480, 0.68). Thus, µ = 1686.4 and σ 2 ≈ 539.65, so this can be approximated by the normal variable
Y ∼ N(1686.4, 539.65), and
 
Y − 1686.4 1745.5 − 1686.4
P (X > 1745) = P (Y ≥ 1745.5) = P > = 1 − Φ(2.54) ≈ 0.0055.
23.23 23.23
This is a good approximation, because P (X > 1745) = 0.0053, which is very close. (

11. The Exponential Distribution: 4/24/13

“Remember, don’t drink and do probability. Not that I know from experience — and we’re just
talking about tea anyways.”
Definition 11.1. X is an exponential random variable, denoted X ∼ Exp(λ) for some λ > 0, if its PDF is
λe−λx , x ≥ 0,

f (x) =
0, x < 0.
Then E[X] = 1/λ and Var(X) = 1/λ2 . This just looks like any decreasing exponential function, as in radioactive
decay. Its CDF is F (x) = 1 − e−λx , since f is easy to integrate. This sort of distribution tends to represent the
amount of time until some event (e.g. earthquakes, requests to a web server, etc.), which is why x ≥ 0 and λ > 0.
This is a “memory-less” distribution, and in fact under reasonably mild conditions is the only such distribution.
Let X ∼ Exp(λ); then,
P (X > s + t and X > s) P (X > s + t)
P (X > s + t | X > s) = =
P (X > s) P (X > s)
1 − F (s + t) e−λ(s+t)
= = = e−λt
1 − F (s) e−λs
= 1 − F (t) = P (X > t).
This means that the starting point is irrelevant: if you’re waiting for something at time t with this distribution
and s years have passed, then the expected value of waiting hasn’t changed! In some sense, this means the
curve is shaped like itself (which follows from the fact that f 0 = λf ).
Example 11.2. Suppose a visitor to some website leaves after X minutes, and on average a visitor leaves after
5 minutes. Thus, X ∼ Exp(1/5), since E[X] = 5 = 1/λ. Then, P (X > 10) = 1 − F (10) = 1 − (1 − e−10λ ) =
e−2 ≈ 0.1353 and P (10 < X < 20) = F (20) − F (10) = (1 − e−4 ) − (1 − e−2 ) ≈ 0.1170. (
Example 11.3. Let X be the number of hours of use until a laptop dies. On average, this is 5000, so λ = 1/5000.
Then, make the rather conservative assumption that the laptop is used five hours per day. The probability that
the laptop lasts four years is (ignoring leap years) P (X > (5)(365)(4)) = P (X > 7300) = 1 − F (7300) =
1 − (1 − e−7300/5000 ) = e−1.46 ≈ 0.2322. After five years, this becomes 0.1612 and after six, it becomes 0.1119.
And thanks to the memoryless property, any one crash doesn’t make the rest any less likely. (
Recall some concepts from calculus: the product rule and integration by parts, which are just inverses of
each other. This is useful for computing the nth moments: E[X n ] = xn λe−λx dx. Then, let u = xn , so that
du = nxn−1 dx, and v = −e−λx , so dv = λe−λx dx. Thus:
Z ∞
n
E[X ] = xn λe−λx dx
0
Z ∞ Z ∞
∞
= u dv = uv|0 − v du
0 0
Z ∞
∞
= −xn e−λx 0 + nxn−1 e−λx dx
0
n ∞ n−1 −λx
Z
n
= x λe dx = E[X n−1 ].
λ 0 λ
This creates a recurrence that can be solved to generate E[X ] = n!/λn . The first two moments are the most
n

useful.
26
Definition 11.4. For two discrete random variables X and Y , define the joint probability mass function of X
P Y to be pX,Y (a, b) = P (X = a, Y = b). Then, the marginal distribution of X is pX (a) = P (X = a) =
and
y pX,Y (a, y), and the marginal distribution of Y can be defined in the same way.

Example 11.5. Suppose a household has C computers, of which X are Macs and Y are PCs. Assume any
given computer is equally likely to be a Mac or a PC. Then, define some probabilities: P (C = 0) = 0.16,
P (C = 1) = 0.24, P (C = 2) = 0.28, and P (C = 3) = 0.32. Thus, one can make a joint distribution table by
taking all the options that correspond to C = c and splitting the probability given that each computer is equally
likely to be of either type. Then, summing along the rows and columns gives the marginal distributions.17 (
This can be generalized to continuous distributions:
Definition 11.6. For two continuous random variables X and Y , their joint cumulative probability distribution is
FX,Y (a, b) = F (a, b) = P (X ≤ a, Y ≤ b) where a, b ∈ R.
Then, the marginal distributions are FX (a) = P (X ≤ a) = FX,Y (a, ∞), and FY is defined in the same way.
This is no longer a table, but rather a two-dimensional graph, as in Figure 5.

Figure 5. A joint probability distribution given by FX,Y (x, y) = 20(y − x)(1 − y) for x, y ≥ 0.

Definition 11.7. Two random variables X and Y are jointly continuous if there exists a PDF fX,Y (x, y) : R2 → R
such that Z Z a2 b2
P (a1 ≤ X ≤ a2 , b1 ≤ Y ≤ b2 ) = fX,Y (x, y) dy dx.
a1 b1
and the CDF is given by
Z a Z b
FX,Y (a, b) = f (x, y) dy dx.
−∞ −∞

∂ 2 FX,Y (a,b)
Conversely, one can obtain the PDF from the CDF as fX,Y (a, b) = ∂x∂y . The marginal density functions
are thus Z ∞ Z ∞
fX (a) = fX,Y (a, y) dy and fY (b) = fX,Y (x, b) dx.
−∞ −∞
If these sorts of multiple integrals are foreign, see them as iterated integrals: integrate the innermost integral
as in single-variable integrals, and treat all other variables as constants. This can be repeated, leading to a
solution. For example,
Z 2Z 1 Z 2 Z 1  Z 2  2 1 Z 2  2 2
x y y
xy dx dy = xy dx dy = y dy = dy = = 1.
0 0 0 0 0 2 0 0 2 2 0
This can be generalized to three or more integrals, but that’s not going to be required in this class. Intuitively,
double integrals should be thought of as integrating over some area.

17This is the source of the name: they’re literally written in the margins of the table.

27
 Additionally,
P (X > a, Y > b) = 1 − P (∼(X > a, Y > b)) = 1 − P ((X ≤ a) ∪ (Y ≤ b))
= 1 − (P (X ≤ a) + P (Y ≤ B) − P (X ≤ A, Y ≤ B))
= 1 − FX (a) − Fy (B) + FX,Y (a, b).
Similary, P (a1 ≤ X ≤ a2 , b1 ≤ Y ≤ b2 ) = FX,Y (a1 , b1 ) + Fx,y (a2 , b2 ) − FX,Y (a1 , b2 ) − FX,Y (a2 , b1 ). This can
be seen by drawing a picture. P
If X is a discrete random variable, then E[X] = i iP (X = i), which can be generalized to the continuous
case:
Lemma 11.8. If Y is a non-negative continuous random variable with PDF F (y), then
Z ∞ Z ∞
E[Y ] = P (Y > y) dy = (1 − F (y)) dy.
0 0

Proof.
Z ∞ Z∞ Z∞
P (Y > y) dy = fY (x) dx dy
0
y=0 x=y
Z∞ Z∞
 

=  dy fY (x) dx
x=0 y=0
Z ∞
= xfY (x) dx = E[Y ]. 
0

12. The Multinomial Distribution and Independent Variables: 4/26/13

“So the moral of the story is to not be so variable in your arrival time.”
The multinomial distribution is used to model n independent trials of some experiment, but each trial has one
of m outputs (unlike the binomial distribution, where m = 2). Additionally, they do not need to have equal
probabilities: outcome i has probability pi , so the sum of the pi s is 1. Then, Xi represents the number of trials
with outcome i. Then,
 Ym m
n X
P (X1 = c1 , . . . , Xm = cm ) = pci i where ci = n.
c1 , . . . , cn i=1 i=1

Example 12.1. Suppose a six-sided die is rolled 7 times. The probability of one 1, one 2, no threes, two 4s, no
5s, and three 6s is
 1  1  0   2  0  3
7! 1 1 1 1 1 1 1
P (X1 = 1, X2 = 1, X3 = 0, X4 = 2, X5 = 0, X6 = 3) =
1!1!0!2!0!3! 6 6 6 6 6 6 6
420
= 7 . (
6
The multinomial distribution can be used in probabilistic text analysis: what is the probability that any given
word appears in a text? In some sense, each word appears with some given probability, which makes this a
giant multinomial distribution. This can be conditioned on the author, which makes sense: different people
use different writing styles. But then, it can be flipped around: using Bayes’ theorem, one can determine the
probability of a writer for a given set of words. This was actually used on the Federalist Papers from early
American history, which allowed for a probabilistic identification of which of Hamilton, Madison, and Jay wrote
each paper. Similarly, one can use this to construct spam filters, as it allows one to guess whether the author of
an email was a spammer.
Independence has already been discussed in the context of events, but it also applies to variables. The
intuition is exactly the same: knowing the value of X indicates nothing about that of Y .
Definition 12.2. Two discrete random variables X and Y are indepedent if p(x, y) = pX (s)pY (y) for all x, y.
Two variables that aren’t independent are called dependent.
28
Example 12.3. Take a coin with a probability p of heads and flip it m + n times. Then, let X be the number of
heads in the first n flips and Y be the number of heads in the next m flips. It is not surprising that
   
n x n−x n
P (X = x, Y = y) = p (1 − p) py (1 − p)m−y = P (X = x)P (Y = y).
x y
Thus, X and Y are independent.
More interestingly, let Z be the number of total heads in all n + m flips. Then, X and Z aren’t independent
(e.g. Z = 0), nor are Y and Z. (
Example 12.4. Let N ∼ Poi(λ) be the number of requests to a web server in a day. If each request comes
from a human with probability p and a bot with probability 1 − p, then let X be the number of humans per
day. This means that (X | N ) ∼ Bin(N, p). Similarly, if Y is the number of bot requests per day, then
(Y | N ) ∼ Bin(N, 1 − p). These can be made into a joint distribution:
P (x = i, Y = j) = P (X = i, Y = i | X + Y = i + j)P (X + Y = i + j) + P (X = i, Y = i | X + Y 6= i + j)P (X + Y 6= i + j)
= P (X = i, Y = i | X + Y = i + j)P (X + Y = i + j).
Then, P (X = i, Y = j | X + Y = i + j) = i+j

i j
i p (1 − p) since it’s multinomial, and P (X + Y = ij ) =
−λ i+j
e λ /(i + j)!, so the overall probability is
λi+j
 
i+j i
P (X = i, Y = j) = p (1 − p)j e−λ
i (1 + j)!
i j
(λp) (λ(1 − p))
= e−λ
i! j!
i
(λp) −λ(1−p) (λ(1 − p))j
= e−λp e = P (X = i)P (Y = j),
i! j!
where X ∼ Poi(λp) and Y ∼ Poi(λ(1 − p)).18 Thus, X and Y are in fact independent, which makes sense. (
Definition 12.5. Two continuous random variables X and Y are independent if P (X ≤ a, Y ≤ b) = P (X ≤
a)P (Y ≤ b) for any a and b.
This is equivalent to any of the following:
• FX,Y (a, b) = FX (a)FY (b) for all a, b.
• fX,Y (a, b) = fX (a)fY (b) for all a, b.
• More generally, fX,Y (x, y) = h(x)g(y) with x, y ∈ R, which has nice consequences for integration. Note
that the constraints have to factor as well (see below).
Example 12.6.
• If fX,Y (x, y) = 6e−3x e−2y , then X and Y are pretty clearly independent: let h(x) = 3e−3x and
g(y) = 2e−2y .
• If fX,Y (x, y) = 4xy for 0 < x, y < 1, then X and Y are still independent, as h(x) = 2x and g(y) = 2y.
• With fX,Y as above, with the additional constraint that 0 < x+y < 1. Here, X and Y aren’t independent,
both intuitively and since the constraints can’t factor independently. (
Example 12.7. Suppose two people set up a meeting at noon, and each arrives at a time uniformly distributed
between noon and 12:30 p.m. Let X be the minutes past noon that the first person arrives, and Y be the minutes
past noon person 2 arrives. Then, X, Y ∼ Uni(0, 30). What is the probability that the first person to arrive waits
more than ten minutes for the other?
This problem is symmetric in X and Y , so P (X + 10 < Y ) + P (Y + 10 < X) = 2P (X + 10 < Y ). Then,
ZZ ZZ
2P (X + 10 < Y ) = 2 f (x, y) dx dy = 2 fX (x)fY (y) dx dy
x+10<y x+10<y
Z 30Z y−10  2 Z 30Z y−10 Z 30
1 2 2 4
=2 dx dy = 2 dx dy = 2 (y − 10) dy =
10 0 30 30 10 0 30 10 9

18For some reason, this was a relevant xkcd today.

29
The hardest part of this was probably finding the boundary conditions; the integration is straightforward, and
some of the steps were skipped. (

Example 12.8. Suppose a disk surface is a disc of radius R and a single point of imperfection is uniformly
distributed on the disk. Then,
 1 2 2 2
fX,Y (x, y) = πR2 , x + y ≤ R
0, x + y > R2
2 2

Notice that X and Y , which are the random variables that are the coordinates of that point, are not independent:
if X = 1 = 0. More formally,
√
∞ Z2 −x2
R √
2 R 2 − x2
Z Z
1
fX (x) = fX,Y (x, y) dx dy = 2
dy = dy = .
−∞ πR √
πR2
x2 +y 2 ≤R2 − R2 −x2
p
By symmetry, fY (y) = 2 R2 − y 2 /πR2 , where −R ≤ y ≤ R. Thus, fX,Y (x, y) 6= fX (x)fY (y).
Often, the distance of this imperfection is√important: more important data tends to be placed on the outside
so that it can be accessed faster. If D = X 2 + Y 2 , then P (D ≤ A) = πa2 /πR2 = a2 /R2 , since it can be
modelled as a random dart landing in the circle of radius a.
Additionally, the expected value can be calculated, though it might be counterintuitive: using Lemma 11.8,
Z R Z R R
a2 a3
 
2R
E[D] = P (D > a) da = 1 − 2 da = a − 2
= . (
0 0 R 3R 0 3
Independence of n variables for n > 2 is a straightforward generalization of the n = 2 case, since it requires the
probabilities to factor through for any possible subset. This is identical to the definition for events. Indepenence
is symmetric: if X and Y are independent, then Y and X are. This is obvious, but consider a sequence
X1 , X2 , . . . of independent and identically distributed (IID)19 Then, Xn is a record value if Xn > Xi for all
i < n (i.e. Xn = max(X1 , . . . , XN )). Let Ai be the event that Xi is a record value. Then, the probability that
tomorrow is a record value seems to depend on whether today was, but flipping it around, does tomorrow being
a record value affect whether today is?
More mathematically, P (An ) = 1/n and P (An+1 ) = 1/(n + 1). Then, P (An An+1 ) = (1/n)(1/(n + 1)) =
P (An )P (An+1 ).

Example 12.9. One useful application
is choosing a random subset of size k from a set of n elements. The goal
of this is to do this such that all nk possibilities are equally likely, given a uniform() function that simulates
Uni(0, 1). There’s a brute-force way to do this by calculating all subsets and then picking a random one. This is
exponential in time and space. Here’s a better solution:
int indicator ( double p ) {
if ( random () < p ) return 1;
else return 0;
}

subset rSubset (k , set of size n ) {

subset_size = 0;
I [1] = indicator (( double ) k / n );
for ( i = 1; i < n ; i ++) {
subset_size += I [ i ];
I [ i +1] = indicator (( k - subset_size )/( n - i ));
}
return ( subset containing element [ i ] iff I [ i ] == 1);
}

19The second condition means that they all have the same distribution, not even depending on i.

30
 13. Adding Random Variables: 4/29/13
Recall Example 12.9 from the previous lecture. The general idea of the algorithm is to create an array I with
exactly k nonzero entries. Each element is added with a probability that depends on the number of elements left
in the set. Also, note that this algorithm is linear, which is a nice speedup from the previous case.
Claim. Any given subset of k elements is equally likely to be returned from this algorithm. (
Proof. Proceed by induction k + n.
In the base case, where k = 1 and n = 1, then S = {a} and rSubset returns
{a} with probability p = 1 = 1/ 11 .
In the general case, suppose the algorithm works for k + x ≤ c. Then:
• If k + n ≤ c + 1, then |S| = n. Suppose that I[1] =
1. Then, by the inductive hypothesis, rSubset returns
a subset S 0 of size k − 1 with probability 1/ n−1
k−1 (i.e. equally likely). Then, since P (I[1] = 1) = k/n,
then the total probability of any subset which contains the first element is k/n · 1/ n−1 n



k−1 = k .
• If k + n
≤ c + 1 and I[1] = 0, then a set of size k is returned out of the remainin n − 1 with probability
1/ n−1 n−1
1/ nk . 



k . Then, P (I[1] = 0) = (1 − k/n), so the overall probability is (1 − k/n)(1/ k−1 ) =

Suppose X and Y are independent random variables X ∼ Bin(n1 , p) and Y ∼ Bin(n2 , p). Then, X + Y ∼
Bin(n1 + n2 , p). Intuitively, do the trials for Y after those of X, and it’s essentially the same variable, since
both X and Y have the same chance of success per trial. This generalizes in the straightforward way: suppose
Xi ∼ Bin(ni , p) for 1 ≤ i ≤ n. Then,
n
! n
!
X X
Xi ∼ Bin ni , p .
i=1 i=1

Now take X ∼ Poi(λ1 ) and Y ∼ Poi(λ2 ), where X and Y are independent. These can be summed as well:
X + Y ∼ Poi(λ1 + λ2 ). The key idea in the proof is that P (X + Y = n) = P (X = k, Y = n − k) over all
0 ≤ k ≤ n. More generally, if Xi ∼ Poi(λi ) are all independent, then
n
! n
!
X X
Xi ∼ Poi λi .
i=1 i=1

Then, some things can be said about density. Suppose that g(X, Y ) = X + Y . Then, E[g(X, Y )] = E[X + Y ] =
E[X] + E[Y ]. (The proof will be given in the next lecture.) This can be generalized to many variables X1 , . . . , Xn :
" n # n
X X
E Xi = E[Xi ].
i=1 i=1

This holds regardless of the dependencies between the Xi . Thus, it is an incredibly useful result, and comes up
all the time in probability (including the next few problem sets).
Then, consider two independent, continuous random variables X and Y . The CDF of X + Y can be given by
Z∞ a−y
ZZ Z Z ∞
FX+Y (a) = P (X+Y ≤ a) = fX (x)fY (y) dx dy = fX (x) dxfY (y) dy = FX (a−y)fY (y) dy.
−∞
x+y≤a y=−∞ x=−∞

FX+Y is called the convolution of FX and FY as given above. This also works for PDFs:
Z ∞
fX+Y (a) = fX (a − y)fY (y) dy.
−∞

Convolutions exist in the discrete case, where integrals are replaced with sums and PDFs are replaced with
probabilities.
Suppose X ∼ Uni(0, 1) and Y ∼ Uni(0, 1) are independent random variables (i.e. f (a) = 1 for 0 ≤ a ≤ 1).
Intuitively, the uniform distribution seems easy to handle, but behaves badly when something more complicated
happens to it: here’s what happens to the PDF of X + Y :
Z 1 Z 1
fX+Y (a) = fX (a − y)fY (y) dy = fX (a − y) dy.
0 0
31
Now some casework is necessary: 0R ≤ a ≤ 2, so consider 0 ≤ a ≤ 1. Then, 0 ≤ y ≤ a, so that 0 ≤ a − y ≤ 0
a
and fX (a − y) = 1, so fX+Y (a) = 0 dy = a. However, if 1 ≤ a ≤ 2, then a − 1 ≤ y ≤ 1, but fX (a − y) = 1
R1
still, so fX+Y (a) = a−1 dy = 2 − a. Thus, the PDF is (as in Figure 6)

 a, 0≤a≤1
fX+Y (a) = 2 − a, 1 < a ≤ 2
0,

otherwise.

Another fact (whose proof is a little complicated for now) is that if X ∼ N(µ1 , σ12 ) and Y ∼ N(µ2 , σ22 ) are

Figure 6. The sum of two uniform distributions on (0, 1). Source

independent, then X + Y ∼ N(µ1 + µ2 , σ12 + σ22 ), which is well-behaved even though it seems more complicated
than the uniform distribution. This can be once again generalized: if X1 , . . . , Xn are independent random
variables, such that Xi ∼ N(µi , σi2 ), then
n n n
!
X X X
2
Xi = N µi , σi .
i=1 i=1 i=1

Example 13.1. Suppose an RCC checks several computers for viruses. There are 50 Macs, each of which is
independently infected with probabiliy p = 0.1 and 100 PCs, each independetly infected with probability p = 0.4.
Let A be the number of infected Macs and B be the number of infected PCs, so A ∼ Bin(50, 0, 1) ≈ N(5, 4.5),
and B ∼ Bin(100, 0.4) ≈ N(40, 24). Thus, P (A + B ≥ 40) ≈ P (X + Y ≥ 39.5). (Recall the continuity correction
and inclusiveness discussed previously.) Since the normal distribution is additive, then X + Y ∼ ∼(45, 28.5), so
P (X + Y ≥ 39.5) = 1 − Φ(−1.03) ≈ 0.8485. (

Conditional probabilities still work for distributions: the conditional PMF of X given Y (where pY (y) > 0)
can be given by
P (X = x, Y = y) pX,Y (x, y)
PX|Y (x | y) = P (X = x | Y = y) = = .
P (Y = y) pY (y)
This is not that different from before, but it is helpful to be clear about the difference between random variables
and events. Since these are discrete random variables, sometimes breaking these into sums is helpful.

Example 13.2. Suppose a person buys two computers over time, and let X be the event that the first computer
is a PC, and Y be the event that the second computer is a PC (as indicators, so X, Y ∈ {0, 1}). Then, various
conditional probabilities can be found using the joint PMF table, which looks not too tricky. Interestingly,
P (X = 0 | Y = 1) can be calculated, which is sort of looking back in time. This can be used to make
recommendations for future products, etc. (

Example 13.3. Suppose a web server receives requests, where X ∼ Poi(λ1 ) is the number of requests by
humans per day and Y ∼ Poi(λ2 ) is the number of requests from bots per day. Then, X and Y are independent,
32
so they sum to Poi(λ1 + λ2 ). Then, given only the total number of requests one can get the probablity that k of
them were by humans:
P (X = k, Y = n − k) P (X = k)P (Y = n − k)
P (X = k | X + Y = n) = =
P (X + Y = n) P (X + Y = n)
e−λ1 λk1 e−λ2 λn−k
2 n!
= · ·
k! (n − k)! e−(λ1 +λ2 ) (λ1 + λ2 )n
n! λk1 λ2n−k
= ·
k!(n − k)! (λ1 + λ2 )n
  k  n−k
n λ1 λ1
= .
k λ1 + λ2 λ1 + λ2
This looks suspiciously like a binomial distribution, and X | X + Y ∼ Bin(X + Y, λ1 /(λ1 + λ2 )). Intuitively, for
every request, a coin is flipped that determines the origin of each request. (
If X and Y are continuous random variables, then conditional probability looks not much different: the
conditional PDF is fX|Y (x | y) = fX,Y (x, y)/fY (y). Intuitively, this is the limit of probabilities in a small area
around x and y. Then, the CDF is
Z a
FX|Y (a | y) = P (X ≤ a | Y = y) = fX|Y (x | y) dx.
−∞

Observe that it does make sense to condition on y even though P (Y = a) = 0, because this is actually a limit
that does make sense. This would require chasing epsilons to show rigorously, however.

14. Continuous Conditional Distributions: 5/1/13

Though the definition of a conditional PDF was given in the last lecture, an example will make it make more
sense.
Example 14.1. Suppose X and Y are continuous random variables with the PDF
 12
f (x, y) = 5 x(2 − x − y), 0 < x, y < 1
0, otherwise.
Then, the conditional density is
fX,Y (x, y) fX,Y (x, y)
fX|Y (x | y) = = R1
fY (y) f (x, y) dx
0 X,Y
(12/5)x(2 − x − y) x(2 − x − y)
= R1 = 1
0
(12/5)(2 − x − y) dx [x − x3 /3 − x2 y/2]0
2

6x(2 − x − y)
= .
4 − 3y
Notice that this depends on both x and y, which makes sense. (
If X and Y are continuous random variables, then fX|Y (x | y) = fX (x), which is exactly the same as in the
discrete case.
Suppose X is a continuous random variable, but N is a discrete random variable. Then, the conditional PDF
of X given N is fX|N (x | n) = pN |X (n | x)fX (x)/pN (n). Notice the use of PMFs for the discrete variable and
PDFs for the continuous variable. What is slightly weirder is the conditional PMF of N given X, where the
discrete distribution changes based on the continuous one: pN |X (n | x) = fX|N (x | n)pN (n)/fX (x). Bayes’
theorem happens to apply here as well.
One good example of this is a coin flip in which the probability of the coin coming up heads is unknown:
it’s a continuous random variable between 0 and 1. Then, the number of coins coming up heads is a discrete
quantity that depends on a continuous one.
33
Definition 14.2. X is a beta random variable, denoted X ∼ Beta(a, b), if its PDF is given by
( a−1
x (1−x)b−1
, 0<x<1
f (x) = B(a,b)
0, otherwise,
R1
where B(a, b) = 0
xa−1 (1 − x)b−1 dx is the normalizing constant.

This distribution can do lots of unusual things, as in Figure 7, though when a = b it is symmetric about
x = 0.5.

Figure 7. Several different examples of the beta distribution. Source

Then, E[X] = a/(a + b) and Var(X) = ab/((a + b)2 (a + b + 1)).

This distribution is used as in the following example: suppose a coin is flipped m + n times and comes up
with n heads. Let X be the probability that the coin comes up heads (which is unknown), so that X ∼ Uni(0, 1)
and fX (x) = 1. If X is known, then different coin flips are independent, but this isn’t the case yet. Let N be the
number of heads, so that N | X ∼ Bin(n + m, x). Then, the conditional density of X given that N = n is
n+m n


P (N = n | X = x)fX (x) n x (1 − x)m xn (1 − x)m
fX|N (x | n) = = = R1 .
P (N = n) P (N = n) xn (1 − x)m dx
0

The value on the bottom is obtained by a sneaky trick: there needs to be a normalizing constant, and P (N = n)
isn’t known, so they have to match up. But this holds only when 0 < x < 1, so X | (N = n, n + m trials) ∼
Beta(n + 1, m + 1). In some sense, the more heads one obtains, the more the beta distribution tends towards
1, and the more tails, the more it skews to zero. As more and more trials are conducted, the beta distribution
narrows in on the likely probability.
Another nice fact is that Beta(1, 1) = Uni(0, 1) (verified by just calculating it out). It is also a conjugate
distribution for itself (as well as the Bernoulli and binomial distributions) because the prior distribution (before
a trial) of a trial given by the beta distribution makes calculating the posterior distribution (after the trial)
easy (they’re both beta distributions). In fact, one can set X ∼ Beta(a, b) as a guess as a prior to test whether
a coin has a certain bias. This is called a subjective probability. After n + m trials, the distribution can be
updated: X | (n heads) ∼ Beta(a + n, b + m). In some sense, the equivalent sample size, or the initial guess, is
overwritten in the long run. It represents the number of trials (specifically, a + b − 2 trials) imagined, of which
a − 1 ended up heads.
34
 Returning to expectation, we know that if a ≤ x ≤ b, then a ≤ E[X] ≤ b, which is fairly easy to see formally.
However, expectation can be generalized: if g(X, Y ) is a real-valued function, then in the discrete case
XX
E[g(X, Y )] = g(x, y)pX,Y (x, y),
x y

since X and Y might not be independent (so their joint distribution is necessary). In the continuous case,
Z ∞Z ∞
E[g(X, Y )] = g(x, y)fX,Y (x, y) dx dy.
−∞ −∞
For example, if g(X, Y ) = X + Y , then
Z ∞Z ∞
E[X + Y ] = (x + y)fX,Y (x, y) dx dy
−∞ −∞
Z ∞Z ∞ Z ∞Z ∞
= xfX,Y (x, y) dy dx + yfX,Y (x, y) dx dy
−∞ −∞ −∞ −∞
Z ∞ Z ∞
xfX (x) dx + yfY (y) dy = E[X] + E[Y ],
−∞ −∞
without any assumptions about X and Y (especially as to their independence).
Here’s another reasonably unsurprising fact: if P (a ≤ x) = 1, then a ≤ E[X]. This is most often useful
when x is nonnegative. However, the converse is untrue: if E[X] ≥ a, then it isn’t necessarily true that X ≥ a
(e.g. if X is equally likely to take on −1 or 3, so that E[X] = 1). Similarly. if X ≥ Y , then X − Y ≥ 0, so
E[X − Y ] = E[X] − E[Y ] ≥ 0, so E[Y ] ≥ E[Y ]. Again, the converse is untrue.
Definition 14.3. Let X1 , . . . , Xn be independently and identically distributed random variables. If F is some
PnE[Xi ] = µ, then a sequence of Xi is called a sample from the distribution F . Then, the
distribution function and
sample mean is X = i=1 Xi /n.
The idea is that X1 , . . . , Xn are chosen from a distribution (students taking a test), and the mean is just
that of this sample. The sample mean can vary depending on how the sample ends up. In some sense, E[X]
represents the average score of a class on a test, the average height on a test, etc. X is a random variable. Then,
" n # n n
X Xi 1X 1X nµ
E[X] = E = E[Xi ] = µ= = µ.
i=1
n n i=1 n i=1 n

Thus, X is an “unbiased” estimate of µ, since E[X] = µ.

Let E1 , . . . , En be events with indicator
P random variables Xi (i.e. Xi = 1 if Ei happens, and is zero otherwise).
Then, E[Xi ] = P P(E i ). Let
P X = X i
P and Y = 1 if X ≥ 1 andSn0 otherwise. Then, X ≥ Y , so E[X] ≥ E[Y ],
and E[X] = E [ Xi ] = E[Xi ] = P (Ei ), but E[Y ] = P ( i=1 Ei ), so
n n
!
X [
P (Ei ) ≥ P Ei .
i=1 i=1
20
This is known as Boole’s inequality.
If Y ∼ NegBin(r, P ), let Xi be the number of trials needed to get success after the (i − 1)st success, so
Xi = Geo(p) and E[Y ] = r/p.

15. More Expectation: 5/3/13

The first revelation of today: apparently I look like Dr. Sahami. I don’t see it, I guess.
Example 15.1. Consider a hash table with n buckets, and each string is equally likely to be hashed into
each bucket.21 Let X be the number of strings to hash until each bucket has at least one string. Finding
E[X] is a somewhat complicated problem, so break it into simpler ones. Let Xi be the number of trials to get
success after the ith success, where “success” means hashing a string into a previously empty bucket. After
i buckets are nonempty, the probability of hashing a string into an empty bucket is p = (n − i)/n. Thus,
20This is the same Boole who invented the notion of Boolean logic.
21This is an example of a coupon collecting model, though in CS applications it tends not to be called that.

35
 Pn−1
P (Xi = k) = ((n − i)/n)(i/k)k−1 , so Xi ∼ Geo((n − i)/n), so E[Xi ] = 1/p = n/(n − i). Then, X = i=0 Xi ,
so
n−1 n−1
X n  
X 1 1
E[X] = E[Xi ] = =n + + · · · + 1 = O(n log n). (
i=0 i=0
n−i n n−1

Example 15.2. Recall Quicksort, a fast, recursive sorting algorithm. Choosing the index is based on a partition
function:

int Partition ( int [] arr , int n ) {

int lh = 1 , rh = n -1;
int pivot = arr [0];
while ( true ) {
while ( lh < rh && arr [ rh ] >= pivot ) rh - -;
while ( lh < rh && arr [ lh ] < pivot ) lh ++;
if ( lh == rh ) break ;
Swap ( arr [ lh ] , arr [ rh ]);
}
if ( arr [ lh ] >= pivot ) return 0;
Swap ( arr [0] , arr [ lh ]);
return lh ;
}

The complexity of this algorithm is determined by the number of comparisons made to the pivot. Though Quicksort
is O(n log n), in the worst case it is O(n2 ), in which every time the pivot is selected, it is the maximal or
minimal remaining element. Then, the probability that Quicksort has the worst-case behavior is the same as the
probability of creating a degenerate BST (as seen on the first problem set), since there are exactly two bad
pivots on each recursive call. Thus, the probability of the worst case is 2n−1 /n!.
This isn’t enough to give a nice description of the running time; the goal is to instead find the expected
running time of the algorithm. Let X be the number of comparisons made when sorting n elements, so that E[X]
is the expected running time of the algorithm.
Let X1 , . . . , Xn be the input to be sorted, and let Y1 , . . . , Yn be the output in sorted order. Let Ia,b = 1 if Ya
and Yb are compared and 0 otherwise. Then, because of the order, each pair of values is only compared once:
Pn−1 Pn
X = a=1 b=a+1 Ia,b .
For any given Ya and Yb , if the pivot chosen is not between them, they aren’t directly compared in that
recursive call. Thus, the only cases we care about are those in which the pivot is in {Ya , . . . , Yb }. Then, in
order to compare then, one of them must be selected as the pivot, so the probability that they are compared is
2/(b − a + 1). This has the interesting consequence that randomly permuting an array can make Quicksort run
faster.
Then, one can make some explicit calculations: first, an approximation is made:

n Z n
X 2 2 db n
≈ = 2 ln(n − a + 1)|a+1 ≈ 2 ln(n − a + 1)
b−a+1 a+1 b−a+1
b=a+1

when n is large. Thus, let y = n − a + 1.

n−1
X
E[X] ≈ 2 ln(n − a + 1)
a=1
Z n−1
≈2 ln(n − a + 1) da
1
Z 2
= −2 ln y dy ≈ 2n ln(n) − 2n
n
= O(n log n). (
36
 Let Ii be indicator variables for events Ai . Then, we can speak of pairwise indicators: the variable Ii Ij is the
indicator for the event Ai ∩ Aj . Thus, one has
 
 
X X X X
E =E Ij Ij  = E[Ii Ij ] = P (Ai ∩ Aj ).
2 i<j i<j i<j

This can also be derived by expanding X

= x(x − 1)/2. Then, E[X 2 ] − E[X] = 2 i<j P (Ai ∩ Aj ), so
P
2
E[X 2 ] = 2 i<j P (Ai ∩ Aj ) − E[X]. Thus, one has the formula for variance, which can be used to obtain the
P
variance of the binomial distribution another way.
Example 15.3. Consider a computer cluster with k servers. Requests independently go to each server i with
probability pi , and then, one can calculate how many machines are being useful: let Ai be the probability that
server i receives no requests. Let X be the number of events Ai that occur, and Y = k − X be the number of
machines that do some work. Since the requests are indepdent, P (Ai ) = (1 − pi )n , so
k
X k
X
E[Y ] = k − E[X] = k − P (Ai ) = (1 − pi )n ,
i=1 i=1

But knowing the variance is also helpful in terms of this real-world model. Since the events Xi are independent,
then P (Ai ∩ Aj ) = (1 − pi − pj )n for i 6= j, so
X X
E[X(X − 1)] = E[X 2 ] − E[X] = 2 P (Ai ∩ Aj ) = 2 (1 − pi − pj )n ,
i<j i<j

so the variance is
X
Var(X) = 2 (1 − pi − pj )n + E[X] − (E[X])2
i<j

k k
!2
X X X
=2 (1 − pi − pj )n + (1 − pi )n − (1 − pi )n = Var(Y ).
i<j i=1 i=1

Places such as Amazon take their excess capacity and sell it to other people who might need servers, and
knowing exactly how much they can sell is a valuable piece of information. (
This example sounds exactly like coupon collecting or using hash tables, but is ostensibly more interesting.
One can take products of expectations: suppose X and Y are independent random variables and g and h
are real-valued functions. Then, E[g(X)h(Y )] = E[g(X)]E[h(Y )]. If X and Y are dependent, this rule doesn’t
hold, but the sum rule still does. Here’s why the product rule works in the independent case:
Z ∞Z ∞
E[g(X)h(Y )] = g(x)h(y)fX,Y (x, y) dx dy
−∞ −∞
Z ∞Z ∞
= g(x)h(y)fX (x)fY (y) dx dy
−∞ −∞
Z ∞ Z ∞
= g(x)fX (x) dx h(y)fY (y) dy
−∞ −∞
= E[g(X)]E[h(Y )].
Independence was necessary so that the joint PDF factored into the marginal density functions.
Definition 15.4. If X and Y are random variables, the covariance of X and Y is Cov(X, Y ) = E[(X −E[X])(Y −
E[Y ])].
Equivalently, Cov(X, Y ) = E[XY − E[X]Y − XE[Y ] + E[Y ]E[X]] = E[XY ] − E[X]E[Y ] − E[X]E[Y ] +
E[X]E[Y ] = E[XY ] − E[X]E[Y ]. This latter formula (Cov(X, Y ) = E[XY ] − E[X]E[Y ]) is particularly
helpful, and indicates how X and Y vary together linearly. Clearly, if X and Y are independent, then
Cov(X, Y ) = 0. However, the converse is not true, and this is a frequently misunderstood aspect of probability:
if X ∈ {−1, 0, 1} with equal likelihood and Y is an indicator for Y = 0, then E[XY ] = 0, since XY = 0. Thus,
Cov(X, Y ) = E[XY ] − E[X]E[Y ] = 0, but X and Y aren’t independent.
37
Example 15.5. Imagine rolling a six-sided die and let X be an indicator for returning a 1, 2, 3, or 4, and let
Y be an indicator for 3, 4, 5, or 6. Then, E[X] = E[Y ] = 2/3 and E[XY ] = 1/3. Thus, the covariance is
Cov(X, Y ) = −1/9, which is negative. This is surprising: the variance can’t be negative, but this is, indicating
that X and Y vary in opposite directions. (

16. Covariance: 5/6/13

“It’s not ‘second hat,’ it’s ‘old hat.’ It’s a mixed metaphor, like ‘that’s the way the cookie bounces.’ ”
Another example of covariance is a table of weights and heights. In a sample mean of several people’s weights
and heights, there is a significant covariance between them.
Here are some properties of covariance:
• Cov(X, Y ) = Cov(Y, X). This is not hard to see from the definition.
• Cov(X, X) = E[X 2 ] − E[X]E[X] = Var(X).
• Cov(aX + bY ) = a Cov(X, Y ). Notice that this is different than the variance, where Var(aX) =
a2 Var(X).
• If X1 , . . . , Xn and Y1 , . . . , Ym are all random variables, then
n m
! n Xm
X X X
Cov Xi , Yi = Cov(Xi , Yj ).
i=1 i=1 i=1 j=1

This allows us to understand the variance of a sum of variables: it isn’t just the sum of the variances.
Claim. If X1 , . . . , Xn are random variables, then
n
! n n n
X X X X
Var Xi = Var(Xi ) + 2 Cov(Xi , Xj ). (
i=1 i=1 i=1 j=i+1

Proof. Because Cov(X, X) = Var(X), then

!  
Xn Xn n
X
Var Xi = Cov Xi , Xj 
i=1 i=1 j=1
n X
X n
= Cov(Xi , Xj )
i=1 j=1
Xn n X
X n
= Var(Xi ) + Cov(Xi , Xj )
i=1 i=1 j=1
j6=i
n
X n
X X n
= Var(Xi ) + 2 Cov(Xi , Xj ).
i=1 i=1 j=i+1

The last step was done because Cov(Xi , Xj ) = Cov(Xj , Xi ), so the sum can be simplified. 

Note that this formula Pnif Xi and Xj are independent when i 6= j (since Xi can’t be independent
Pnmeans that
with itself), then Var( i=1 Xi ) = i=1 Var(Xi ). This is an important difference between the expected value
and the variance.
Example 16.1. Let Y ∼ Bin(n, p) and let Xi be an indicator for the ith trial being successful. Specifically,
Xi ∼ Ber(p) and E[Xi ] = p. Notice that E[X 2 ] = E[X], since X is only 0 or 1, each of which squares to itself.
Then,
n
X n
X n
X n
X
Var(Y ) = Var(Xi ) = (E[Xi2 ] − E[Xi ]2 ) = (E[Xi ] − E[Xi ]2 ) = (p − p2 ) = np(1 − p).
I=1 i=1 i=1 i=1

This is a fact that you should have already seen before, but this is an interesting derivation. (
38
 Pn
Recall the sample mean X = i=1 Xi /n for independently and identically distributed random variables
X1 , . . . , Xn . If E[Xi ] = µ and Var(Xi ) = σ 2 , then E[X] = µ, but the variance is slightly more interesting:
n
!   n
!
2
X Xi 1 X
Var(X) = Var = Var Xi
i=1
n n i=1
 2 X n
1
= Var(Xi )
n i=1
 2 X n
1 σ2
= σ2 = .
n i=1
n
The variance gets smaller as there are more variables in the mean. This makes sense: as more things (e.g.
people’s heights) are added to the sample, then they on average look like the mean, so the variance decreases.
One can also define the sample deviation X − Xi for each i, and then obtain the sample variance, which is
the guess at what the standard deviation should be:
n
X (Xi − X)2
S2 = .
i=1
n−1
Then, E[S 2 ] = σ 2 . This means that S 2 is an “unbiased estimate” of σ 2 , which is a bit of jargon. The n − 1 in
the denominator comes out after an intimidating computation to show that E[S 2 ] = σ 2 .
Definition 16.2. If X and Y are two arbitrary random variables, the correlation of X and Y is
Cov(X, Y )
ρ(X, Y ) = p .
Var(X) Var(Y )
In some sense, this takes the covariance, which already illustrates something similar, and divides out to
account for possibly different units. The correlation is always within the range [−1, 1]. This measures how much
X and Y vary in a linear relationship: if ρ(X, Y ) = 1, then Y = aX + b, where a = σy /σX (the standard
deviations of Y and X, respectively). If ρ(X, Y ) = −1, then Y = aX + b, but a = −σy /σX . If ρ(X, Y ) = 0,
then there is no linear relationship between the two. Note that there may still be a relationship, just of a higher
order. However, the linear component is typically the one that people care the most about.
If ρ(X, Y ) = 0, then X and Y are uncorrelated; otherwise, they are correlated. Notice that independence
implies two variables are uncorrelated (since the covariance is zero), but the converse is false!
Suppose IA and IB are indicator variables for events A and B. Then, E[IA ] = P (A) and E[IB ] = P (B), so
E[IA IB ] = P (A ∩ B). Thus,
Cov(IA , IB ) = E[IA IB ] − E[IA ]E[IB ] = P (A ∩ B) − P (A)P (B)
= P (A | B)P (B) − P (A)P (B)
= P (B)(P (A | B) − P (A)).
Thus, if P (A | B) > P (A), then the correlation is positive; when A happens, then B is more likely. Similarly, if
P (A | B) = P (A), then ρ(IA , IB ) = 0, and if P (A | B) < P (B), then observing A makes B less likely, so they
are negatively correlated.
Example 16.3. Suppose n independent trials of some experiment are performed, and each trial results in one
of m outcomes, each with probabilities p1 , . . . , pm , where the pi sum to 1. Let Xi be the number of trials with
outcome i. For example, one could count the number of times a given number comes up on a die when rolled
repeatedly. Intuitively, two numbers should be negatively correlated, since more of one number implies less
room, so to speak, for another. Pn
Let Ii (k) be an indicator variable that trial k has outcome i. Then, E[Ii (k)] = pi , Xi = k=1 Ii (k). If a 6= b,
then trials a and b are independent, so Cov(Ii (b), Ij (a)) = 0, and when a = b, then E[Ii (a)Ij (a)] = 0 (you can’t
roll both a 1 and a 2 in the same trial), so Cov(Ii (a), Ij (a)) = −E[Ii (a)]E[Ij (a)], since the first term drops out.
Thus,
Xn X n Xn n
X X n
Cov(Xi , Xj ) = Cov(Ii (b), Ij (a)) = Cov(Ii (a), Ij (a)) = −E[Ii (a)]E[Ij (a)] = −pi pj = −npi pj ,
a=1 a=1 a=1 a=1 i=1
39
so they are in fact negatively correlated. This is important because multinomial distributions happen in many
cases in applications. For example, if m is large, so that pi is small, and the outcomes are equally likely (such
that pi = 1/m), then Cov(Xi , Xj ) = −n/m2 , so the covariance decreases quadratically as m increases. This is
why the Poisson paradigm works so well: this quantifies how accurate the approximation can be. (
Suppose that X and Y are jointly discrete random variables, so their conditional PMF is pX|Y (x | y) =
P (X = x | Y = y) = pX,Y (X, y)/pY (y). Then, the conditional expectation of X given that Y = y is
X X
E[X | Y = y] = xP (X = x | Y = y) = pX|Y (x | y),
x x

and in the continuous case we have the analogous result:

Z∞
E[X | Y = y] = xfX|Y (x | y) dx.
−∞
P
Example 16.4. Roll two six-sided dice D1 and D2 . Then E[X | Y = 6] = x xP (X = x | Y = 6) =
(1/6)(7 + · · · + 12) = 9.5. This makes sense, as it is also E[X] + Y . (
Example 16.5. If X and Y are independent random variables, such that X, Y ∼ Bin(n, p), to compute E[X |
Y = m] it will be necessary to compute P (X = k | X + Y = m):
P (X = k, X + Y = m) P (X = k, Y = m − k) P (X = k)P (Y = m − k)
P (X = k | X + Y = m) = = =
P (X + Y = m) P (X + Y = m) P (X + Y = m)
n k n n

n−k

m−k n−(m−k)

n

p (1 − p) · p (1 − p)
= k
m−k
2n m 2n−m
= k 2nm−k

.
m p (1 − p) m

This is a hypergeometric distribution: (X | X + Y = m) ∼ HypG(m, 2n, n). This actually has an intuitive
explanation: in a total of m draws, there are 2n total balls in an urn, of which n are white. This is exactly what
the definition specifies. Thus, the expectation is already known: E[X | X + Y = m] = nm/(2n) = m/2. Of
course, by symmetry, this makes sense, and you could even write it down ahead of time with some foresight. (
The sum of expectations still holds, no matter what you throw at it: if X1 , . . . , Xn and Y are random variables,
then " n #
X Xn
E Xi | Y = y = E[Xi | Y = y].
i=1 i=1

Since E[E | Y = y] sums over all values of X, then it can be thought of as a function of Y but not X, so
g(Y ) = E[X | Y ] is a random variable. For any given Y = y, g(Y ) = E[X | Y = y]. Thus, we can take
E[g(Y )] = E[E[X | Y ]]. Intuitively, this is the expectation of an expectation, which simplifies easily, but these
are expectations of different variables, so they do not cancel out. The notation might be confusing, so add
subscripts corresponding to the expected variables if necessary. However, in this case the expectation can be
taken one layer at a time:
X
E[E[X | Y ]] = E[X | Y = y]P (Y = y)
y
!
X X
= xP (X = x | Y = y) P (Y = y)
y x
X X X
= xP (X = x, Y = y) = x P (X = x, Y = y)
x,y x y
X
= xP (X = x) = E[X].
x

By replacing the sums with integrals, the same result can be found for the continuous case.
Example 16.6. Consider the following code, which recurses probabilistically:
40
int Recurse () {
int x = randomInt (1 ,3); // equally likely values
if ( x == 1) return 3;
else if ( x == 2) return (5 + Recurse ());
else return (7 + Recurse ());
}
Let Y be the value returned by Recurse(). What is E[Y ]? This may seem like a weird example, but it could
indicate the running time of a snippet of code given some different recursive calls.
Notice that E[Y | X = 1] = 3, E[Y | X = 2] = 5 + E[Y ], and E[Y | X = 3] = 7 + E[Y ], where X is
the value of x in the program. Thus, E[Y ] = 3(1/3) + (5 + E[Y ])/3 + (7 + E[Y ])/3, which can be solved for
E[Y ] = 15. (

17. Predictions: 5/8/13

“Once the blue lightning comes out of [my daughter’s] fingers, it’s all over.”
This example will exhibit a random number of random variables.
Example 17.1. Consider a website www.PimentoLoaf.com. Let Xi ∼ N(50, 25) be the number of people who
visit per day, and let Yi ∼ Poi(8) be the number of minutes spent by visitor i. X and the Yi are all independent,
so what is the expected number of minutes W that people spend on the site per day?
"X # " "X ##
X X
E[W ] = E Yi = E E Yi | X .
i=1 i=1
If it is known that X = n, then this becomes
" n # n n
X X X
E[W ] = E Yi | X = n = E[Yi | X = n] = E[Yi ] = nE[Yi ]
i=1 i=1 i=1
because the Yi are identically distributed and independent of X, and thus the overall expectation becomes
XE[Yi ]. (
An important application of probability is to make predictions about some random variable Y given some other
random variable X. This has lots of applications, such as high-frequency trading or prediction of genetic-based
diseases. Let g(X) be the function used to predict Y : the predicted value is Ŷ = g(X). g(X) is chosen so as
to minimize E[(Y − g(X))2 ], which ends up being g(X) = E[Y | X], which isn’t actually a surprise. Intuitively,
E[(Y − c)2 ] is minimized when c = E[Y ], and if X is observed, then let c = E[Y | X]. Under some relatively
mild assumptions, it’s not actually possible to do better. This is the beginning of machine learning.
Example 17.2. Suppose a father has height X = 71 inches, and historically, a son’s height is Y ∼ N(X + 1, 4)
(i.e. Y = (X + 1) + C, where C ∼ N(4)). Then, the predicted value for the height of the son is E[Y | X =
71] = E[X + 1 + C | X = 71] = E[72 + C] + E[72] + E[C] = 72. (
Thus, one can compute probabilities by conditioning: if X is an indicator variable for the event A, then
E[X] = P (A) so E[X | Y = y] = P (A | Y = Py) for any Y . Thus, E[X] = EY [EX [X | Y ]] = EY [P (A | Y )].
In the discrete case, this becomes E[X] = y P (A | Y = y)P (Y = y) = P (A) by marginalizing.22 More
generally, if Fi is the indicator variable for Y = yi , then
Xn
P (A) = P (A | Fi )P (Fi ),
i=1
which is called the law of total probability.
Example 17.3. Suppose n candidates for a job are interviewed for one position, and there is a factor of α (for
some α) difference in productivity between the best and average candidate.23 Imagine in this scenario a hiring
decision has to be made immediately after the interview, which gives a ranking of the candidates that have
already been interviewed.
22This also holds in the continuous case.
23For software engineers, Steve Jobs claimed α ≈ 25, but Mark Zuckerberg claims that α ≈ 100.

41
 One strategy is to first interview k candidates, and then hire the next candidate better than all of the first k
candidates. Then, it’s possible to calculate the probability that the best person is hired. Let X be the position
of the best candidate in the line of interviews. Then, Pk (Best | X = i) = 0 if i ≤ k. This is a good argument
for making k small. In general, the best person of the first k sets the bar for hiring, so the best candidate
in position i will be selected if the best of the first i − 1 is in the first k interviewed. Thus, if i > k, then
Pk (Best | X = i) = k/(i − 1). Then,
n n
k n
Z
1X 1 X k di k n
Pk (Best) = Pk (Best | X = i) = ≈ = ln .
n i=1 n i=1 n k+1 i − 1 n k
i=k+1

Then, to optimize the value of k, differentiate this and set it equal to zero. This yields k = n/e, for which the
probability of hiring the best person is about 1/e ≈ 37%. This is interesting because it doesn’t depend on the
number of people interviewed.
Nonetheless, this isn’t optimal, which is why most companies don’t use this strategy. (
 tX 
Definition 17.4. The moment-generating function (MGF) of a random variable X is M (t) = E e where
t ∈ R.
X is discrete, this is just M (t) = x etx p(x), and when X is continuous, this is instead M (t) =
P
R ∞When
−∞
etx fX (x) dx. Interesting things happen when you calculate M 0 (0): derivatives commute with taking the
expected value, so M 0 (X) = E[XetX ], so M 0 (0) = E[X]. Taking the derivative again yields M 00 (t) = E[X 2 etX ],
so M 00 (0) = E[X 2 ], or the second moment. This generalizes: M (n) (0) = E[X n ].
Example 17.5. If X ∼ Ber(p), then M (t) = E[etX ] = e0 (1 − p) + et p = et p − p, so M 0 (0) = p and M 00 (t) = et p
again, so M 00 (0) = p again. This is stuff we already knew, but it’s nice to see it derived in a new way. (
Example 17.6. Now let X ∼ Bin(n, p). Then,
n   n  
 X n k X n
M (t) = E etX = etk p (1 − p)n−k = (pet )k (1 − p)n−k = (pet + 1 − p)n

k k
k=0 k=0

using the Binomial Theorem. Thus, M (t) = n(pe + 1 − p)n−1 pet , so M 0 (0) = E[X] = np.24 Then, in a similar
0 t

process, one can show that M 00 (0) = n(n − 1)p2 + np, so Var(X) = np(1 − p). (
If X and Y are independent random variables, then MX+Y (t) = E[et(X+Y ) ] = E[etX ]E[etY ] = MX (t)MY (t),
and conversely, if the MGF factors, then these two variables are independent. Note that because of the exponential,
the sum becomes a product. Additionally, MX (t) = MY (t) iff X ∼ Y .
One can generalize to joint MGFs: if X1 , . . . , Xn are random variables, then their joint MGF is M (t1 , . . . , tn ) =
E[et1 X1 +···+tn Xn ]. Then, MXi (T ) = M (0, . . . , 0, t, 0, . . . , 0), where the t is in the ith position. Then, X1 , . . . , Xn
n
Y
are independent iff M (t1 , . . . , tn ) = MXi (t), and the proof is the same as in the n = 2 case.
i=1
If X ∼ Poi(λ), then
∞
X e−λ λn t
M (t) = E[e tX
]= etn = eλ(e −1) ,
n=0
n!
so after a bunch of somewhat ugly math, M 0 (0) = λ and so on, as we already saw, but in a new way. This
is significant because if X ∼ Poi(λ1 ) and Y ∼ Poi(λ2 ) are independent, this offers a simple proof that
X + Y ∼ Poi(λ1 + λ2 ).

18. Moment-Generating Functions: 5/10/13

“In personality, consider an equilateral triangle with Sheen, his character, and myself at the
three vertices.” - Herman Chernoff
Recall the definition of moment-generating functions from the previous lecture, which allowed one to obtain the
nth moment from the nth derivative of the function. The MGF of a normal distribution X ∼ N(µ1 , σ12 ) is
 2 t2

σ1
+µ1 t
MX (t) = E etX = e
2
 
.
24This is. . . the seventh such derivation for this?

42
 0 0 00
Then, the first derivative is MX (t) = (µ1 +tσ12 )M (t), so MX (0) = µ1 , and after a bit more math, MX (0) = µ21 +σ12 .
2
This makes sense. As for why it’s actually important, suppose Y ∼ N(µ2 , σ2 ) is independent of X.25 Then, Y
has the same MGF with indices changed, but one can calculate the convolution:
 2 t2
  2 +σ 2 )t2

σ1 2 t2 (σ1 2
+µ1 t σ2 +(µ1 +µ2 )t
2 +µ2 t 2
MX (t)MY (t) = e e 2 =e = MX+Y (t),
so the normal distribution is additive: X + Y ∼ N(µ1 + µ2 , σ12 + σ22 ).
This has other meanings beyond the scope
of this class, such as uses in Fourier analysis.
Here’s another question: if V = X + Y and W = X − Y , where X, Y ∼ N(µ, σ 2 ) are independent, are V
and W independent? Take the joint MGF of V and W :
h i h i
M (t1 , t2 ) = E et1 V et2 W = E et1 (X+Y ) et1 (X−Y ) = E e(t1 +t2 )X e(t1 −t2 )Y
 
h i h i
= E e(t1 +t2 )X E e(t1 −t2 )Y because X and Y are independent.
σ 2 (t1 +t2 )2 σ 2 (t1 −t2 )2
= eµ(t1 +t2 )+ eµ(t1 −t2 )+
2 2

2 2 2 2
= e2µt1 +2σ t1 /2 e2σ t2 /2 = E et1 V E et2 B ,
   

where the last step can be seen by writing down the distributions for V and W . Notice that this result doesn’t
hold when X and Y aren’t identically distributed, since the like terms can’t be gathered.
In many cases, the true form of a probability distribution encountered in the real world isn’t obvious. For
example, this class’ midterm didn’t appear to be normally distributed. However, certain information (e.g. mean,
sample variance) can be computed anyways, and certain properties can be seen by looking at the application of
the problem (here, midterm scores are known to be nonnegative).
Proposition 18.1 (Markov’s26 Inequality). Suppose X is a nonnegative random variable. Then, for any a > 0,
P (X ≥ a) ≤ E[X]/a.
Proof. Let I be an indicator for the event that X ≥ a. Since X ≥ 0, then I ≤ X/a: if X < a, then I = 0 ≤ X/a,
and if X ≥ a, then X/a ≥ 1 = I. Then, take expectations: E[I] = P (X ≥ a) ≤ E[X/a] = E[X]/a, since
constants factor out of expectation. 

Applying this to the midterm, let X be a score on the midterm. Then, the sample mean is X = 95.4 ≈ E[X],
so P (X ≥ 110) ≤ E[X]/110 = 95.4/110 ≈ 0.8673. This says that at most 86.73% of the class got greater than
a 110. Since 27.83% of the class did this, it’s clear that Markov’s inequality is a very loose bound, but that
makes sense because the only thing it knows is the mean.
Proposition 18.2 (Chebyshev’s27 Inequality). Suppose X is a random variable such that E[X] = µ and
Var(X) = σ 2 . Then, P (|X − µ| ≥ k) ≤ σ 2 /k 2 for all k > 0.
Proof. Since (X − µ)2 is a nonnegative random variable, then apply Markov’s inequality with a = k 2 , so
P ((X − µ2 ) ≥ k 2 ) ≤ E[(X − µ)2 ]/k 2 = σ 2 /k 2 . However, (X − µ2 ) ≥ k 2 iff |X − µ| ≥ k, so the inequality
follows. 

This is what is known as a concentration inequality, since it puts bounds on how spread the data can be
from the mean. Using this for the midterm, the sample mean is X = 95.4 ≈ E[X], and the sample variance is
S 2 = 400.40 ≈ σ 2 , so P (|X − 95, 4| ≥ 22) ≤ σ 2 /222 ≈ 0.8273, so at most 82.73% of the class was more than 22
points away from the mean. This can also be used to calculate how much of the class is within a distance from
the mean: at least 17.27% of the class is within 22 points of the mean. Once again, this is very loose: about
83% of the class was within 22 points of the mean. Chebyshev’s inequality is most useful as a theoretical tool.
There’s another formulation called the one-sided Chebyshev inequality. In its simplest form, suppose that X
is a random variable with E[X] = 0 and Var(X) = σ 2 . Then, P (X ≥ a) ≤ σ 2 /(σ 2 + a2 ) for any a > 0. This

25µ not to be confused with µ .

2 2
26Andrey Andreyevich Markov was a Russian mathematician who was responsible for this inequality as well as Markov chains, which are
the basis of Google’s PageRank algorithm.
27Pafnuty Lvovich Chebyshev was also a 19th -Century Russian mathematician who didn’t actually formulate the inequality named after
him. Interestingly enough, he was Markov’s doctoral advisor and may have actually formulated Markov’s inequality before Markov.
43
generalizes: if E[Y ] = µ and Var(Y ) = σ 2 , then P (Y ≥ µ+a) ≤ σ 2 /(σ 2 +a2 ) and P (Y ≤ µ−a) ≤ σ 2 /(σ 2 +a2 )
for any a > 0. This follows directly from the first version by setting X = Y − µ.
Applying this, at most 80.02% of the class could have gotten more than a 105.4, which is a pretty loose bound
again, since 38.70% of the class did so. Nonetheless, it’s interesting that something can be said with so little,
and this is a better bound than Markov’s inequality, which reports 0.9051 as an even looser bound.
Proposition 18.3 (Chernoff28 bounds). Let M (t) be an MGF for a random variable X. Then, P (X ≥ a) ≤
e−ta M (t) for all t > 0 and P (X ≤ a) ≤ e−ta M (t) for all t < 0. This holds for all nonzero t, so one can pick
the value that minimizes the bound.
Proof. Using Markov’s inequality, P (X ≥ a) = P (etX ≥ eta ) ≤ E[etX ]/eta = e−ta E[etX ] = e−ta M (t), and the
other bound is about the same. 
2
If Z ∼ N(0, 1) (i.e. the standard normal), then its MGF is MZ (t) = et /2 , so the Chernoff bounds are
2 2
P (Z ≥ a) ≤ eta et /2 = et /2−ta for all t > 0. Using calculus, the minimum value of this function is t = a, so
2 2
P (Z ≥ a) ≤ e−a /2 for all a > 0, and P (Z ≤ a) ≤ e−a /2 for all a < 0. This is nice because it provides an
approximation despite the fact that the PDF of the normal distribution isn’t integrable.
In some sense, this reasonably tight bound exists because there’s a one-to-one correspondence between the
MGF and the distribution. However, one doesn’t always know the MGF, so this is once again a theoretical tool
in a lot of cases.
t t t
If X ∼ Poi(λ), the MGF is MX (t) = eλ(e −1) , so the Chernoff bound is P (X ≥ i) ≤ eλ(e −1) e−it = eλ(e −1)−it
for any t > 0. Thus, the goal is to minimize λ(eλ − 1) − it with respect to t, so that et = i/λ. Then, if t > 0, so
that i/λ > 1, then P (X ≥ i) ≤ et (i/λ − 1)(i/λ)i = (eλ /i)i e−λ . Notice that this matches fairly closely to the
functional form of the PMF.
Definition 18.4. A function f is convex if f 00 (x) > 0 for all x ∈ R. f is concave if −f is convex.
Intuitively, a convex function is bowl-shaped: functions such as y = x2 and y = ex are convex.
Proposition 18.5 (Jensen’s29 inequality). If f is a convex function, then E[f (X)] ≥ f (E[X]).
The proof of this is a fairly unpleasant Taylor expansion around µ. Intuitively, if f is convex, then it looks sort
of like a bowl, and the line between two points on the graph of f is always above the graph itself. The expected
value of f (x) lies on this line, and f (E[X]) lies on the graph, so the former is greater than the latter. Similarly,
if f (E[X]) = E[f (X)], then they must both lie on the same line, so f is a line, since f 00 (x) = 0.
This has applications in utility theory, a branch of finance or economics. The utility U (x) is the value that
one derives from some object x. For example, one could play a game in which one wins $20000 with a 50%
chance, or receive $10000 for not playing. Most people would decline to play in such a situation, even though
the expected values are the same, so the first and second $10000 have a different utility. Sometimes, the utility
is nonmonetary (e.g. quality of life).
The notion of a utility curve is also helpful, in which x (here representing quantities of money) is graphed
against U (x). A concave utility curve indicates a risk-averse strategy: the first $10000 has greater utility than
subsequent ones, and a flat curve indicates there is no change (risk-neutral). However, a concave curve has a
risk-preferring strategy. Then, Jensen’s inequality says that if one can invest in something that always returns µ
(some sort of bonds) or something which returns a value X such that E[X] = µ and Var(X) 6= 0 (represented by
stocks), then let R be the return value. Then, the goal is to maximize U (R). The inequality says that a convex
utility curve should invest more in stocks, and a concave utility curve should prefer bonds.

19. Laws of Large Numbers and the Central Limit Theorem: 5/13/13
Proposition 19.1 (Weak Law of Large Numbers). Suppose X1 , X2 , . . . are independently and P identically
n
distributed random variables, with a distribution F , such that E[Xi ] = µ and Var(Xi ) = σ 2 . Let X = i=1 Xi /n.
Then, for any ε > 0, limn→∞ P (|X − µ| ≥ ε) = 0.
Proof. Since E[X] = µ and Var(X) = σ 2 /n, then by Chebyshev’s inequality, P (|X − µ| ≥ ε) ≤ σ 2 /(nε2 ), so
as n → ∞, this probability goes to zero. 
28Herman Chernoff isn’t Russian, and is still a Professor Emeritus at Harvard and MIT. He did discover Chernoff’s Bound. Since he’s still
around, someone in 109 last quarter actually emailed him and asked him if he was a Charlie Sheen fan. He was.
29Johan Ludwig William Valdemar Jensen was a Dutch mathematician who was responsible for the inequality named after him.

44
Proposition 19.2 (Strong Law of Large Numbers). With X1 , X2 , . . . and X as before,
   
X1 + X2 + · · · + Xn
P lim = µ = 1.30
n→∞ n
Notice that the Strong Law implies the Weak Law, but not vice versa: the Strong Law says that for any ε > 0,
there are only a finite number of values n such that |X − µ| ≥ ε holds. The proof of the Strong Law isn’t all
that easy, and has been omitted.
Consider some set of repeated trials of an experiment, and let E be some outcome. Then, let Xi be an
indicator for trial i returning E. Then, the Strong Law implies that limn→∞ E[X] = P (E), which justifies more
rigorously the fact shown in the first week of class, that P (E) = limn→∞ n(E)/n = P (E). These laws are
mostly philosophical, but they provide a mathematical definition of probability.
One misuse of the laws of large numbers is to justify the gambler’s fallacy, in which after multiple losses
of some sort, one expects a win. This is incorrect, because the trials are independent (and the distribution is
memoryless anyways), and the averaging out happens as n → ∞, which is not nearly small enough to fit in a
lifetime.
The following result is one of the most useful in probability theory:
Theorem 19.3 (Central Limit Theorem). Let X1 , X2 , . . . be a set of independently and identically distributed
random variables with E[Xi ] = µ and Var(Xi ) = σ 2 . Then,
X1 + X2 + · · · + Xn − nµ
lim √ = N(0, 1).
n→∞ σ n
Pn

There is an equivalent formulation: if X = i=1 Xi /n, then Theorem 19.3 states that X ∼ N µ, σ 2 /n as
p
n → ∞: if Z = (X − µ)/ σ 2 /n, then Z ∼ N(0, 1), but also
Pn Pn
Xi /n − µ n( i=1 Xi /n − µ) X1 + X2 + · · · + Xn − nµ
Z = i=1 p = p = √ .
σ 2 /n n σ 2 /n σ n
This means that if one takes a large number of sample means of any distribution, they form a normal distirbution.
This is why the normal distribution appears so often in the real world, and allows things such as election
polling (a reasonable sample of people can lead to a probability that the whole group will vote one way). The
distribution can be discontinuous, skewed, or anything else horrible. And this is why this theorem is so good.
However, polling can be misused: lots of election polls end up sampling nonrepresentative populations and
wildly mis-predicting the election. Similarly, McDonalds rolled out the McRib, a flop, after successful testing in
South Carolina, which clearly represents the rest of the nation in terms of barbecue.
Example 19.4. Applying this to the midterm, there are 230 Xi , and E[Xi ] = 95.4 and Var(Xi ) = 400.40. Thus,
one can create ten disjoint samples Yi and their sample means Y i . Then, using the central limit theorem, these
should all be turned into Z ∼ N(0, 1), which ended up being really close. (
Example 19.5. Suppose one has an algorithm which has a mean running time of µ seconds and a variance of
σ 2 = 4. Then, the algorithm can be run repeatedly (which makes for independent, identically distributed trials).
Then, how many trials are needed in order to estimate µ ± 0.5 with 95% certainty? By Theorem 19.3, if Xi is
the running time of the ith run, then
Pn Pn
Xi − nµ Xi − nµ
Zn = i=1 √ = i=1 √ ∼ N(0, 1).
σ n 2 n
Thus,
 Pn   √ √ Pn √   √ √ 
i=1 Xi −0.5 n n i=1 Xi − nt 0.5 n −0.5 n 0.5 n
P −0.5 ≤ =P ≤ ≤ =P ≤ Zn ≤
n − t ≤ 0.5 2 2 n 2 2 2
√   √  √ 
n n n
=Φ −Φ − = 2Φ − 1 ≈ 0.95,
4 4 4
√
which means that (calculating Φ−1 with the table of normals) n/4 ≈ 1.96, or n = d(7.84)2 e = 62.

30Note that one still needs the probabilistic statement; things with probability zero can still happen, in some sense, but that requires a
digression into measure theory.
45
 Chebyshev’s inequality provides a worse bound: µs = µ but
n
! n
nσ 2
 
2
X Xi X Xi 4
σs = Var = Var = 2 = ,
i=1
n i=1
n n n
so   !
n
X Xi  4/n
P  − t ≥ 0.5 ≤ ,
 
 n  0.52
i=1
which implies that n ≥ 320. This is a much worse bound. (
Example 19.6. Suppose the number of visitors to a web site in a given minute is X ∼ Poi(100), and the server
crashes if there are at least 120 requests
Pn in a given minute. Then, one can use the CLT to break this into lots of
small distribiutions: Poi(100) = i=1 Poi(100/n), with all of the distributions independent, which allows one
to use a normal distribution to approximate a Poisson one. (
Example 19.7. Suppose someone rolls ten 6-sided dice, giving events X1 , . . . , X10 . Let X be the total value of all
of the dice. What is the probability that X ≤ 25 or X ≥ 45? Using the Central Limit Theorem, µ = E[Xi ] = 3.5
and σ 2 = Var(Xi ) = 35/12. Then, approximating with a normal distribution, one shows that there is about
an 0.08 probability that this will happen. Note that Chebyshev would only say that there is at most a 0.292
probability of it happening. (

20. Parameters: 5/15/13

All of the distributions specified so far are parametric models: the distribution is given in terms of parameters
that yield the actual distribution, such as Uni(α, β), where α and β are the parameters. Generically, these
parameters are denoted θ, so for a Poisson distribution Poi(λ), θ = λ, and sometimes it’s a tuple: for a uniform
dustribution, θ = (α, β).
This is important because the true parameters aren’t always known; instead, we only have experimental
evidence. Thus, one might assume the real-world data is given by a distribution, and try to find that istribution.
Definition 20.1. An estimator is a random variable that represents a parameter.
If X ∼ Ber(p), p is the truth. If we can’t see it, we might instead have a model p̂ that estimates it, but then p̂
is a random variable. The goal is to have a point estimate, which makes for a single value for the parameter
rather than a distribution. This allows a better understanding of the process, yielding data, predictions, and
simulations. P
Suppose X1 , . . . , Xn are IID random variables. Then, the sample sample mean X = Xi /n is a random
variable, so one can take the sampling distribution of the mean, which is the distribution of X. The Central Limit
Theorem shows that this is a roughly normal distribution when n is large (specifically, it’s a good approximation
when x P> 30 and is very good when x > 100). Then, we have Var(X) = σ 2 /n, where Var(Xi ) = σ 2 , and we have
n √ √
S = i=1 (Xi −X)2 /(n−1). For large n, the 100(1−α)% confidence interval is (X−zα/2 S/ n, X+zα/2 S/ n),
2
√
where Φ(zα/2 ) = 1 − α/2. Here, S/ n can be thought of as the square root of the standard variance. For
example, if α = 0.05, then Φ(zα/2 ) = 0.975, so zα/2 ≈ 1.96 with the help of a table of normals.
The meaning of a confidence interval is: with probability 1 − α when the confidence interval is computed from
a sample, the true mean µ is in the interval. Note that this is not the probability that µ itself is in the interval
for some given interval, which is either 0 or 1.
Example 20.2. Suppose a company wants to estimate the average number of hours a CPU spends idle. Then,
225 computers are monitored. Suppose X = 11.6, S 2 = 16.81 hours, and S = 4.1 hours. Thus, for a 90%
confidence interval, zα/2 = 1.645 (which is another common quantity you’ll see), so the interval is
   
S S 4.1 4.1
X − zα/2 √ , X + zα/2 √ = 11.6 − 1.645 √ , 1.6 + 1.645 √ = (11.15, 12.05). (
n n 225 225
Pn
Suppose X1 , . . . , Xn is a set of IID random variables. Then, the empirical nth moments are m̂1 = i=1 Xi /n.
These are also referred to as the sample moments, and m̂1 is the sample mean. The method of moments uses the
approximation mi ≈ m̂i to estimate the parameters of the model. For example,
n
! n n n
2 1X 2 2 1X 2 1X 2 1X 2 2
Var(X) ≈ m̂2 − (m̂1 ) = Xi − X = Xi − X = (X − X ).
n i=1 n i=1 n i=1 n i=1 i
46
Notice that this is different than the sample variance, which is S 2 = n(m̂2 − m̂21 )/(n + 1). This difference is
significant when n is small, so consider a sample of size 1. Then, the sample variance is undefined, because the
variability is meaningless with only one data point. However, the method of moments just reports the variance as
zero, since only the one data point matters.
The bias of an estimator θ̂ is E[θ̂] − θ. An estimator with a bias of zero is called unbiased.31 For example,
the sample mean is unbiased: E[X] = µ. Then, the sample variance is unbiased, but the method of moments is
biased (though as n → ∞ it becomes less asymptotically biased).
An estimator θ̂ is (weakly) consistent if for any ε > 0, limn→∞ P (|θ̂ − θ| < ε) = 1. This intuitively means
that as more and more data is fed to the estimate, it gets more and more accurate. Note the similarity in the
expression to that of Proposition 19.1. The sample mean is also consistent, but there is not generally a relation
between consistency and presence or lack of bias. Method-of-moments estimates are also generally consistent.
Example 20.3. Suppose X1 , . . . , Xn are IID random variables, with Xi ∼ Poi(λ). Then,
n
1X
λ = E[Xi ] ≈ m̂1 = X = X − i = λ̂.
n i=1

Since λ = Var(Xi ) as well, one has another estimate of λ which isn’t always the same:
n
1X 2 2
λ = E[Xi2 ] − E[Xi ]2 ≈ m̂2 − m̂21 = (X − X ) = λ̂0 .
n i=1 i

In general, the estimator of the lowest order or the one that is easiest to compute is used, and they tend to be
the same. (
Example 20.4. If X1 , . . . , Xn are IID and Xi ∼ N(µ, σ 2 ), then µ ≈ µ̂ =
P
Xi /n, but estimated variance is
not equal to the sample distribution. One is biased, and the other isn’t. If Xi ∼ Uni(α, β), one can√obtain
similar expected
√ means µ̂ and variance σ̂ 2 , which leads to a system of equations in α and β: α̂ = X − σ̂ 3 and
β = X + σ̂ 3. Notice that there could be data outside of these data points, so be careful. (

21. Likelihood: 5/17/13

Definition 21.1. Consider n IID variables X1 , . . . , Xn . In some sense, X
Qi nis a sample from a density function
f (Xi | θ) for some parameter θ. Then, the likelihood function is L(θ) = i=1 f (Xi | θ).
This represents how likely some set of data (x1 , . . . , xn ) is given the density, though it becomes a product
because the Xi are indepedent.
Definition 21.2. The maximum likelihood estimator (MLE) of θ is the value (or set of values, if θ is a vector) of θ
that maximizes the likelihood function L(θ). More formally, θMLE = arg maxθ L(θ).32
For this, it is often more convenient to use the log-likelihood function
n
Y n
X
LL(θ) = log L(θ) = log f (Xi | θ) = log f (Xi | θ).
i=1 i=1

Here, log denotes the natural log, though the base doesn’t matter for maximizing. This still works for maximizing
θ because it is monotone, so x ≤ y iff log x ≤ log y for x, y > 0. Thus, arg maxθ L(θ) = arg maxθ LL(θ) =
arg maxθ cLL(θ) for any positive constant c, and this can be found with a bit of linear algebra:
• Find the formula for LL(θ) and differentiate it with each ∂ LL(θ)
∂θ , and set it to zero.
• This is a set of equations, which can be solved simultaneously.
• Then, it’s necessary to check that the critical points aren’t minima or saddle points by checking nearby
values.

31Bias is a loaded word in the outside world, but depending on the application having some bias in an estimator may cause more
accurate results.
32The function max(f (x)) returns the maximum of f (x), but arg max(f (x)) returns the value x such that f (x) is maximized.

47
Example 21.3. Suppose X1 , . . . , Xn are IID random variables such that Xi ∼ Ber(p). Then, the PMF can be
rewritten in the form f (Xi | p) = pxi (1 − p)1−xi , where xi = 0 or xi = 1. This has the same meaning, but is
now differentiable. Thus, the likelihood function is
n
Y
L(θ) = pXi (1 − p)1−Xi ,
i=1

and therefore the log-likelihood is

n
X n
X
LL(θ) = log(pXi (1 − p)1−Xi ) = (Xi log p + (1 − Xi ) log(1 − p)) = Y log p + (n − Y ) log(1 − p),
i=1 i=1
Pn Pn
where Y = i=1 Xi . After a bunch of calculus, the MLE is pMLE = Y /n = i=1 Xi . This agrees with intuition,
and looks like sample mean and the method of moments for this distribution. (
Example 21.4. Suppose the X1 , . . . , Xn are now IID Q Poisson variables, Xi ∼ Poi(λ). Then, the PMF is
f (Xi | λ) = e−λ λxi /xi !, which makes the likelihood e−λ λxi /xi !. Thus, the log-likelihood is
n  −λ Xi  X n
X e λ
LL(θ) = log = (−λ log e + Xi log λ − log(Xi !))
i=1
Xi ! i=1
n
X n
X
= −nλ + log(λ) Xi − log(Xi !).
i=1 i=1

P looks alarming until you realize we only have to differentiate with respect to λ, showing that λMLE =
This
Xi /n. (
Example 21.5. If Xi ∼ N(µ, σ 2 ), then one obtains the log-likelihood
n 2 2
! n 
e−(Xi −µ) /(2σ)  √  (X − µ)2 
i
X X
L(θ) = log √ = log σ 2π − 2)
.
i=1
σ 2π i=1
(2σ

Now, there are two things to differentiate:

n n
∂ LL(µ, σ 2 ) X 2(Xi − µ) 1 X
= = (Xi − µ) = 0,
∂µ i=1
(2σ 2 ) σ 2 i=1

and σ (not σ 2 , since this is with respect to a variable):

n
∂ LL(µ, σ 2 ) n X (Xi − µ)2
=− + = 0.
∂σ σ i=1 σ3
P P
Then, Xi = nµ, so µMLE = Xi /n, which is the sample mean again. This is useful, but there are some
distributions where it isn’t the sample mean. Notice that µ is unbiased, but σ is biased:
n
2 1X
σMLE = (Xi − µ2MLE ).
n i=1

This seems counterintuitive, but there’s no reason that it shouldn’t be the case. (
Example 21.6. Now, suppose Xi ∼ Uni(α, β). Then, the PDF is f (Xi | α, β) = 1/(β − α) if α ≤ xi ≤ β and is
zero otherwise, so the overall likelihood is
(  n
1
β−α , α ≤ x1 , . . . , x n ≤ β
L(θ) =
0, otherwise.
This can be solved with Lagrange multipliers, which is complicated and unfortunate, so turn instead to intuition.
The goal is to make the interval as small as possible while guaranteeing that all of the values are still within
the region, which means that αMLE = min(x1 , . . . , xn ) and βMLE = max(x1 , . . . , xn ). (
48
These do interesting things in small sample sizes. In many cases, the sample mean is obtained, which is pretty
nice. But the normal distribution is biased, and underestimates for small n, and the uniform distribution is also
biased, and behaves badly for small samples (e.g. if n = 1, so that αMLE = βMLE , which isn’t necessarily right).
This just fits the distribution best to what has already been seen, but this doesn’t always generalize well and
the variance might be too small.
However, the MLE is frequently used, because it has lots of nice properties:
• It’s consistent: lim P (|θMLE − θ| < ε) = 1 for any ε > 0.
n→∞
• Though it is potentially biased, the bias disappears asymptotically as n increases.
• The MLE has the smallest estimate of all good estimators for large samples.
This, this is frequently used in practice where the sample space is large relative to the parameter space. However,
this is a problem if the variables are dependent: joint distributions cause the number of parameters of n variables
to be about 2n . This can cause issues in modeling.
Example 21.7. Suppose Y1 , . . . , Yn arePindependently and identically distributed random variables, such that
m
Pi m∼ Multinomial(p1 , . . . , pm ), where i=1 pi = 1, and let Xi be the number of trials of outcome i where
Y
i=1 Xi = n. Thus, the PDF is
m
Y pxi i
f (X1 , . . . , Xm | p1 , . . . , pm ) = n! .
i=1
xi !

Thus, the log-likelihood is

m
X m
X
LL(θ) = log n! − log(Xi !) + Xi log pi .
i=1 i=1
This time, Lagrange multipliers are necessary. The goal is to maximize
m
X m
X
A(θ) = Xi log pi + λ pi − 1,
i=1 i=1
P P
so after differentiating, pi = −Xi /λ, so using the initial constraints Xi = n and pi = 1 one can see that
λ = −n. Intuitively, this means that the probability pi is the proportion of outcome i. This makes sense. (
If one has a six-sided die and rolls 12 rolls, none of which is a 3, which suggests that maybe one would never
roll a 3. In the frequentist interpretation (probability is frequency in the limit). more rolls are necessary to
understand this, but there’s another interpretation. Bayesian probability is a belief in something, maybe given
some prior information, but still a belief. If you were asked the probability of it raining tomorrow, how many
tomorrows have you experienced in order to make a valid prediction?
Example 21.8. Suppose one has two envelopes, one of which contains $X and the other of which contains $2X.
If someone takes one envelope and opens it, seeing $Y , the expected value of switching the envelope is 5Y /4.
But this didnt depend on knowing the value of Y , so one should always switch envelopes. . . even after one has
already switched. . . right? (

22. More Bayesian Probability: 5/20/13

Returning to Example 21.8, we know that if Y is the money in the envelope one selects and Z is the money
in the other envelope, then E[Z | Y ] = 5Y /4, which leads to the confusing switching strategy. Various sorts of
bringing other information in (e.g. if there’s an odd number of cents in one) can lead to better guesses. However,
the switching strategy assumes that all values of X (the smaller amount of money in the envelopes) are equally
likely, but this means that the PDF would have to be everywhere zero.
This is an example of Bayesian probability, which is more subjective: since the distribution can’t be uniform
and it must exist, then what is it? The frequentist must play infinitely many times to determine this, but if only
one trial is allowed, the Bayesian must work based on prior beliefs: what would be the most reasonable value to
see here? This leads to the prior belief of the distribution of X.
For example, suppose someone has a coin that’s never been flipped before. It seems reasonable that it will be
a fair coin, but do we know that? A purely frequentist approach has no idea what the answer is.
49
 A Bayesian would approach the envelope problem by determining a prior value for E[Z | Y ], and then keep
the envelope if Y > E[Z | Y ], and switch otherwise. Opening the envelope provides data to compute P (X | Y )
and therefore E[Z | Y ]. In order to make any decision at all, one has to make a prior decision.33
In this interpretation, Bayes’ theorem P (θ | D) = P (D | θ)P (θ)/P (D) can be thought of with θ as the
parameters of a model and D as the data. Then, P (θ) is the prior estimate, and P (D | θ) is the likelihood this
has been seen before. Then, the updated R (posterior) probability is P (θ | D). However, P (D) is more nuanced to
calculate: it can be found by P (D) = P (D | θ)P (θ) dθ, but it is better to think of as a constant that turns the
left-hand side into a probability distribution.
Example 22.1. Suppose one has a prior distribution θ ∼ Beta(a, b) with data D that is n heads and m tails.
Then,
fD|θ (D | θ = p)fθ (θ = p)
fθ|D (θ = p | D) =
fD (D)
n+m n


p (1 − p)m (pa−1 (1 − p)b−1 )
= n
= C3 pn+a−1 (1 − p)m+b−1 .
C1 C2
Here, C1 and C2 are the original normalizing constants for the original distributions, and C3 is the normalizing
distribution for the final distribution. θ is a prior distribution based on what one thinks the data is before any
trials. These can be very different, but as long as enough trials are done, the posteriors will always converge to
the true value. (
Qn
Recall the MLE θMLE = arg maxθ i=1 f (Xi | θ); one has a more real-world estimate of θ given the data.
This is known as the maximum a posteriori estimate: suppose X1 , . . . , Xn are IID.
f (X1 , . . . , Xn | θ)g(θ)
θMAP = arg maxθ f (θ | X1 , . . . , Xn ) = arg maxθ
h(X1 , . . . , Xn )
Qn n
( i=1 f (Xi | θ))g(θ) Y
= arg maxθ = arg maxθ g(θ) f (Xi | θ),
h(X1 , . . . , Xn )| i=1

where g(θ) is the prior distribution. Notice that there is a significant philosophical distinction, but not as much
of a mathematical distribution. Again, it can be convenient to use logarithms:
n
!
X
θMAP = arg maxθ log g(θ) + log(f (Xi | θ)) .
i=1

Recall the beta distribution with parameters a and b. These are called hyperparameters, because they are
parameters that determine the parameters of the distribution. For example, a fair coin is given a prior Beta(x, x)
distribution, with higher values of x correlated with a lower variance.
In order to do this with a multinomial distribution, introduce the Dirichlet distribution: the way that the
beta distribution is conjugate to the binomial (or Bernoulli) distribution is the way the Dirichlet distribution is
conjugate to the multinomial distribution. This distribution is written Dirichlet(x1 , . . . , xn ):
m
1 Y
f (x1 , . . . , xn ) = xai −1 .
B(a1 , . . . , am ) i=1 i
Pm
The prior (imaginary trials) represent seeing i=1 ai − m imaginary trials such that ai − 1 are of outcome
i. Then, after observing n1 + · · · + nm new trials (such that ni had outcome i), the new distribution is
Dirichlet(a1 + n1 , . . . , am + nm ). Here is an animation of the logarithmic density of Dirichlet(a, a, a).
The Dirichlet model allows one to work around the multinomial problem presented in the previous lecture:
if one rolls a die but never sees a 3, one could do an MAP estimate in which some number of prior trials did
give a 3. The MAP estimate for Dirichlet(k + 1, . . . , k + 1) is pi = (Xi + k)/(n + mk) for this experiment. Now,
the probability of rolling a 3 is nonzero, which is nice. This method with k = 1 is also known as Laplace’s law.
Notice that this is a biased estimate.

33Frequentist and Bayesian viewpoints are the closest thing probability has to a religion. The professor, for example, is a Bayesian, and
the author of the textbook is very frequentist.
50
 The conjugate distribution for the Poisson and exponential distributions is Gamma(α, λ), which represents a
prior of α imaginary events in a time period of λ. To update this for a posterior distribution, one observes n
events in k time periods, leading to a posterior distribution of Gamma(α + n, λ + k).
Finally, conjugate to the normal distribution there are several of these depending on which of the mean and
variance are known. For example, Normal(µ0 , σ02 ) where σ 2 is known but µ isn’t, is a hyperparameter, where
one believes the true µ ∼ N(µ0 , σ02 ). After observing n data points, the posterior distribution is
 Pn     −1 !
µ0 i=1 xi 1 n 1 n
N + / 2+ 2 , + 2 .
σ02 σ2 σ0 σ σ2 σ
Generally, one doesn’t know the variance without other information, but it could be estimated without knowing
the mean, making this useful.

23. Machine Learning: 5/22/13

Today, the parameter estimation that has been discussed in the previous lectures will be grounded with some
actual algorithms. These methods will be applied in the next (and last) problem set.
Notice that in the real world, one doesn’t have perfect distributions; instead, there’s just data, and one should
try to make predictions based on the data. Often, with enough data, one doesn’t need any expertise in the
specific field, since it can be viewed as abstraction.
Machine learning in general is pretty useful, and the small amount covered in this class will form a sample of
the AI that every CS major should know. Machine learning is a wide field, but this class will deal with a subfield
of preduction: given some data, one wishes to build a model for it that allows for the making of predictions. The
probability formalism is very useful for this process.
Formally, one has a vector of m observed variables X ~ = hX1 , . . . , Xm i. These are called the input features or
input variables. They are the things we can observe about the world, and are fed into the black-box algorithm.34
Then, the goal is to make a prediction for a variable that we don’t know, called the output feature or variable Y .
There are generalizations in which Y is a vector, but they won’t be considered here. Y is called a dependent
variable.
The goal is to “learn” a prediction function g(X) ~ that spits out predictions of Y : Ŷ = g(X). ~ When Y is
discrete, this is called a classification task, and when Y is continuous, this is called a regression.
Machine learning has lots and lots of applications:
• Stock prediction: one has a lot of economic variables (e.g. mortgage rates, interest rates from the Fed,
current stock prices), and wishes to predict what the stock market does. If a sufficiently good model can
be found, then there’s a nice way to make money (e.g. D.E. Shaw).
• Computational biology: given some input data (one’s DNA and environmental risk factors), what is the
likelihood for getting a certain disease?
• This can be used to predict what products a set of consumers might prefer.
• Credit card fraud can be detected: a model can be set up for a given user using machine learning, and
actions particularly outside of that model is flagged. For a while, AT&T was a world leader in machine
learning because of credit and telephone fraud.
• Spam detection is another example. In the header of some email systems, there’s a field called the
Bayesian spam probability, which indicates a prediction as to how spammy an email might be given
the distribution of the words and a model. The order of the words is completely ignored, interestingly.
Using increasingly sophisticated models, more and more spam is correctly detected even as more and
more email sent is spam, illustrating that models are generally better when they have more data.
The data is given as a set of N training instances, each of which is a data points (often a vector of values). This
is a pair (hx1 , . . . , xm i, y) of observed inputs and the correct result given those inputs. These are previously
observed data, or can be thought of as historical data. This illustrates what we want to predict, and with enough
data, the probabilistic dependencies can be elucidated.
The predictor g is often parametric: it has some parameters that get determined by the training data, such
as g(X) = aX + b, in which the parameters are a and b. The choice of parameters is something that can be
discussed in considerably more detail; take CS 229 if interested. For a regression, the goal is to minimize
34Sometimes statisticians call these independent variables (as opposed to dependent variables), but they are often not independent in
the probabilistic sense. Be careful.
51
the mean-squared error E[(Y − g(X)) ~ 2 ] (MSE), which is sometimes known as a loss function (in which the
lost object is truth). There are other ways to judge models, such as mean absolute value or logarithmic ones;
the former is suboptimal because it isn’t differentiable at zero. For classification, the best Y has generally
~ = arg max P̂ (Y | X).
g(X) ~ Under some surprisingly mild assumptions, this is the best that can be done.
Consider a set of N training instances (~x1 , y 1 ), . . . , (~xn , y n ). (Here, superscripts are used as indices, not
powers). The learning algorithm processes these inputs into a classifier g, which can be used to make predictions
for new datasets.
Example 23.1. One concrete example is linear regression in which a variable Y is predicted given a single variable
X. One assumes the model is linear: Ŷ = g(X) = aX + b. Then, given N pairs of points (x1 , y 1 ), . . . , (xn , y n ).
Geometrically, one has a set of points in the plane, and which line minimizes the error of the predictions?
Typically, the mean-squared error is used.
Some math shows that the mean-squared error is E[(Y − aX − b)2 ], and its partial derivatives with respect
to a and b can be calculated:
∂
E[(Y − aX − b)2 ] = E[−2X(Y − aX − b)] = −2E[XY ] + 2aE[X 2 ] + 2bE[X]
∂a
∂
E[(Y − aX − b)2 ] = E[−2(Y − aX − b)] = −2E[Y ] + 2aE[X] + 2b.
∂b
This gives a system of equations whose solution is a = Cov(X, Y )/ Var(X) = σy /σx ρ(x, y) and b =
σy /σx ρ(x, y)µ. These quantities are obtained as estimates from the training data, using any of the meth-
ods discussed in previous lectures. (
Example 23.2. Now, suppose X and Y are both discrete, making this a classification problem: suppose
X ∈ {1, 2, 3, 4}. One way this can be done is to take a continuous range and pick several subsets (e.g. the
temperature is in the 60s, 70s, or 80s, which form the discrete domain on three elements). Then, Y is the weather
for the next day: either rainy or sunny the next day. (
Given the training data, one can estimate the joint PMF p̂X,Y (X, Y ), and then, using Bayes’ theorem, one
has P (Y | X) = pX,Y (x, y)/pX (x). Then, for a new X, predict Ŷ = g(X) ~ = arg maxY P̂ (Y | X).
~ However,
since pX (x) isn’t modified by the choice of Y , it can be ignored (since one takes the arg max anyways):
Ŷ = arg maxY P̂ (X | Y )P̂ (Y ).
There is more than one way to estimate the joint PMF:
• The MLE method just counts the number of times each pair appears and then normalizes them to get
probability.
• One can also add a Laplace prior: add 1 to each pair (not just the ones that appear in the training
data), and normalize so that the total value is 1. This is useful if all of the events have some nonzero
probability and is thus helpful when the data is limited.
A new observation of X = 4 can be used to make a prediction of the weather tomorrow: as Ŷ = arg maxY P̂ (X, Y ) =
arg maxY P̂ (X | Y )P̂ (Y ). Here, the MLE and Laplace estimates agree (with the given table). The similarities
and differences are worth noting. Specifically, the probabilities differ: without the Laplace prior, some events
seem to have zero probability, which might not happen in the real world. It is a biased estimate in that we
introduce our own subjective belief about the world, but this is sometimes better.
Now, suppose there are m input values X ~ = hX1 , . . . , Xm i. In theory, one could just take the table of
P (X1 , . . . , Xn | Y ) for all possible values of all of the Xi , which is incredibly huge and therefore not all that
insightful: most of the entries will be zeros, so the predictions won’t be all that helpful. The asymptotic size is
actually important: it’s exponential in m, which was a problem in simulations in which m ≈ 10000.
Thus, a simpler model is necessary: enter the Naı̈ve Bayesian Classifier35 This makes the assumption that all
of the Xi are conditionally independent given Y . This is often untrue in practice, but allows the predictions to
be simplified while still being reasonably accurate.
The beauty of this is that the conditional probability factors:
m
Y
P (X~ | Y ) = P (X1 , . . . , Xm | Y ) = P (Xi | Y ),
i=1

35This is apparently referred to as the Idiot Bayes Classifier by statisticians.

52
so the table now has linear space, with only the one probabilistic assumption!
Example 23.3. Let X1 be an indicator variable that indicates that someone likes Star Wars, and X2 is that a
person likes Harry Potter. Then, Y is an indicator variable for whether someone likes the Lord of the Rings.
This is a standard marketing application, for finding recommended products, and previous purchases can be used
to make a model. Thus, we have the joint and marginal probability tables.
Now, suppose a person comes along and X1 = 1, but X2 = 0. Then, under the Naı̈ve Bayes assumption,
Ŷ = arg maxY P̂ (X~ | Y )P̂ (Y ) = arg maxY P̂ (X1 | Y )P̂ (X2 | Y )P̂ (Y ). The arg max can be evaluated by
picking Y = 0 and Y = 1 and picking whichever makes the result better. (
The number of parameters can grow very large: in one model of spam classification, m ≈ 100000, and there
are m variables corresponding to indicators for whether a given word (out of all English words) appeared in an
email. Since m is extremely large, then the Naı̈ve Bayes assumption is necessary. Additionally, since there are
lots of words that could be used in spam but that aren’t in the sample dataset, the Laplace estimate is probably
a good idea. Then, the data is tested with some other testing data (you can’t use the same training data; that
would be cheating).
There are various criteria for effectiveness, such as precision (correctly predicted the most cases relative to
incorrect ones) or recall (correctly predicted the most cases relative to all cases).

24. Logistic Regression: 5/24/13

There are a couple formulas that might be helpful for the next problem set, regarding a Naive Bayesian
classifier: P̂ (Y = 0)is the number of instances in the class with value 0 over the total number of instances,
and P̂ (Xi = 0, Y = 0) is the number of instances where Xi = 0 and the class is zero. Thus, P̂ (Xi = 0 |
Y = 0) = P̂ (Xi = 0, Y = 0)/P (Y = 0) and P̂ (Xi = 0 | Y = 1) = P̂ (Xi = 0, Y = 1)/P̂ (Y = 1). Finally,
P (Xi = 1 | Y = 0) = 1 − P̂ (Xi = 0 | Y = 0), and the estimate for y is ŷ − arg maxy P (X | Y )P (Y ) =
arg maxy (log(P (X | Y )P (Y ))). The log estimate is helpful to prevent numerical underflow (i.e. small data
values becoming eaten by rounding error). It can be computed specifically as
m
Y m
X
log P (X | Y ) = log(P (X1 , . . . , Xm | Y )) = log P (Xi | Y ) = log P (Xi | Y ).
i=1 i=1

This is the MLE formula, and the formula for the Laplacian is similar. In some sense, this classifier models
the joint probability P (X, Y ). However, the goal is to find P (Y | X) (since X was observed, and Y is to be
predicted), so the goal is to find ŷ = arg maxy P (Y | X).
One can use a technique called logistical regression to model P (Y | X), which can be called the conditional
likelihood. This uses a function calledPthe logistic function (also sigmoid or “squashed S” function): P (Y = 1 |
m
X) = 1/(1 + e−z ),36 where z = α + j=1 βj Xj . In some sense, one takes this nice linear function and runs it
Pm are the α and βj , and often, the simplifying trick
through the logistic function. The parameters that can change
of letting X0 = 1 and β0 = α, so the sum becomes z = j=0 βj Xj .
Since Y = 0 or Y = 1, then P (Y = 0 | X) = e−z /(1 + e−z ), with z as before. This still doesn’t illuminate
why the function was chosen, but calculate the log-odds:
P (Y = 1 | X) 1
log = log −z = z.
P (Y = 0 | X) e
This functional form guarantees that the log-odds is a linear function, which is pretty convenient.
Now, the log-conditional likelihood of the data is
n
X
LL(θ) = (yi log(P (Y = 1 | X)) + (1 − yi ) log(P (Y = 0 | X))).
i=1

This sum represents dividing the data into two cases, those with yi = 0 and yi = 1. Given this, one might want
to know the βj that create the MLE, but, unfortunately, there is no analytic derivation of it.

36This is an arctangent function, strangely enough.

53
 Figure 8. The standard logistic function. Source

However, the log-conditional likelihood function is concave, so it must have a single global maximum. This
can be repeatedly estimated by computing a gradient repeatedly somewhat like Newton’s Method:
 
∂ LL(θ) 1
= xj y − = xj (y − P (Y = 1 | X)).
∂βj 1 − e−z
Thus, LL(θ) can be iteratively updated using the formula
 
new old 1
βj = βj + cxj y − ,
1 − e−z
Pm
where z = j=0 βjold and c is a constant representing the step size.37 Then, this is done repeatedly, until a
given stopping criterion is met: one may just run it a fixed number of times or until some error bound is met.
Here’s the derivation:
LLi (θ) = yi log P (Y = 1 | X) + (1 − yi )P (Y = 0 | X)
P (Y = 1 | X)
= yi log + log P (Y = 0 | X)
P (Y = 0 | X)
m
X
= yi βj Xj + log e−z − log(1 + e−z )
j=0
m
X m
X
= yi βj Xj − βj Xj − log(1 + e−z ).
j=0 j=0

xj e−z 1 + e−z e−z

 
∂ LLi (θ)
=⇒ = yxj − xj + = xj y − +
∂βj 1 + e−z 1 + e−z 1 + e−z
 
1
= xj y − .
1 + e−z
Here, y = yi , though it’s not actually ambiguous because the function is LLi (θ) only. However, the above model
seems to only compute the gradient based on one data point. which seems suboptimal. This isn’t actually the
case, since the contributions to the gradient by each data point are all averaged to give the next step.
This is known as a batch logistic regression algorithm, and is an example of a gradient-ascent algorithm:
(1) Initialize βj = 0 for all 0 ≤ j ≤ m. There’s no particular reason to initialize them in a specific place, so
zero is a good choice.
37One should do this with care, since if c is too large, it might diverge, and smaller step sizes will take longer to converge. There are
several ways to choose c, and one of them is called annealing, in which c starts large and gradually decreases.
54
 (2) An epoch is the number of passes over the data that the algorithm makes. Thus, for each i less than the
number of epochs,
(a) Initialize the gradient vector gradient to zero.
(b) Then, for each instance (x1 , . . . , xm , y) in the training data:
(i) For each xj , update the gradient:
 
1
gradient[j]+ = xj y − ,
1 + e−z
Pm
where z = j=0 βj xj . This is the batch part of the algorithm.
(3) Finally, update all of the βj as βj + = η · gradient[j] for 0 ≤ j ≤ m. Here η is a preset constant
called the learning rate.
This determining of the parameters βj is called training. Once this has been done, one Pm can test the classifier:
for each new test instance X, compute p = P (Y = 1 | X) = 1/(1 + e−z ), where z = j=0 βj Xj as before, and
classify ŷ = 1 if y > 0.5 and 0 otherwise. Notice that the βj are not updated during the testing phase, though
in some real systems, they may be updated periodically.
Geometrically, the goal of a logistic regression is to find a hyperplane in a high-dimensional space that
separates the data instances where Y = 1 from those where Y = 0 (i.e. it is (n − 1)-dimensional where the
space is n-dimensional; the key is that it’s linear). Functions or data sets in which this can be done are called
linearly separable. Naı̈ve Bayes is also a linear function, and even has the same functional form, but they
compute different things: Naı̈ve Bayes tries to best fit P (X, Y ), and the logistic regression models P (Y | X),
though both use the same linearity assumption. Thus, their solutions are given by different algorithms.
Not all functions are linearly separable; just take the xor function. In practice, though, data that isn’t linearly
separable can still be approximated reasonably well. This dilemma halted research in neural networks for a
couple of decades.
Notice that Naı̈ve Bayes computes the joint probability in order to calculate the conditional probability, which
is a bit of extra work (though it could use P (X, Y ) to generate new data points). The logistic regression model
is a discriminative model, because it doesn’t model the whole distribution, but just tries to separate the data.
Logistic regression can be generalized when Y takes on more than two values, in which case the vector of βj
becomes a matrix. There are other choices that one may wish to make:
• What if the input variables are continuous, rather than discrete? The logistic regression is fine, though
Naı̈ve Bayes has a harder time. Specifically, some parametric form is necessary, or one could discretize
the continuous values into ranges.
• If the values are discrete, Naı̈ve Bayes handles it fine, because multi-valued data is just a multinomial
distribution. The logistic regression has a harder time, since it requires its inputs to be numbers (unlike
the Naı̈ve Bayes model), which sometimes misrepresents the data.
The overall goal of machine learning is to build new models that generalize well to predicting new data.
Sometimes, this even causes something called overfitting, in which the model fits so well that generality is lost.
For example, one could use am interpolating polynomial to exactly hit all data points when a linear approximation
is more likely to be correct (small errors in the data may be amplified in overfitting). The logistic regression
is more prone to overfitting, because it doesn’t model the whole distribution, but only focuses on Y . It can
also happen if the training and testing data differ significantly (so that the nuances of the training data are
overstated, as in McDonald’s McRib release discussed earlier in this class). There are methods to mitigate
overfitting in logistic regression, analogous to using priors in Naı̈ve Bayes, though the optimization proces sis
more complicated because there isn’t an analytic solution.
In some sense, a logistic regression takes n inputs xi with parameters βi is a one-node neural network. Thus,
one could connect many of them to create a neural network, though the algorithms are trickier: something called
back-propagation allows one to approximate any function with a sufficient number of nodes, something called a
uniform approximator.

25. Bayesian Networks: 5/29/13

With regards to machine learning, it is important to keep some perspective: there are these powerful tools
for making predictions, but viewing these algorithms as black boxes isn’t ideal. It is generally also useful to
understand the domain: what things are correlated, what things are independent, etc.
55
 For example, palm size negatively correlates with life expectancy. This surprising result occurs because
women tend to have smaller palms than men, but on average live longer. Here, the correlation is definitely not
the whole story, in a reasonably obvious way — but what about all of the other medical correlations that might
be explained by other effects?
The way to formalize this is something called a Bayesian network (for more on this subject, take CS 228 or
228T). This is a directed graph in which the nodes are random variables, and an arc from X to Y is present if X
has a direct influence on Y , in which case X is called a “parent” of Y .38 Then, each node X has probability
P (X | parents(X)). Finally, in order for the math to make sense, it is necessary that there are no cycles (where
direction matters), so that the graph is a directed acyclic graph (DAG).

Figure 9. An example of a simple Bayesian network. Source

One important idea is that each node is conditionally independent of its non-descendants, given the parents. For
example, given gender G, palm size S and life expectancy L are independent: P (S, L | G) = P (S | G)P (L | G).
With each node one can associate a conditional probability table (CPT) of the probability of it taking on some
value given the parents.
Thus, it is possible to obtain all of the joint probabilities given the CPT: the conditional independence of the
graph modularizes the calculation, yielding
m
Y
P (X1 , . . . , Xm ) = P (Xi | parents(Xi )).
i=1

The CPT is useful because the number of parameters in a full table is exponential in the number of nodes, but
with this setup, it’s linear, and a lot more of the parameters can be determined locally from each other.
The Naı̈ve Bayes model can be incorporated into this: a parent Y gives some probability to its (conditionally
independent) children X1 , . . . , Xm . Then, the network structure encodes the assumption
m
Y
~ | Y ) = P (X1 ∩ · · · ∩ Xm | Y ) =
P (X P (Xi | Y ).
i=1

Then, things such as the linearity make more sense. Additionally, if the Naı̈ve Bayesian assumption is
violated, then the dependencies between the Xi can be added as arcs, yielding a better model that is still
a Bayesian network. But Bayesian networks are more general than the specific prediction problem: instead
of observing all of the Xi , only some subset E1 , . . . , Ek is observed. Then, the goal is to determine the
probability of some set Y1 , . . . , Yc of unobserved variables, as opposed to a single Y . Formally, the goal is to
find P (Y1 , . . . , Yc | E1 , . . . , Ek ).

Example 25.1. Consider the following Bayesian network representing conditions around doing well on the CS
109 final:

38Does this mean causality or just direct correlation? This is a philosophical problem, and doesn’t have a clear answer, given that many
physicists claim causality can’t really exist.
56
 Figure 10. Will the presence of this diagram affect how you study for the final?

Here, the professor can know B, A, and R, but doesn’t know E. One can calculate P (A = T | B = T, R = T )
by summing over the unseen variables:39
X
P (A = T, B = T, R = T, E)
P (A = T, B = T, R = T ) E=T,F
P (A = T | B = T, R = T ) = = X X .
P (B = T, R = T ) P (B = T, R = T, E, A)
E=T,F A=T,F

Notice how many fewer parameters are necessary for this, and the joint probability decomposes as P (A, B, E, R) =
P (E)P (B | E)P (R)P (A | E, R): the product of the probabilities given the parents. (
Another useful concept is a probability tree, which is a tree that models the outcome of a probabilistic event.

3 6
P (B1 ∩ B2 ) = 7 · 9
W
6
Bag 2 9
3W, 6B B
B 3 3 3
3 9 P (B1 ∩ W2 ) = 7 · 9
7
Bag 1
4W, 3B
W P (W1 ∩ B2 ) = 4
· 5
B 7 9
4
7 5
Bag 2 9
4W, 5B W
4 4 4
9 P (W1 ∩ W2 ) = 7 · 9

Figure 11. An example of a probability tree. Source

Probability trees are traditionally written sideways, and can be thought of as probabilistic flowcharts. This is
useful for modelling decisions.
For example, consider a game where one can receive $10 for not playing, or $20 with probability 0.5 when
playing, and $0 otherwise. If the numbers are multiplied by 1000, people much more strongly prefer not to play
the game. This is an example of utility, as seen before, which means that the leaves of the decision tree are
utilities rather than dollar values. This can be useful for making decisions about life expectancy, etc., and are
often difficult choices.
In a similar game, the expected monetary value is the expected dollar value of winning, and the risk premium
is the amount that one would be willing to give up to eliminate risk: certainty has a utility. This is exactly why
39This is the simple algorithm, but not the most efficient one. That’s a discussion for CS 228.

57
insurance companies exist; how much money would you pay to mitigate the existence of a big loss, even if the
expected loss is greater.
Utility is nonlinear: suppose one could gain $5 with probability 0.5 and otherwise get nothing, or just receive
$1 for not playing. Most people would play, but if the dollar values were scaled by a million, most people
wouldn’t. In some sense, $10000000 is life-changing, but the next $40000000 isn’t. Thus, the shape of the utility
curve isn’t linear, and depends on one’s preference for risk. Most people are risk-averse, so their utility functions
are concave down. Some people are risk-preferring, so their utility curves are concave up. A typical utility curve
looks something like U (x) = 1 − e−x/R . Here, R is one’s risk tlerance, and is rough;y the highest value of Y for
which one would play a game in which one wins $Y with probability 0. and loses $Y /2 with probability 0.5, and
receives nothing for not playing.
Then, one can calculate how irrational or irrational one is: consider a game in which one receives $1000000
for not playing, and when playing, receives $1000000 with probability 89%, $5000000 with probability 10%,
and nothing with the remaining 1%. Similarly, one could choose to play a game where $1000000 is won with
probability 0.11 (and nothing happens otherwise), or $5000000 is won with probability 0.1 and nothing is won
otherwise.
Sometimes, people pick the second options in both cases, which is inconsistent with any utility function. This
is known as the Allais paradox, since the percentages are the same.
A “micromort” is a one-in-a-million chance of death. This leads to an interesting question: how much would
one want to be paid to take on the risk of a micromort? How much would you pay to avoid one micromort?
People answer differently to these questions. Though this sounds abstract, it comes up in everyday risks that
people need to take every day when making decisions. Understanding this makes these decisions a little more
insightful.
Finally, consider an example from sports, which has lots of applications of probability. Two batters have
their batting averages over two years. If player B has better batting averages each year, he must be better. . .
but what if the first year A has 0.250 and B has 0.253, and in the second year, A has 0.314 and B has 0.321,
giving them combined averages of 0.310 and 0.270. Oops; is A better? This is known as Simpson’s paradox,
and actually happened to Derek Jeter and David Justice in 1995 and 1996. This has applications to marketing
(people may prefer different products) or effectiveness of medicine. One might have one medicine favored when
the data is broken down by gender, but the other when the total data is considered. This can happen because
one treatment has significantly fewer trials than the other in one case, but not the other. Unsurprisingly, this
comes up all the time in the real world.

26. Generating Random Numbers: 5/31/13

In many applications, one wants to be able to generate random numbers (and therefore also random
permutations, etc.). However, computers are deterministic. Thus, one often settles for pseudo-randomness, which
is a condition that a sequence of numbers “looks” random (for a definition that can be made precise; intuitively,
it shouldn’t be possible to easily predict the next number), but is in fact deterministic.
Most random number generators use a linear congruential generator (LCG), in which one starts with a seed
X0 and gives the next number by a function Xn+1 = (aXn + c) mod m. The effectiveness of this algorithm
is quite sensitive to the choices of a, c, and m (e.g. m = 1), and often, people take them to be large primes.
Generally, language standards don’t specify these numbers, or even whether one should use LCG at all, but it’s
fast and useful.
Notice that the sequence of random numbers must cycle, since there are only m choices for an Xi . This
relates to an idea called Kolmogorov complexity.
In C and C++, the <stdlib.h> library contains a function int rand(void) which returns a value between
zero and RAND_MAX, inclusive (where the latter quantity is at least 32767 and is nowadays about 264 ). Ideally, the
values returned are uniformly distributed. Most implementations use an LCG method. The seed can be set using
void srand(unsigned int seed), and is often set to the current system time using seed(time(NULL));.
Thus, it is a reasonable approximation that (rand()/((double)RAND_MAX + 1)) ∼ Uni[0, 1). Notice the
interval is half-open, since zero can be generated, but not 1, and this will be useful in practice.
Then, one has an algorithm for generating a random permutation of a set, such that all permutations are
equally likely. Think of this as shuffling a deck of cards.
void shuffle ( int arr [] int n ) {
for ( int i = n - 1; i > 0; i - -) {
58
 double u = uniformRand (0 , 1); // u in [0 , 1)
// pick one of the " remaining " i positions
int pos = ( int )(( i + 1) * u );
swap ( arr [ i ] , arr [ pos ]);
}
}
This swaps each element of the array and swaps it with some other random element, which does guarantee tht
all permutations are equally likely. Notice that the use of uniformRand(0,1) prevents index n (which doesn’t
exist) from being used.
Here’s a subtle, but incorrect, variation:
void badShuffle ( int arr [] int n ) {
for ( int i = 0; i < n ; i ++) {
double u = uniformRand (0 , 1); // u in [0 , 1)
// pick any position
int pos = ( int )( n * u );
swap ( arr [ i ] , arr [ pos ]);
}
}
Note that this has nn execution paths, but there are n! permutations. Thus, by divisibility, these permutations
can’t venly go into the execution paths, so some of them must be more likely than the others.
Here’s another choice, which is easier but isn’t in-place.
void shuffle ( int [] arr , int n ) {
double * keys = new double [ n ];
for ( int i = 0; i < n ; i ++) {
keys [ i ] = uniformRand (0 , 1);
}
SortUsingKeys ( arr , keys , n );
delete [] keys ; // this ain ’t Java
}
Here, SortUsingKeys() sorts the array arr using the values given in keys as weights. Thus, this method is
O(n log n), unlike the first one, which was O(n). Nonetheless, it is still useful in some cases.
Thus, uniform distributions can be made; but what about other ones? A method called an inverse transform
offers one solution: if one has a continuous distribution F , let U ∼ Uni(0, 1) and define X = F −1 (U ). Since F
is a CDF, then it is necessarily invertible unless it is completely degenerate (in which case we don’t really care
about it anyways). Then, we have that
P (X ≤ x) = P (F −1 (U ) ≤ x) = P (U ≤ F (x)) = F (x).
This last identity is somewhat tricky, but that is the meat of this transform. Thus, X has exactly the distribution
we care about. This can also be used in the discrete case, albeit with some modification.
Example 26.1. Suppose X ∼ Exp(λ). Then, its CDF is F (x) = 1 − e−λx) for x ≥ 0. This can be inverted to
obtain x(u) = − log(1 − u)/λ. Thus, F −1 (u) = − log(1 − u)/λ. Since U ∼ Uni(0, 1), then 1 − U ∼ Uni(0, 1)
as well, which allows for a simplification X = F −1 (U ) = − log(U )/λ. (
This was really nice, but sometimes the closed-form inverse might not exist (particularly in the case of the
normal distribution).
Example 26.2. Suppose X ∼ Poi(3). The discrete case is slightly different: given some U ∼ Uni(0, 1) and a u
generated by that, let x be the smallest value such that F (x) ≥ u. Since the CDF is increasing, then this is
what we want. (
There’s a simple iterative algorithm to do this, which assumes the existence of a PMF p(x) that can be called.
int d i s c r e t e I n v e r s e T r a n s f o r m () {
double u = uniformRand (0 , 1); // again , we don ’t want 1
int x = 0;
double F_so_far = p ( x );
while ( F_so_far < u ) {
59
 x ++;
F_so_far += p ( x );
}
return x ;
}
The last case is that of the normal distribution. Suppose one has a random variable X with a PDF f (x). A
method called rejection filtering can be used, which actually simulates another variable with a different density
function, though it is at least supported on the same set. Here’s the code, and the explanation will follow:
double r ej ect io nF i lt er () {
while ( true ) {
double u = uniformRand (0 , 1);
double y = r a n d o m V a l u e F r o m D i s t r i b u t i o n O f Y ();
if ( u <= f ( y )/( c * g ( y ))) return y ;
}
}
Here, c ≥ f (y)/g(y) for all y so that everything is normalized. Notice that the number of iterations in the loop
is Geo(1/c), which can be unpleasant for large c. Unfortunately, the proof that it works is a bit beyond the
scope of the class, though it is in the book.
Example 26.3. Consider
√ simulating the unit normal (since then any other can be created by scaling), √
which has
2 2
pdf f (z) = e−z /2 / 2π. It can be folded in half, to take the PDF of |Z|, obtaining f (z) = 2e−z /2 / 2π. This
is useful because we already can generate for an exponential distribution using the inverse transform method, so
let Y ∼ Exp(1). Thus, g(y) = e−y for any y ≥ 0. Then,
r r r r
f (x) 2e −(x2 −2x)/2 2e −(x2 −2x+1)/2+1/2 2e −(x−1)2 /2 2e
= e = e = e ≤ = c ≈ 1.32.
g(x) π π π π
2
Thus, f (x)/cg(x) = e−(x−1) /2 , and c is small enough that the loop is mostly efficient. Here,
randomValueFromDistributionOfY() is just − ln(V ), where V ∼ Uni[0, 1), and the method can be imple-
mented.
Y comes from a half-normal distribution, and can be turned into a normal distribution by flipping a coin to
determing the sign of Y to make a normally distributed variable. (
Random number generation can be used to numerically approximate integrals, particularly those without
a closed form. Intuitively, one takes the graph of the function on the interval and throw darts at it, in some
sense, randomy choosing points on the graph. Then, one takes the percentage of the points below the curve and
multiply the area in which the points were chosen (the domain of the integral and the maximum value of the
function). This is known as Monte Carlo integration, named after a casino town in Monaco.
R2
Example 26.4. As a comparison for accuracy, calculate 0 ex dx. Use the following algorithm:
double integrate () {
// number of " darts " to throw
int NUM_POINTS = 1000000;
int numBelowFn = 0;
for ( int i = 0; i < NUM_POINTS ; i ++) {
double x = uniformRand (0 , 1) * 2.0;
double y = uniformRand (0 , 1) * exp (2.0);
if ( y < exp ( x )) numBelowFn ++;
}
return 1.0 * numBelowFn / NUM_POINTS * 2.0 * exp (2.0);
}
Here are some computed values: with 10 darts, the result is about 5.911, and it converges to 6.391 and 6.388 as
the number of darts frows to about 106 . The analytic value is about 6.389, which is pretty nice. Notice that
since the Monte Carlo method is random, sometimes it gets worse when NUM_POINTS is increased. (
Some other methods (e.g. the Runge-Kutta methods) try to distribute the points uniformly, but this requires
knowing some things about the function.
60
 Another application of random numbers is to simulate statistical questions.40 However, one should take care:
bugs in the simulation or even coding approximations have compromised some part of the answer.

27. Some Review for the Final: 6/3/13

“Threee statisticians are out hunting geese. One shoots a goose, but is off by a meter to the
right. The second misses by a meter to the left. The third says, ‘we got him!’ ”
The final exam is approaching; here are some topics that will be worth reviewing. Notice that the final covers
material from the whole class but is weighted in favor of concepts from the second half.
• Counting and combinatorics.
• Conditional probability and Bayes’ theorem.
• Independence and conditional independence.
• Distributions, including the Bernoulli distribution (a coin flip), the binomial distribution (n coin flips),
the multinomial distribution (several different outcomes, rather than two), Poisson, geometric (trials until
first success), negative binomial (trials until r successes), and hypergeometric (how many balls drawn
from an urn are white, without replacement? Note that with replacement, this would just be binomial).
• Distributions, cont.: uniform (equally likely over a range), exponential, and normal.
• Joint distributions and marginal distributions.
• Expectation, conditional expectation, and variance.
• Covariance (not normalized) and correlation (normalized covariance).
• Moment-generating functions. Notice what happens when one multiplies them.
• Inequality-land: Markov’s, Chebyshev’s, Chernoff’s and Jensen’s.
• The Central Limit Theorem, which is the big important part of the theory.
• Machine learning: parameter estimation, whether an estimator is biased and/or consistent. Also recall
the method of moments and the maximum likelihood estimator (MLE).
• Bayesian estimation, including hyperparameters such as the beta distribution and the Laplace prior
(which is a special case of both the beta and the multinomial distributions).
• Relating to machine learning, there are the actual algorithms, such as Naı̈ve Bayes and logistic regression.
When should each be used? It’s also worth remembering overfitting.
• Finally, the basics of utility theory, which were discussed a couple times in class, might come up.
Here is a much shorter list of things which are not on the final exam:
• Bayesian networks are super interesting, but because they weren’t written into the homework, they
won’t be on the final. If you have a burning desire to be tested on these (or to learn more about them
because they’re cool), then take CS 228 or CS 228T.
• Computer generation of probabilities will also not be tested for similar reasons. Relatedly, Monte Carlo
simulation isn’t tested (partly because the exam is closed-computer).
• The general Dirichlet and gamma distributions won’t be tested, though specific examples of them appear
above and will be tested (e.g. the Laplacian).
There is a game called Acey Deucey, in which one takes a standard 52-card deck and flips two cards and then
makes the wager that the next card is strictly between the two cards. Notice that an ace is high, but a two is
low. What is the probability of winning this wager?
Like many problems, the trick is picking the right variables. Let X be the difference between the ranks of the
first two cards. Thus, P (X = 0) = 3/51, because after the first card is chosen, there are three more cards that
can be chosen. More generally, P (X = i) = (13 − i)(2/13)(4/51). The 13 − i term comes because there are
multiple ways to get that difference (e.g. 5 and 7, 6 and 8, etc.). Then, there are two ways to draw the first
number once the specific pair has been specified by the 13 − i term. Then, the remaining card happens four
times out of 51.
If Y is the probability that the third card is between the first two, then the goal is to find
12 12
X X 8(13 − i)
P (X = i)P (Y | X = i) = P (Y | X = i).
i=1 i=1
13 · 51

40A good example of this is the problems on our problem sets. Here Dr. Sahami claims that it was a good thing we didn’t do this to
check our problem sets, though I definitely did. . .
61
Of the 50 cards that remain, there are four cards of each valid rank, and there are i − 1 valid (i.e. winning) ranks.
Then, when everything is plugged in, this ends up as about 0.2761.
There’s another way to solve the problem, which exploits the symmetry of the problem. Clearly, it’s necessary
for all three cards to have different ranks, which happens with probability (48/51)(44/50). Then, it is necessary
to have the third card in the middle of the first two, which is just a computation on the six permutations of
three cards: two of them have the third card in the middle, so the final answer is (48/51)(44/50)(1/3) ≈ 0.2761.
Much simpler.
Another example: in a group of 100 people, let X be the number of days that are the birthdays of exactly
three people in the group. What is E[X]?
When one sees expected value, the first step is to decompose the problem. Let Ai be
the number of people
who have a birthay on day i, so that Ai ∼ Bin(100, 1/365). Then, p = P (Ai = 3) = 100 3
3 (1/365) (364/365) .
97

Thus, let Xi = 1 if Ai = 3 and Xi = 0 otherwise, so that

" 365 # 365 365 365
X X X X
E[X] = E Xi = E[Xi ] = P (Ai = 3) = p = 365p,
i=1 i=1 i=1 i=1
Notice that the events aren’t independent, but in expectation it just doesn’t matter.
Another problem: in a group of 100 people, let Y be the number of distinct birthdays. What is E[Y ]?
Define indicator variables for each day: let Yi = 1 if someone has a birthday on day i and 0 otherwise. Then,
E[Yi ] = P (Yi ) = 1 − P (∼Yi ) = 1 − (364/265)100 , so
" 365 # 365  100 !
X X 364
E[Y ] = E Yi = E[Yi ] = 365 1 − .
i=1 i=1
365
But this can be reformulated into a problem about hash tables.
Another question: suppose X1 , . . . , Xn ∼ Geo(p) are IID random variables. How would one estimate p using
the method of moments? E[Xi ] = 1/p, so p = 1/E[Xi ], and one would estimate
1 1 1 n
p= ≈ = = Pn = p̂.
E[Xi ] m̂i X i=1 Xi
For some retrospective, probability has become incredibly important in computer science. Though computers
are deterministic, they are so complex that probability makes a lot of questions easier to answer. This relates
hugely to understanding data: the data that users leave on the web provides information that can be used for
probabilistic analysis. In some sense, each user is a giant vector with likes, dislikes, written text, etc. Many
large companies exist solely to collect and analyze this sort of data. For example, there was a recent controversy
in which Target was able to predict whether its users were pregnant (for the purposes of targeted41 advertising),
sometimes better than they were. This is kind of scary, but it all ties back to probability and machine learning.
If you want to track how information about you flows, try using different middle intials to see where your junk
mail comes from.
This also applies to one’s credit card purchases, and loan applications: each purchase (or loan application) is
run through an algorithm to determine how likely it is to be legitimate. But in mortgages, all denials are done
by a person, in order to eliminate some sort of discrimination. This has connections to artificial intelligence,
strangely enough.
One more story: the Department of Defense made a neural network which was to identify tanks in images.
It worked perfectly in the lab, but in the field it failed. People had thrown machine learning at the problem
without looking at the dataset, because the pictures with tanks were taken on sunny days and those without
tanks were taken on cloudy days, and this caused problems. Make sure to understand the problem the machine
learning is applied to.

41No pun intended.

62