1

Sample Space and

Probability

Contents

1.1. Sets . . . . . . . . . . p. 3
1 .2 . Probabilistic Models . . . p. 6
1 .3. Conditional Probability p. 18
1 .4. Total Probability Theorem and Bayes' Rule p. 28
1 .5. Independence . . . . . . p. 34
1 .6. Counting . . . . . . . p. 44
1 .7. Summary and Discussion p. 5 1
Problems . . . . . . . . p. 53

1
2 Sample Space and Probability Chap. 1

"Probability" is a very useful concept, but can be interpreted in a number of

ways. As an illustration, consider the following.

A patient is admitted to the hospital and a potentially life-saving drug is

administered. The following dialog takes place between the nurse and a
concerned relative.
RELATIVE: Nurse, what is the probability that the drug will work?
NURSE: I hope it works, we'll know tomorrow.
RELATIVE: Yes, but what is the probability that it will?
NURSE: Each case is different, we have to wait.
RELATIVE: But let's see, out of a hundred patients that are treated under
similar conditions, how many times would you expect it to work?
NURSE (somewhat annoyed) : I told you, every person is different, for some
it works, for some it doesn't.
RELATIVE (insisting) : Then tell me, if you had to bet whether it will work
or not, which side of the bet would you take?
NURSE (cheering up for a moment) : I'd bet it will work.
RELATIVE (somewhat relieved) : OK, now, would you be willing to lose two
dollars if it doesn't work, and gain one dollar if it does?
NURSE (exasperated) : What a sick thought! You are wasting my time!

In this conversation, the relative attempts to use the concept of probability

to discuss an uncertain situation. The nurse's initial response indicates that the
meaning of "probability" is not uniformly shared or understood, and the relative
tries to make it more concrete. The first approach is to define probability in
terms of frequency of occurrence, as a percentage of successes in a moderately
large number of similar situations. Such an interpretation is often natural. For
example, when we say that a perfectly manufactured coin lands on heads "with
probability 50%," we typically mean "roughly half of the time." But the nurse
may not be entirely wrong in refusing to discuss in such terms. What if this
was an experimental drug that was administered for the very first time in this
hospital or in the nurse's experience?
While there are many situations involving uncertainty in which the fre­
quency interpretation is appropriate, there are other situations in which it is
not. Consider, for example, a scholar who asserts that the Iliad and the Odyssey
were composed by the same person, with probability 90%. Such an assertion
conveys some information, but not in terms of frequencies, since the subject is
a one-time event . Rather, it is an expression of the scholar's subjective be­
lief. One might think that subjective beliefs are not interesting, at least from a
mathematical or scientific point of view. On the other hand, people often have
to make choices in the presence of uncertainty, and a systematic way of making
use of their beliefs is a prerequisite for successful, or at least consistent, decision
making.
 Sec. 1 . 1 Sets 3

In fact, the choices and actions of a rational person can reveal a lot about
the inner-held subjective probabilities, even if the person does not make conscious
use of probabilistic reasoning. Indeed, the last part of the earlier dialog was an
attempt to infer the nurse's beliefs in an indirect manner. Since the nurse was
willing to accept a one-for-one bet that the drug would work, we may infer
that the probability of success was judged to be at least 50%. Had the nurse
accepted the last proposed bet (two-for-one) , this would have indicated a success
probability of at least 2/3.
Rather than dwelling further on philosophical issues about the appropriate­
ness of probabilistic reasoning, we will simply take it as a given that the theory
of probability is useful in a broad variety of contexts, including some where the
assumed probabilities only reflect subjective beliefs. There is a large body of
successful applications in science, engineering, medicine, management, etc. , and
on the basis of this empirical evidence, probability theory is an extremely useful
tool.
Our main objective in this book is to develop the art of describing un­
certainty in terms of probabilistic models, as well as the skill of probabilistic
reasoning. The first step, which is the subject of this chapter, is to describe
the generic structure of such models and their basic properties. The models we
consider assign probabilities to collections (sets) of possible outcomes. For this
reason, we must begin with a short review of set theory.

1.1 SETS
Probability makes extensive use of set operations, so let us introduce at the
outset the relevant notation and terminology.
A set is a collection of objects, which are the elements of the set. If S is
a set and x is an element of S, we write x ES. If x is not an element of S, we
write x � S. A set can have no elements, in which case it is called the empty
set, denoted by 0.
Sets can be specified in a variety of ways. If S contains a finite number of
elements, say Xl , X2 , . . . , Xn,
we write it as a list of the elements, in braces:

For example, the set of possible outcomes of a die roll is {I, 2, 3, 4,5, 6 } , and the
set of possible outcomes of a coin toss is {H, T} , where H stands for "heads"
and T stands for "tails."
If S contains infinitely many elements Xl, X2 ,
, which can be enumerated
. . •

in a list (so that there are as many elements as there are positive integers) we
write
X2 ,
S = {x I , . . } , .

and we say that S is countably infinite. For example, the set of even integers
can be written as {O, 2, -2, 4, -4, . } , and is count ably infinite.
. .
4 Sample Space and Probability Chap. 1

Alternatively, we can consider the set of all x that have a certain property
P, and denote it by
{x I x satisfies Pl·
(The symbol "I" is to be read as "such that." ) For example, the set of even
integers can be written as {k I k /2 is integer} . Similarly, the set of all scalars x
in the interval [0, 1] can be written as {x I 0 ::; x � I}. Note that the elements x
of the latter set take a continuous range of values, and cannot be written down
in a list (a proof is sketched in the end-of-chapter problems) ; such a set is said
to be uncountable .
If every element of a set S is also an element of a set T, we say that S
is a subset of T, and we write SeT or T ::J S. If SeT and T C S, the
two sets are equal , and we write S = T. It is also expedient to introduce a
universal set , denoted by f2, which contains all objects that could conceivably
be of interest in a particular context. Having specified the context in terms of a
universal set f2, we only consider sets S that are subsets of f2.

Set Operations

The complement of a set S, with respect to the universe f2, is the set {x E
f2 I x � S} of all elements of f2 that do not belong to S, and is denoted by Sc.
Note that f2C = 0.
The union of two sets S and T is the set of all elements that belong to S
or T (or both), and is denoted by S U T. The intersection of two sets S and T
is the set of all elements that belong to both S and T, and is denoted by S n T.
Thus,
S U T = {x I xE S or xE T},
and
S n T = {x I xES and xE T} .
In some cases, we will have to consider the union or the intersection of several,
even infinitely many sets, defined in the obvious way. For example, if for every
positive integer n, we are given a set Sn , then

U Sn
00

= S1 U S2 U . . . = {x I x ESn for some n} ,

n= 1
and
n Sn
00

= SI n S2 n ... = {x I xESn for all n} .

n=1
Two sets are said to be disjoint if their intersection is empty. More generally,
several sets are said to be disjoint if no two of them have a common element. A
collection of sets is said to be a partition of a set S if the sets in the collection
are disjoint and their union is S.
 1.1 5

If x yare two we use y) to of x

The set of scalars (real numbers) denoted by �: the set (or

. respectively ) .
i.e .. the two-dimensiona1 plane (or three-dimensional space,

...... ......
(or
A o p erat i ons are
1.1.
...-."..,'"-J """''-& to visualize in terms
as in

1.1: of Venn T. (b) is S n

The shaded is S U T. (c) The T C (d)
is the complement of S. (e) The sets S. T. and U are disjoint.
and U form a
The shaded
(f) The sets S, of the set n.

are nr\1r'\C!'O, ...... '£],TlII·'''''-'

of the

Su u Su (Tu U) = (SuT) u U,
Sn(TuU) ==(5nT)U(SnU). 5 u (Tn U) :::: (5UT) n (SuU).
) c ==S. 5n 0.
==

SuO = O. 5nO S.
=

particularly are which

state

x E )c. T hen . x ¢
(Un which 1

1:: Sn. Thus, x b elo n gs to the

To suppose
n, we have x c ompleme n t
 6 SampJe Probability Chap. 1

Sn. and x E nn S�. This

IS by '-£..1""".-00":

argument for is

probabil istic an
,.... ,... ... ,.,.
in ..........
It must
1
that we discuss in thi s
two main ingredients are

• IS set of possible tC()mleS of an

e xp er i me nt .
•

or
probability law must satisfy

1.2: The main of a model.

Every probabi listic model involves an

will one out of �"""""'r�

aU possi b l e outcomes is called t h e oi:::!III:,.IL.JLJl.JI. ...lJI.'I;;i

n. A subset of the sam p le s pac e that is, a collection of possible

the experiment,
denoted ,
Sec. 1 .2 Probabilistic Models 7

outcomes, is called an event. t There is no restriction on what constitutes an

experiment. For example, it could be a single toss of a coin, or three tosses,
or an infinite sequence of tosses. However, it is important to note that in our
formulation of a probabilistic modeL there is only one experiment . So, three
tosses of a coin constitute a single experiment . rather than three experiments.
The sample space of an experiment may consist of a finite or an infinite
number of possible outcomes. Finite sample spaces are conceptually and math­
ematically simpler. Still, sample spaces with an infinite number of elements are
quite common. As an example, consider throwing a dart on a square target and
viewing the point of impact as the outcome.

Choosing an Appropriate Sample Space

Regardless of their number. different elements of the sample space should be

distinct and mutually exclusive , so that when the experiment is carried out
there is a unique outcome. For example, the sample space associated with the
roll of a die cannot contain "l or 3" as a possible outcome and also "l or 4"
as another possible outcome. If it did, we would not be able to assign a unique
outcome when the roll is a 1 .
A given physical situation may be modeled in several different ways, de­
pending on the kind of questions that we are interested in. Generally, the sample
space chosen for a probabilistic model must be collectively exhaustive, in the
sense that no matter what happens in the experiment , we always obtain an out­
come that has been included in the sample space. In addition, the sample space
should have enough detail to distinguish between all outcomes of interest to the
modeler, while avoiding irrelevant details.

Example 1 . 1 . Consider two alternative games, both involving ten successive coin
tosses:
Game 1: We receive $ 1 each time a head comes up.
Game 2: We receive $1 for every coin toss. up to and including the first time
a head comes up. Then. we receive $2 for every coin toss. up to the second
time a head comes up. More generally, the dollar amount per toss is doubled
each time a head comes up.

t Any collection of possible outcomes, including the entire sample space nand
its complement, the empty set 0, may qualify as an event. Strictly speaking, however,
some sets have to be excluded. In particular, when dealing with probabilistic models
involving an uncountably infinite sample space. there are certain unusual subsets for
which one cannot associate meaningful probabilities. This is an intricate technical issue,
involving the mathematics of measure theory. Fortunately, such pathological subsets
do not arise in the problems considered in this text or in practice. and the issue can be
safely ignored.
8 Probabili ty 1

Models

space by means of
a

3 ...

2 ..

1
1 2 3 .:)

1.3: of sample space of an o v '.... or· .......

involving two rolls of a 4-sided die. The p os s ible outcomes are all the ordered pairs
of the form (i. j). w here i is the result of the roll, and j is the r esul t of the
second. These outcomes can be in a 2-dimensional as in the
on the or they can be described by the tree on the right. which
character of the
U\.,oll,-, lUI outcome 1"1'\'� ... 0<c::nr'\nr1c:

to a leaf of the tree and is ass o c iated with the path from the root to
that shaded area on the left is the event {(1,4), (2 4) (3,4), (4,4)}
that the result of the second roll is 4. That same event can be described by the
, ,

set of leaves highlighted on the right. Note al so that every node of the tree can
be identified with an the set of all leaves downstream from that
n o de For the node labeled a 1 can be identified with the e vent
{(l. 1), (1,2). (1,3), (1, 4)} t h at the result of t h e first roB is 1.
.

,In.,,,... ,..,,::.£>
C,,",'lIr-g n with an experiment.
we have on the
complete the ..., ............... �J"' .. "'" model: we now introd uce a law .
Sec. 1.2 Probabilistic Models 9

Intuitively, this specifies the "likelihood" of any outcome, or of any set of possible
outcomes (an event. as we have called it earlier) . More precisely. the probability
law assigns to every event A. a number P(A) , called the probability of A.
satisfying the following axioms.

Probability Axioms
L (Nonnegativity) P(A) > 0, for every event A.
2. (Additivity) If A and B are two disjoint events, then the probability
of their union satisfies

P(A U B) = P(A) + P(B ) .

More generally, if the sample space h as an infinite number o f elements

and Al , A2 , . . . is a sequence of disjoint events, then the probability of
their union satisfies

3. (Normalization) The probability of the entire sample space n is

equal to 1 , that is, p(n) = 1.

In order to visualize a probability law. consider a unit of mass which is

i'spread" over the sample space. Then, P(A) is simply the total mass that was
assigned collectively to the elements of A. In terms of this analogy, the additivity
axiom becomes quite intuitive: the total mass in a sequence of disjoint events is
the sum of their individual masses.
A more concrete interpretation of probabilities is in terms of relative fre­
quencies: a statement such as P(A) = 2/3 often represents a belief that event A
will occur in about two thirds out of a large number of repetitions of the exper­
iment. Such an interpretation, though not always appropriate, can sometimes
facilitate our intuitive understanding. It will be revisited in Chapter 5, in our
study of limit theorems.
There are many natural properties of a probability law. which have not been
included in t he above axioms for t he simple reason that they can be derived
from them. For example, note that the normalization and additivity axioms
imply that
1 = p(n) = p(n U 0) = p(n) + P( 0) = 1 + P(0) .
and this shows that the probability of the empty event is 0:
P(0) = o.
10 Sample Space and Probability Chap. 1

As another example, consider three disjoint events A I , A2 , and A3. We can use
the additivity axiom for two disjoint events repeatedly, to obtain
P(A1 U A2 U A3) = P (A1 U (A 2 U A3) )
= P(At) + P(A2 U A3 )

= P(AI ) + P(A2 ) + P(A3) .

Proceeding similarly, we obtain that the probability of the union of finitely many
disjoint events is always equal to the sum of the probabilities of these events.
More such properties will be considered shortly.

Discrete Models
Here is an illustration of how to construct a probability law starting from some
common sense assumptions about a model.

Example 1 .2. Consider an experiment involving a single coin toss. There are two
possible outcomes, heads (H) and tails (T) . The sample space is n = {H, T}, and
the events are
{H. T}, {H}, {T}, 0.
If the coin is fair, i.e. , if we believe that heads and tails are "equally likely," we
should assign equal probabilities to the two possible outcomes and specify that
P( {H}) = P( {T}) = 0.5. The additivity axiom implies that
P({H, T}) = P({H}) + P({T}) = 1,
which is consistent with the normalization axiom. Thus, the probability law is given
by
P ({H, T}) = 1 , p( {H}) = 0.5. P({T}) = 0.5, P(0) = 0,
and satisfies all three axioms.
Consider another experiment involving three coin tosses. The outcome will
now be a 3-long string of heads or tails. The sample space is
n = {HHH, HHT, HTH, HTT, THH, THT, TTH, TTT}.
We assume that each possible outcome has the same probability of 1 /8. Let us
construct a probability law that satisfies the three axioms. Consider, as an example.
the event
A = {exactly 2 heads occur} = {HHT, HTH, THH}.
Using additivity, the probability of A is the sum of the probabilities of its elements:
P({HHT, HTH. THH}) = P({HHT}) + P({HTH}) + P({THH})
1 1 1
=8 + 8 + 8
-
3
8
Sec. 1 .2 Probabilistic Models 11

Similarly, the probability of any event is equal to 1/8 times the number of possible
outcomes contained in the event. This defines a probability law that satisfies the
three axioms.

By using the additivity axiom and by generalizing the reasoning in the

preceding example� we reach the following conclusion.

Discrete Probability Law

If the sample space consists of a finite number of possible outcomes, then the
probability law is specified by the probabilities of the events that consist of
a single element. In particular, the probability of any event {SI' S2 , . . . , Sn}
is the sum of the probabilities of its elements:

P ( { SI , S2 , . . . , Sn}) = P (SI ) + P (S2) + ... + P ( sn ) .

Note that we are using here the simpler notation P(Si) to denote the prob­
ability of the event {Si}, instead of the more precise P( {Si} ) . This convention
will be used throughout the remainder of the book.
In the special case where the probabilities P(SI ), . . . , P(s ) are all the same
n

( by necessity equal to lin, in view of the normalization axiom ) , we obtain the

following.

Discrete Uniform Probability Law

If the sample space consists of n possible outcomes which are equally likely
( Le., all single-element events have the same probability ) , then the proba­
bility of any event A is given by

number of elements of A
P(A) = .
n

Let us provide a few more examples of sample spaces and probability laws.

Example 1.3. Consider the experiment of rolling a pair of 4-sided dice (cf. Fig.
1 .4). We assume the dice are fair, and we interpret this assumption to mean that
each of the sixteen possible outcomes [pairs (i, j ) , with i. j = 1 , 2. 3, 4] has the same
probability of 1 / 16. To calculate the probability of an event, we must count the
number of elements of the event and divide by 16 (the total number of possible
 1

are some event in way:

sum is = 8/ = 1
= =

IS - -
(

is to )

...·i ............... 1.4: Various events in the of a of 4-sided

and their calculated to the discrete uniform law.
 1.2 Models

probability l arger than 1. Therefore, the probability of any event that consists of a

satisfies the three probability axioms and

a at a given time, and will

b etween 0 and 1 with
will 15

delays is to Jet area.

7/16.
law satisfies the three probability axioms. The event that Romeo and Juliet wi1l
meet is the shaded region in 1 and its probability is calculated to

1.5: The event Al t hat Romeo and Juliet will arrive within 15 minutes
of each other (d. ). h;

lVl = Ilx yl �
{ y) -

and is shaded in the The area of M is 1 minus the area of the two unshaded
....... 1'".-..
..... or1 - (3/4) . (3/4) = 7/1 6. the of is 7/1 6.

t U]engthl! a subset S of [0, 1] is the integral

"nice" sets S, in the usual calculus sense.

imply the whole zero

the normalization axiom.
14 Sample Space and Probability Chap. 1

Properties of Probability Laws

Probability laws have a number of properties, which can be deduced from the
axioms. Some of them are summarized below.

Some Properties of Probability Laws

Consider a probability law, and let A, B, and C be events.
( a) If A c B , then P(A) :s; P(B).
( b ) P(A U B) = P(A) + P(B) - P(A n B) .
( c ) P(A U B) :s; P(A) + P (B).
( d ) P(A U B U C) = P(A) + p(Ae n B) + p(Ae n Be n C).

These properties, and other similar ones, can be visualized and verified
graphically using Venn diagrams, as in Fig. 1.6. Note that property ( c ) can be
generalized as follows:

L P(Ai).
n

P(A I U A2 U .. . U An ) :s;
i =I
To see this, we apply property ( c ) to the sets Al and A2 U .. . U An , to obtain

We also apply property ( c ) to the sets A2 and A3 U . . . U A n , to obtain

We continue similarly, and finally add.

Models and Reality

The framework of probability theory can be used to analyze uncertainty in a
wide variety of physical contexts. Typically, this involves two distinct stages.
( a) In the first stage, we construct a probabilistic model by specifying a prob­
ability law on a suitably defined sample space. There are no hard rules to
guide this step, other than the requirement that the probability law con­
form to the three axioms. Reasonable people may disagree on which model
best represents reality. In many cases, one may even want to use a some­
what "incorrect" model, if it is simpler than the "correct" one or allows for
tractable calculations. This is consistent with common practice in science
1

a involves a
tween accuracy, sinlplicity, tractability. Sometimes, a model is chosen
on of or
statistical Tn."', ..... """,.., nlethods, which will

9.

Visualization and verification of various properties of

Venn If A C then B is the union of the two ....u ,;;" IUlJ,I"

events A and AC n B; see (a).. by the we

+ AC n B ) 2 P(A),

where the inequality follows from the nonnegativity axiom. and verifies prop­
(a).
From (b), we can express the even t s Au Band B as unions of
events;
B= n

the additivity we have

Au = P( n n B)+p(ACn
from the first and terms. we obtain
n B), nrr\l'"\ortv (b). Using also the fact
uB) � P(A) + P(B).

From diagram (c) we see that

1 the event A u Bu C can be
union of three disjoint events:

AUBUC=Au(ACnB)u
as a consequence of the axiom.
16 1

(b) we work within a fully probabilistic AAA'-' .......... �

derive the probabilities

erties. While the
the

IS no room
it is

full of calculation
though:
..... �...".�'-' ... �'-' ....... or ambiguous
......... '.......... � is shown Fig. 1

pr esente d b y L. F. B ert r and in 1889. illustrates the

a model. Consider a circle an d an
inscribed in the circle. What is the that the
than the side of the
The answer here depends on the of "randomly chosen." The two
meth ods illustrated in parts (a) and (b) of the figure lead to contradictory results.
In (a). we take a radius of the su ch as an d we choose a
C on that radius. with all being equally l i ke l y . We then draw the chord
through C that is to AB. From geometry. AB intersects
the at the of AB. so the that the of the chord
IS than the side is 1/2.
In (b), we take a point on the such as the vertex V we draw the
tangent to the circle thr ou gh V, and we draw a line through V that forms a random
1

� with the with all likely. We consider the

chord obtained the intersection of this line with the circle. From plpmpnt!:!
the of is of
between 1r/3 and 2-rr/3. Since 4> t akes values between 0 and 11".
that the length of the chord is greater than the side is 1/3,
Sec. 1 .2 Probabilistic Models 17

A Brief History of Probability

• B.C.E. Games of chance were popular in ancient Greece and Rome, but
no scientific development of the subject took place, possibly because the
number system used by the Greeks did not facilitate algebraic calculations.
The development of probability based on sound scientific analysis had to
await the development of the modern arithmetic system by the Hindus and
the Arabs in the second half of the first millennium, as well as the flood of
scientific ideas generated by the Renaissance.

• 16th century. Girolamo Cardano, a colorful and controversial Italian

mathematician, publishes the first book describing correct methods for cal­
culating probabilities in games of chance involving dice and cards.

• 1 7th century. A correspondence between Fermat and Pascal touches upon

several interesting probability questions and motivates further study in the
field.

• 18th century. Jacob Bernoulli studies repeated coin tossing and introduces
the first law of large numbers, which lays a foundation for linking theoreti­
cal probability concepts and empirical fact. Several mathematicians. such as
Daniel Bernoulli, Leibnitz, Bayes, and Lagrange, make important contribu­
tions to probability theory and its use in analyzing real-world phenomena.
De Moivre introduces the normal distribution and proves the first form of
the central limit theorem.

• 19th century. Laplace publishes an influential book that establishes the

importance of probability as a quantitative field and contains many original
contributions, including a more general version of the central limit theo­
rem. Legendre and Gauss apply probability to astronomical predictions,
using the method of least squares, thus pointing the way to a vast range of
applications. Poisson publishes an influential book with many original con­
tributions, including the Poisson distribution. Chebyshev, and his students
Markov and Lyapunov, study limit theorems and raise the standards of
mathematical rigor in the field. Throughout this period, probability theory
is largely viewed as a natural science, its primary goal being the explanation
of physical phenomena. Consistently with this goal, probabilities are mainly
interpreted as limits of relative frequencies in the context of repeatable ex­
periments.

• 20th century. Relative frequency is abandoned as the conceptual foun­

dation of probability theory in favor of a now universally used axiomatic
system, introduced by Kolmogorov. Similar to other branches of mathe­
matics, the development of probability theory from the axioms relies only
on logical correctness, regardless of its relevance to physical phenomena.
Nonetheless, probability theory is used pervasively in science and engineer­
ing because of its ability to describe and interpret most types of uncertain
phenomena in the real world.
 18 Sample Space and Probability Chap. 1
1.3 CONDITIONAL PROBABILITY

Conditional probability provides us with a way to reason about the outcome

of an experiment, based on partial information. Here are some examples of
situations we have in mind:
(a) In an experiment involving two successive rolls of a die, you are told that
the sum of the two rolls is 9. How likely is it that the first roll was a 6?
(b) In a word guessing game, the first letter of the word is a ·t". What is the
likelihood that the second letter is an "h" ?
(c) How likely is it that a person has a certain disease given that a medical
test was negative?
(d) A spot shows up on a radar screen. How likely is it to correspond to an
aircraft?
In more precise terms, given an experiment, a corresponding sample space,
and a probability law, suppose that we know that the outcome is within some
given event B . We wish to quantify the likelihood that the outcome also belongs
to some other given event A . We thus seek to construct a new probability law
that takes into account the available knowledge: a probability law that for any
event A. specifies the conditional probability of A given B. denoted by
P(A I B).
We would like the conditional probabilities P ( A I B) of different events A
to constitute a legitimate probability law, which satisfies the probability axioms.
The conditional probabilities should also be consistent with our intuition in im­
portant special cases, e.g., when all possible outcomes of the experiment are
equally likely. For example. suppose that all six possible outcomes of a fair die
roll are equally likely. If we are told that the outcome is even, we are left with
only three possible outcomes. namely, 2. 4, and 6. These three outcomes were
equally likely to start with, and so they should remain equally likely given the
additional knowledge that the outcome was even. Thus, it is reasonable to let

P (the outcome is 61 the outcome is even) = �.

This argument suggests that an appropriate definition of conditional probability
when all outcomes are equally likely, is given by

number of elements of A n B
P ( A I B) = .
number of elements of B
Generalizing the argument, we introduce the following definition of condi­
tional probability:
P(A n B)
P(A I B) =
P(B) ,
Sec. 1.3 Conditional Probability 19

where we assume that P (B) > 0; the conditional probability is undefined if the
conditioning event has zero probability. In words, out of the total probability of
the elements of B, P (A I B) is the fraction that is assigned to possible outcomes
that also belong to A.

Conditional Probabilities Specify a Probability Law

For a fixed event B, it can be verified that the conditional probabilities P(A I B)
form a legitimate probability law that satisfies the three axioms. Indeed, non­
negativity is clear. FUrthermore,

P (O n B) P (B)
P (O I B) = = = 1,
P(B) P(B)

and the normalization axiom is also satisfied. To verify the additivity axiom, we
write for any two disjoint events Al and A2,

P ( ( AI A2 ) n B )
A2 I B) =
u
P (A 1 u
P (B)
P ( ( A1 n B) u (A2 n B))
P (B)
P(AI n B) + P (A2 n B)
P (B)
P (A 1
= +
P (B) P (B)
= P(AI I B) + P(A21 B),

where for the third equality, we used the fact that Al n B and A2 n B are
disjoint sets, and the additivity axiom for the (unconditional) probability law.
The argument for a countable collection of disjoint sets is similar.
Since conditional probabilities constitute a legitimate probability law, all
general properties of probability laws remain valid. For example, a fact such as
P (A U C) :S P (A) + P (C) translates to the new fact

P(A U C I B) :S P(A I B) + P(C I B).

Let us also note that since we have P(B I B) = P (B)/P(B) = 1, all of the con­
ditional probability is concentrated on B. Thus, we might as well discard all
possible outcomes outside B and treat the conditional probabilities as a proba­
bility law defined on the new universe B.
Let us summarize the conclusions reached so far.
20 Sample Space and Probability Chap. 1

Properties of Conditional Probability

• The conditional probability of an event A, given an event B with
P (B) > 0, is defined by
P(A n B)
P (A I B) = ,
P (B)
and specifies a new ( conditional ) probability law on the same sample
space n. In particular, all properties of probability laws remain valid
for conditional probability laws.
• Conditional probabilities can also be viewed as a probability law on a
new universe B , because all of the conditional probability is concen­
trated on B.
• If the possible outcomes are finitely many and equally likely, then

number of elements of A n B
P(A I B) = .
number of elements of B

Example 1 .6. We toss a fair coin three successive times. We wish to find the
conditional probability P(A I B) when A and B are the events

A = {more heads than tails come up} , B = { 1st toss is a head } .

The sample space consists of eight sequences.

n = {HHH. HHT. HTH, HTT, THH, THT. TTH, TTT}.

which we assume to be equally likely. The event B consists of the four elements
H H H, H HT. HT H. HTT, so its probability is

P(B) = �.
The event A n B consists of the three elements H H H. H HT. HT H, so its proba­
bility is

P(A n B) = �.
Thus, the conditional probability P(A I B) is
P(A n B) 3/8 �
P(A I B) = P(B) = =
4/8 4'
Because all possible outcomes are equally likely here, we can also compute P(A I B)
using a shortcut. We can bypass the calculation of P(B) and P(An B), and simply
 1

A is3)
eleInents of B(which is4) saIne result3/4.

1 A IS we assume

possible outcomes are equally likely. Let and Y be the result of the 1 st and the
2nd roll: respectively. We wish to determine the conditional probability P(A I B),
where
A= { Y)= = {
a ndm takes each of the values l. 2. 3�
we can
and P(B)
by 16. Alternative]y, we can directly d ivide the number of elenlents of
see 1

1 .8: Sample space of an involving two rolls of a 4-sided die.

1 . The event B = { Y) = consists of the
set. The set A= { Y) = m} with B two AIDrnDnrc
= = = if Tn = 1. we have
{
if m 3 or m 4. one element if m 2, and no elemen t

2/5. if m = 3 or m = 4.
p( {max(X. Y) = m} I B) = 1 ifm = 2.
O. if m =-= 1.

it C. and an .•.•."""'£A,

team, caB it N, are asked to separately design a n ew product within a month. From
past experience we know that:
( a ) The probability that team C i s successful is23
/ .
(b) The probability that teanl N is is 12
/ .
( c) at one teanl is sw:ceSSI is
22 Sample Space and Probability Chap. 1

Assuming that exactly one successful design is produced, what is the probability
that it was designed by team N?
There are four possible outcomes here, corresponding to the four combinations
of success and failure of the two teams:
SS: both succeed, FF: both fail,
SF: C succeeds, N fails, FS: C fails, N succeeds.
We were given that the probabilities of these outcomes satisfy

P(SS) + P(SF) = 32 ' P(SS) + P(FS) = 21 ' P(SS) + P(SF) + P(FS) = �.

From these relations, together with the normalization equation

P(SS) + P(SF) + P(FS) + P(FF) = 1 ,

we can obtain the probabilities o f individual outcomes:
5
P(SS) = 12 '
P(SF) = 41 ' P(FS) = 121 ' P(FF) = 41 '
The desired conditional probability is
1
P (FS I {SF, FS}) =
12
1 1
4 + 12

Using Conditional Probability for Modeling

When constructing probabilistic models for experiments that have a sequential
character, it is often natural and convenient to first specify conditional prob­
abilities and then use them to determine unconditional probabilities. The rule
p(AnB) = P(B)P(A I B), which is a restatement of the definition of conditional
probability, is often helpful in this process.

Example 1 .9. Radar Detection. If an aircraft is present in a certain area, a

radar detects it and generates an alarm signal with probability 0.99. If an aircraft is
not present. the radar generates a ( false ) alarm, with probability 0. 10. We assume
that an aircraft is present with probability 0.05. What is the probability of no
aircraft presence and a false alarm? What is the probability of aircraft presence
and no detection?
A sequential representation of the experiment is appropriate here, as shown
in Fig. 1 .9. Let A and B be the events

A = { an aircraft is present } ,
B = {the radar generates an alarm } ,
 1 Probability

is n o t }.
B(' ;;;;;;; { the radar does not generate an alarm} .
The given probabilities are recorded along t he corresponding branches o f the tree de-
scribing t he sanlple s pace , 8.::) shown in 1 . 9 . Each possible o u tcome corresponds
to a leaf of the to of
UAJ..,1J v l,U. '''., '-& with root to the ..
I" A .· ..,. '� ...... ..... rI
f'

desired probabilit ies are

P ( not present , fal se al ar m ) = n B) = P ( AC ) P ( B l A C ) = 0.95 · 0. 10 = 0.095,

P ( p res en t , no d e t e ctio n ) = n Be ) = P ( A ) P ( BC I A) = 0.05 · 0.01 ;;;;;;; 0.0005.

of the for t he rad a r detection

Extending the preced ing example, for calculati ng

a of

set up t r ee
so an event is with a
v ie w the occurrence of the event as a sequence of stepsl namely, the
of leaf.

(b)
tree.

(c ) We obtai n the of a by multiplyi ng probabilities

along ,.. .... �• .,.,c>C:- r.... ,{,�,n','lJL ... ... � pa th of the t ree .
24 1

In mathematical terms, we are dealing with an event A which occurs if and

only if one of events A I l . . . An has occurred ,
� A A l n A2 n ! =

. . . n An . A is as an occurrence of A I .

i t is
occurrence by the
as a pat h n b r an ch e s ,
corresponding to t he events A I , . . . , A n .
occurrence of of A3 ,
probability of A is given by the
fo l lowi n g r u l e ( see also Fig. 1 . 10) .

we

The multiplication rule can be verified by writing

( ni= l )= t ) . ----

1 . 10 : Visualization of t he m u l t i plication rule . The i nterse c t ion event

A = Al n n . . . n A n i� a.s�oc iated w i t h a on a t ree that
desc ri bes t h e We associate t he branches of t h i s w i t h t h e events
. . . . . A n . and we record next to

co rres ponds
probabi l ities.
The fi nal node t he path to the i n te rsect ion event A! and
its is obtai ned m u l t i pl y i n g t he con d i t ional recorded
along the branches of the path

1 n A2 n · · · n ) ;;;; P ( A d P ( A 2 1 A d · · · n n · . · n An - d·

N ote any i ntermed iate n ode alon g t he p a t h also to some i n te r-

sect ion event and its is
con d i tional probab i l ities u p to that node. For exampl e , t he e vent Al n A2 n A3
to the node s hown i n figure . and i ts is

n
Sec. 1.3 Conditional Probability 25

and by using the definition of conditional probability to rewrite the right-hand

side above as

For the case of just two events, A l and A2 , the multiplication rule is simply the
definition of conditional probability.

Example 1 . 10. Three cards are drawn from an ordinary 52-card deck without
replacement (drawn cards are not placed back in the deck). We wish to find the
probability that none of the three cards is a heart. We assume that at each step,
each one of the remaining cards is equally likely to be picked . By symmetry, this
implies that every triplet of cards is equally likely to be drawn. A cumbersome
approach, which we will not use, is to count the number of all card triplets that
do not include a heart, and divide it with the number of all possible card triplets.
Instead, we use a sequential description of the experiment in conjunction with the
multiplication rule (cf. Fig. 1 . 1 1 ) .
Define the events

Ai = {the ith card is not a heart} . i = 1 . 2. 3.

We will calculate P(A1 n A2 n A3), the probability that none of the three cards is
a heart, using the multiplication rule

We have
P(Ad = 39 '
52
since there are 39 cards that are not hearts in the 52-card deck. Given that the
first card is not a heart, we are left with 51 cards. 38 of which are not hearts, and

Finally, given that the first two cards drawn are not hearts. there are 37 cards which
are not hearts in the remaining 50-card deck. and

These probabilities are recorded along the corresponding branches of the tree de­
scribing the sample space. as shown in Fig. 1 . 1 1 . The desired probability is now
obtained by mUltiplying the probabilities recorded along the corresponding path of
the tree:
P(A1 n A2 n A3) = 39 38 37 .
_ . - . -

52 51 50
 1

1 . 1 1 : Sequential descri pt ion

3-card se-
1. 10.

Note that o nce t he probabi l it ies are recorded along t he tree1 t he probabi l i ty
events c an be '.H A " & & A'''''' 4

( 1 st is not a heart and 2nd is a heart) = 39 . 1 3 ,

13
and 2nd are not 3rd is a = _ . _ m _

52 51 50

consist i ng of 4 grad uate and

4
a to Inean that given
assign tnent of SOlne students t o certain slots , a ny of the remai n i ng students is equall y
likely to be to any of the We then

1 2 are

1, and 3 are different groups } ,

1.

win

We have
P (A1 ) = �'
15

since there are 1 2 student slots i n

are 1 5 slots

=
8
)
14 '
 1 .3 27

1 . 1 2 : Sequential
tion of the i n the stu-
dent problem of Example 1 . 1 1 .

are 8
there are 1 4

4
) = 13 '

si n ce are 4 student s lots i n other than t hose of

are 1 3 s lots, 1, 3. desi red
is

is by multiply i n g t he '"''V .... ....... .. ... ..... ' probabilities along

" . 'V , corresponding
p at h of t he t ree in Fig. 1 . 1 2.

1. Monty Problem. This i s a much puzzle�

based on an o ld A meri c an game show . are told that a pr i z e is equaUy likely to
beh ind any one to one
A you one of mak i ng sure
that the p rize is not behind i t . At t h is poi nt, you can st ick to you r i ni t i al choice ,
o r switch t o t he other u nopened door. You win t he prize i f it lies behind your fi nal
v. . .... ..... .... of a the

(a ) S ti ck to your initial choice .

(b) Switch to the other u nopened door.

(c) You first point to door 1 . I f do or 2 is opened , you do not switch . I f door 3 is
opened , you

Which is the best strategy? To answer t he questio n , let us calculate the probability

no wi]] .... "'.-........ ,.......

you win or not , and the probability is 1J"" _ u. �.&.:J'V is
equal ly like l y to be behind door .

2/3) ,
the of "-" '- .l A ... ....... t he
 28 Sample Space and Probability Chap. 1

another door without a prize has been opened for you, you will get to the winning
door once you switch. Thus. the probability of winning is now 2/3, so (b) is a better
strategy than (a) .
Consider now strategy (c ) . Under this strategy, there is insufficient informa­
tion for determining the probability of winning. The answer depends on the way
that your friend chooses which door to open. Let us consider two possibilities.
Suppose that if the prize is behind door 1 , your friend always chooses to open
door 2. (If the prize is behind door 2 or 3, your friend has no choice.) If the prize
is behind door 1 . your friend opens door 2, you do not switch, and you win. If the
prize is behind door 2, your friend opens door 3, you switch. and you win. If the
prize is behind door 3. your friend opens door 2. you do not switch, and you lose .
Thus, the probability of winning is 2/3. so strategy (c) in this case is as good as
strategy (b) .
Suppose now that if the prize is behind door 1 . your friend is equally likely to
open either door 2 or 3. If the prize is behind door 1 (probability 1/3). and if your
friend opens door 2 (probability 1/2), you do not switch and you win (probability
1/6). But if your friend opens door 3, you switch and you lose. If the prize is behind
door 2, your friend opens door 3. you switch, and you win (probability 1 /3 ) . If the
prize is behind door 3, your friend opens door 2, you do not switch and you lose.
Thus. the probability of winning is 1 /6 + 1/3 = 1/2, so strategy (c) in this case is
inferior to strategy (b).

1 .4 TOTAL PROBABILITY THEOREM AND BAYES' RULE

In this section, we explore some applications of conditional probability. We start

with the following theorem. which is often useful for computing the probabilities
of various events. using a "divide-and-conquer" approach.

Total Probability Theorem

Let AI , . . . , An be disjoint events that form a partition of the sample space
(each possible outcome is included in exactly one of the events A I , . . . , An)
and assume that P (Ai ) > 0, for all i. Then, for any event B, we have

P ( B) = P ( A 1 n B) + . . . + P ( A n n B)
= P(AdP(B I A d + . . . + P ( An ) P ( B I An ) .

The theorem is visualized and proved i n Fig. 1 . 13. Intuitively, we are par­
titioning the sample space into a number of scenarios (events) Ai . Then, the
probability that B occurs is a weighted average of its conditional probability
under each scenario, where each scenario is weighted according to its (uncondi­
tional) probability. One of the uses of the theorem is to compute the probability
of various events B for which the conditional probabilities P(B I Ai) are known or
 1.4 29

to The key is to choose appropriately the A I . . . . . An . and

choice is by problem structure. Here are some exal nples .

1.13: Visualization and verification of the total theorem . T he

events , . . . . form a of t he space. so t he event B c a n be
decomposed i nto the disj oint union of i ts i ntersections n B with the �ets

B= I n B) u . . . u ( A rl n

Usi n g t he a d d it i v i t y ax iom. it follows that

P ( B ) = P( A I n B) + . . . + n B).
t he d e fi n i t i o n cond itional

P ( Ai n B) = ).
t he

B ) = P ( A t )P ( B I A d + . . . + BI ).
For a n alternat ive view. consider an equivalent seq uential model . as shown
on the right . The probabi l i ty of the leaf A i n B is the prod uct ) P ( B I ) of
the probabilities along path lead i ng to t hat The event B consists of t h e
th ree leaves and is o b t a i n ed by the i r

enter a
chess tournament w here your of w i n n i ng
(cal l type 1 ) . 0 . 4 a

the players (call them type 1 and against t he remai n i ng q uarter of the p l ayers
( call them type 3 ) . You play a ganle agai nst a randomly chosen opponent . What
is
z.

P (A t ) = 0 . 5 , P(A;J) = 0 . 25 .

A lso . let be t he event of w i n n i ng . We have

I A d = 0. 3 . B I A2 ) = 0.4. P(B I ) = 0.5.

30 Sample Space and Probability Chap. 1

Thus, by the total probability theorem, the probability of winning is

P C B) = P (At )P(B I A d + P (A2)P(B I A2) + P (A3)P(B I A3)

= 0.5 . 0. 3 + 0.25 . 0.4 + 0.25 . 0.5
= 0.375.

E xample 1 . 14. You roll a fair four-sided die. If the result is 1 or 2, you roll once
more but otherwise, you stop. What is the probability that the sum total of your
rolls is at least 4?
Let Ai be the event that the result of first roll is i, and note that peAt ) = 1 / 4
for each i. Let B be the event that the sum total is at least 4. G iven the event A I ,
the sum total will b e at least 4 if the second roll results i n 3 or 4, which happens
with probability 1 /2. Similarly, given the event A2, the sum total will be at least
4 i f the second roll results in 2, 3, or 4, which happens with probability 3/4. Also,
given the event A3 , you stop and the sum total remains below 4 . Therefore,

1
P ( B I Ad = 2 '

By the total probability theorem.

- · 0 + -1 . 1 = 9 .
1 1 1 3 1
P CB) = -. -+ . -+
4 2 4 4 4 4 16

The total probability theorem can be applied repeatedly to calculate proba­

bilities in experiments that have a sequential character, as shown in the following
example.

Example 1 .15. Alice is taking a probability class and at the end of each week
she can be either up-to-date or she may have fallen behind. If she is up-to-date in
a given week, the probability that she will be up-to-date ( or behind ) in the next
week is 0.8 ( or 0.2, respectively ) . If she is behind in a given week, the probability
that she will be up-to-date (or behind ) in the next week is 0.4 (or 0.6, respectively ) .
Alice is ( by default ) up-to-date when she starts the class. What is the probability
that she is up-to-date after three weeks?
Let Ui and Bi be the events that Alice is up-to-date or behind, respectively,
after i weeks. According to the total probability theorem, the desired probability
P(U3 ) is given by

The probabilities P(U2) and P (B2) can also be calculated using the total probability
theorem:

P ( U2 ) = P (UdP(U2 1 Ud + P ( B dP(U2 1 Bd = P(Ud · 0. 8 + P(Bd · 0. 4,

P(B2 ) = P (Ut )P ( B2 1 ud + P(Bt }P(B2 1 Bd = P(Ut } · 0.2 + P(Bt ) · 0.6.
Sec. 1 . 4 Total Probability Theorem and Bayes ' Rule 31

Finally, since Alice starts her class up-to-date, we have

P(Ud = 0.8, P(B1 ) = 0.2.
We can now combine the preceding three equations to obtain
P(U2) = 0.8 · 0.8 + 0.2 · 0.4 = 0.72,
P(B2 ) = 0.8 · 0.2 + 0.2 · 0.6 = 0.28,
and by using the above probabilities in the formula for P(U3):
P(U3 ) = 0.72 · 0.8 + 0.28 · 0.4 = 0.688.
Note that we could have calculated the desired probability P(U3) by con­
structing a tree description of the experiment, then calculating the probability of
every element of U3 using the multiplication rule on the tree, and adding. However.
there are cases where the calculation based on the total probability theorem is more
convenient. For example. suppose we are interested in the probability P(U20) that
Alice is up-to-date after 20 weeks. Calculating this probability using the multipli­
cation rule is very cumbersome. because the tree representing the experiment is 20
stages deep and has 220 leaves. On the other hand, with a computer, a sequential
calculation using the total probability formulas
P(Uz+d = P(Ud · 0.8 + P(Bi ) · 0.4.
P(BHd = P(Ui ) · 0.2 + P(Bi ) · 0.6.
and the initial conditions P(U1 ) = 0.8. P(B1 ) = 0.2. is very simple.

Inference and Bayes' Rule

The total probability theorem is often used in conjunction with the following
celebrated theorem, which relates conditional probabilities of the form P(A I B)
with conditional probabilities of the form P(B I A), in which the order of the
conditioning is reversed.

Bayes' Rule
Let A I , A2 , . . . , An be disjoint events that form a partition of the sample
space, and assume that P (Ai ) > 0, for all i. Then, for any event B such
that P(B) > 0, we have

P(Ai)P(B 1 Ai )
P(A . I B) =
t
P(B)
P (Ai)P(B 1 Ai)
- P(AI )P(B 1 A I ) + . . . + P(An)P(B 1 An ) '
 1

1 . 14: A n of t h e i n ference context that is im plicit i n

r u l e . We observe a s h ad e i n a perso n 's ( t h i s is event t he "effect" ) and
we wan t to esti m ate t he l i ke l i h o o d o f t h ree m u t u al ly exclusive and col lectively
exh austive p otential causes : cause 1 ( event A I ) is t h at there is a m al ignant t u mo r ,
c au se 2 ) is that t here is a tumor and cause 3 ( eve n t
corresPoll1ds to reasons ot her than a t u m o r We ass u m e that we k n ow t he
I

3. t h at w e see a shade
.

probabili ties P ( A i ) and P ( B I ) , i = 1 . 2.

B occurs) . r u le the of the var ious causes as

P ( Ai I B ) = -----------------
peAl )P ( B I Ai ) -3-) ! i = 1 . 2, 3.
P(A1 ) A

For an a l ter n at ive consider an mode l , as s h ow n

on t he rig h t T h e probab i l ity I B) o f a malignant t u m or is the p r o babi l ity
the fi rst h ig h l i ghted leaf. which is by t h e total p ro b a b i lity
.

of P(A 1 n B ) . d ivided
of t h e leaves. w hich is P(B).

To no t e by of
we

(B I Ad =
( A i I B ) P { B) .

This y ields the first follows from

tot al probability
is used are a . ... ...... .. .L ...J
. ..... ...

that Inay result in a certain "effect . " We observe the effect , and we to infer

a
effect B has been observed , we
to
P (Ai I B )
cause
as the of event Ai given the information, to
be d istinguished from P{ A d 1
which we call t he
Sec. 1.4 Total Probability Theorem and Bayes ' Rule 33

Example 1 .16. Let us return to the radar detection problem of Example 1.9 and
Fig. 1 .9. Let
A = { an aircraft is present } ,
B { the radar generates an alarm } .
We are given that

peA) = 0.05, PCB I A ) = 0.99,

Applying Bayes ' rule, with Al = A and A 2 AC , we obtain

P (aircraft present I alarm) = p e A I B)

_ P(A)P(B I A )
-
PCB)
P(A)P(B I A)
P (A)P(B I A ) + P(k- )P(B l Ac)
0.05 · 0.99
0.05 · 0.99 + 0.95 · 0 . 1
:::::: 0.3426.

Example 1 . 11. Let us return to the chess problem of Example 1 . 1 3. Here. At is

the event of getting an opponent of type i, and

p e A l ) = 0.5 , P(A3 ) = 0.25.

Also, B is the event of winning, and

PCB I Ad = 0.3. PCB I A3 ) = 0.5.

Suppose that you win. What is the probability peA l I B) that you had an opponent
of type I?
Using Bayes' rule, we have

P(Al )P(B I A d
peA l I B) =
P(At )P(B I A d + P (A2 )P(B I A 2 ) + P ( A3 )P{B I A3)
0.5 · 0.3
0.5 . 0.3 + 0.25 . 0.4 + 0.25 . 0.5
- �������--���

= 0.4.

Example 1 . 1 8. The False-Positive Puzzle. A test for a certain rare disease is

assumed to be correct 95% of the time: if a person has the disease, the test results
are positive with probability 0.95, and if the person does not have the disease,
the test results are negative with probability 0.95. A random person drawn from
 34 Sample Space and Probability Chap. 1

a certain population has probability 0.001 of having the disease. Given that the
person just tested positive, what is the probability of having the disease?
If A is the event that the person has the disease, and B is the event that the
test results are positive, the desired probability. P{A I B), is

P(A I B) = P{A)P(B I A)
P(A)P(B I A) + P{Ac )P{B l Ac )
0.00 1 · 0.95
0.001 · 0.95 + 0.999 . 0.05
= 0.0187.

Note that even though the test was assumed to be fairly accurate, a person who has
tested positive is still very unlikely (less than 2%) to have the disease. According
to The Economist (February 20th. 1 999). 80% of those questioned at a leading
American hospital substantially missed the correct answer to a question of this
type; most qf them thought that the probability that the person has the disease
is 0.95!

1.5 INDEPENDENCE

We have introduced the conditional probability P(A I B) to capture the partial

information that event B provides about event A. An interesting and important
special case arises when the occurrence of B provides no such information and
does not alter the probability that A has occurred, i.e. ,

P(A I B) = P(A).

When the above equality holds. we say that A is independent of B. Note that
by the definition P(A I B) = P(A n B)/P(B) , this is equivalent to

P(A n B) = P(A)P (B) .

We adopt this latter relation as the definition of independence because it can be

used even when P(B) = 0, in which case P(A I B) is undefined. The symmetry
of this relation also implies that independence is a symmetric property; that is,
if A is independent of B, then B is independent of A, and we can unambiguously
say that A and B are independent events.
Independence is often easy to grasp intuitively. For example, if the occur­
rence of two events is governed by distinct and noninteracting physical processes,
such events will turn out to be independent . On the other hand, independence
is not easily visualized in terms of the sample space. A common first thought
is that two events are independent if they are disjoint, but in fact the oppo­
site is true: two disjoint events A and B with P(A) > 0 and P (B) > 0 are
never independent, since their intersection A n B is empty and has probability O.
Sec. 1.5 Independence 35

For example, an event A and its complement Ac are not independent unless
P (A) = 0 or P (A) = 1 ] , since knowledge that A has occurred provides precise
[
information about whether Ac has occurred.

Example 1 . 19. Consider an experiment involving two successive rolls of a 4-sided

die in which all 16 possible outcomes are equally likely and have probability 1 / 16.
(a) Are the events

Ai = { 1st roll results in i } , Bj = {2nd roll results i n j } ,

independent? We have

(
P(Ai n Bj ) = P the outcome of the two rolls is (i, j)) = �
1 '
number of elements of At 4
P ( Ai ) =
total number of possible outcomes 16 '
number of elements of B) 4
P(B). ) =
total number of possible outcomes 16 ·

We observe that P(Ai n B) ) = P (Ai)P(Bj ) , and the independence of At and

Bj is verified. Thus, our choice of the discrete uniform probability law implies
the independence of the two rolls.
(b) Are the events

A = { 1st roll is a I } , B = {sum of the two rolls is a 5},

independent? The answer here is not quite obvious. We have

(
P(A n B) = P the result of the two rolls is ( 1 ,4) = ) �'
1
and also

..! .
number of elements of A
P( A) = _
total number of possible outcomes 16

The event B consists of the outcomes ( l ,4 ) , (2,3), (3,2) , and (4, 1 ) . and

number of elements of B 4
P(B) =
total number of possible outcomes 16 ·

Thus, we see that P(A n B) P(A)P(B) , and the events A and B are
independent.
( c) Are the events

A = {maximum of the two rolls is 2 } , B = {minimum of the two rolls is 2 } ,

36 Sample Space and Probability Chap. 1

independent? Intuitively. the answer is "no" because the minimum of the

minimum is 2, the maximum cannot be

and B are not independent, we calculate
1.
two rolls conveys some information about the maximum. For example, if the
More precisely, to verify that A

P( A n B) = P ( the result of the two rolls is (2,2) ) =

116 ,
and also

P( A) =
number of elements of A
total number of possible outcomes
=
16
�.

PCB) =
number of elements of B
total number of outcomes 156 '
We have P(A)P(B) =
B are not independent.
15/(16)2 , so that P(A n B) =I- P(A)P(B). and A and

We finally note that, as mentioned earlier, if A and B are independent, the

occurrence of B does not provide any new information on the probability of A
occurring. It is then intuitive that the non-occurrence o f B should also provide
no information on the probability of A. Indeed. it can be verified that if A and
B are independent, the same holds true for A and Be (see the end-of-chapter
problems) .

Conditional Independence
We noted earlier that the conditional probabilities of events, conditioned on
a particular event. form a legitimate probability law. We can thus talk about
independence of various events with respect to this conditional law. In particular,
given an event G. the events A and B are called conditionally independent
if
P ( A n B I G) p e A I G)P ( B I G) .
To derive an alternative characterization of conditional independence, we use the
definition of the conditional probability and the multiplication rule, to write
p ( A n B n G)
P ( A n B I G) =
P (G)
P (G)P(B I G)P ( A I B n G)
P (G)
= P ( B I G)P(A I B n G).

We now compare the preceding two expressions. and after eliminating the com­
mon factor P ( B I G) , assumed nonzero . we see that conditional independence is
the same as the condition

P ( A I B n G) = peA I G) .
Sec. 1.5 Independence 37

In words, this relation states that if C is known to have occurred, the additional
knowledge that B also occurred does not change the probability of A.
Interestingly, independence of two events A and B with respect to the
unconditional probability law. does not imply conditional independence, and
vice versa, as illustrated by the next two examples.

Example 1 .20. Consider two independent fair coin tosses, in which all four possible
outcomes are equally likely. Let
HI = { 1st toss is a head } ,
H2 = {2nd toss is a head } ,
D = {the two tosses have different results} .
The events HI and H2 are (unconditionally) independent. But
1
P(H 1 1 D) = 2 ' P (HI n H2 1 D) = 0,

so that P (HI n H2 1 D) =f P (H1 I D)P(H2 1 D) , and HI , H2 are not conditionally

independent.
This example can be generalized. For any probabilistic model, let A and B be
independent events, and let C be an event such that P( C ) > 0, P (A I C) > 0, and
P(B I C) > 0, while A n B n C is empty. Then, A and B cannot be conditionally
independent (given C) since P(A n B I C) 0 while P(A I C) P(B I C) > O.
=

Example 1 . 2 1 . There are two coins, a blue and a red one. We choose one of
the two at random, each being chosen with probability 1 /2, and proceed with two
independent tosses. The coins are biased: with the blue coin, the probability of
heads in any given toss is 0.99, whereas for the red coin it is O.O l .
Let B b e the event that the blue coin was selected. Let also Hi b e the event
that the ith toss resulted in heads. Given the choice of a coin, the events Hl and
H2 are independent. because of our assumption of independent tosses. Thus,
P (H1 n H2 1 B) = P (Hl I B)P(H2 1 B) = 0.99 . 0.99.
On the other hand, the events HI and H2 are not independent. Intuitively, if we
are told that the first toss resulted in heads, this leads us to suspect that the blue
coin was selected, in which case, we expect the second toss to also result in heads.
Mathematically, we use the total probability theorem to obtain

P(HI ) = P(B)P(H1 I B) + P (BC)P(H1 I BC) = � · 0.99 + � · 0.01 = �,

as should be expected from symmetry considerations. Similarly, we have P(H2) =
1/2 . Now notice that

P(H1 n H2 ) = P (B)P(HI n H2 1 B) + P (BC)P(H1 n H2 1 BC)

= 2 · 0.99 · 0.99 + 2 · 0.01 · 0.01 "2 '
1 1 1
�
38 Sample Space and Probability Chap. 1

Thus, P(H1 n H2) =f. P (H1 )P(H2 ) , and the events Hl and H2 are dependent, even
though they are conditionally independent given B.

We now summarize.

Independence
• Two events A and B are said to be independent if

P {A n B) = P (A)P{B).

If in addition, P {B) > 0, independence is equivalent to the condition

P {A I B) = P (A) .

• If A and B are independent, so are A and Be.

• Two events A and B are said to be conditionally independent,
given another event e with P ( e) > 0, if

p (A n B I e) = P (A l e) p ( B I e) .

If i n addition, P ( B n e) > 0, conditional independence is equivalent

to the condition
P ( A I B n e) P (A I e).
=

• Independence does not imply eonditional independence, and vice versa.

Independence of a Collection of Events

The definition of independence can be extended to multiple events.

Definition of Independence of Several Events

We say that the events At , A2 , , An are independent if
. • •

for every subset S of {1, 2, . . , n}.

.
Sec. 1.5 Independence 39

For the case of three events, AI , A2 , and A3, independence amounts to

satisfying the four conditions
P (A1 n A2 ) = P (A I ) P (A2 ) ,
P ( A 1 n A3) = P(At ) P(A3 ) ,
P (A2 n A3) = P (A2 ) P (A3) ,
P(A1 n A2 n A3) = P (A1 ) P (A2 ) P (A3 ) .
The first three conditions simply assert that any two events are independent,
a property known as pairwise independence. But the fourth condition is
also important and does not follow from the first three. Conversely, the fourth
condition does not imply the first three; see the two examples that follow.

Example 1.22. Pairwise Independence does not Imply Independence.

Consider two independent fair coin tosses, and the following events:
HI = { 1st toss is a head } ,
H2 = { 2nd toss is a head } ,
D = { the two tosses have different results } .
The events HI and H2 are independent, by definition. To see that HI and D are
independent, we note that
P(HI n D) = 1 / 4 !
P(D I H1 ) = = P(D).
1 /2
=

P(HI ) 2
Similarly, H2 and D are independent. On the other hand, we have
P ( H. ) P ( H2 ) P ( D ) ,
1 1 1
P(HI n H2 n D ) = 0 =1= .
2 2.2 =

and these three events are not independent.

Example 1 .23. The Equality P(A 1 n A2 n A3) = P(A t ) P(A2 ) P(A3) is not
Enough for Independence. Consider two independent rolls of a fair six-sided
die, and the following events:
A = { lst roll is 1 , 2, or 3 } ,
B = { lst roll is 3, 4, or 5} ,
C = { the sum of the two rolls is 9 } .
We have
1 11
P(A n B) = 6 =1= .
2 2 = P(A)P(B) ,
= P(A)P(C) ,
1 1 4
P(A n C) = 36 =1= 2 . 36

P(B n C) =
1 1 4
= P( B )P (C).
1 2 =1=
36
2 .
 1

the .....",,,, ,,., ,,,,,, no two

events are

n n

events A I ,

1 U n u

or
n ) t U );
see

1 . 24 . The number next to each l i n k

i n d icates the Series and connections
of three
Sec. 1.5 Independence 41

This is a typical problem of assessing the reliability of a system consisting of

components that can fail independently. Such a system can often be divided into
subsystems, where each subsystem consists in turn of several components that are
connected either in series or in parallel; see Fig. 1 . 15(b).
Let a subsystem consist of components 1 , 2, . . . , m , and let Pi be the prob­
ability that component i is up ( "succeeds'l ) . Then, a series subsystem succeeds
if all of its components are up, so its probability of success is the product of the
probabilities of success of the corresponding components, i.e ..

P(series subsystem succeeds) = PIP2 . . . pm ·

A parallel subsystem succeeds if any one of its components succeeds, so its prob­
ability of failure is the product of the probabilities of failure of the corresponding
components, i.e. ,

P (parallel subsystem succeeds) = 1 - P (parallel subsystem fails)

= 1 - ( 1 - P 1 ) ( 1 - P2 ) . . . ( 1 - Pm ) .

Returning now to the network of Fig. 1 . 15(a), we can calculate the probabil­
ity of success (a path from A to B is available) sequentially, using the preceding
formulas, and starting from the end. Let us use the notation X to denote the ---+ Y
event that there is a (possibly indirect) connection from node X to node Then. Y.
P (C ---+ B) = 1 - ( 1 - P(C ---+ E and E ---+ B ) ) ( l - P(C ---+ F and F ---+ B) )
= 1 - ( 1 - pCEPEB ) ( l - PC FPFB )
= 1 - ( 1 - 0.8 . 0.9 )(1 - 0 . 95 . 0.85)
= 0.946,
P(A ---+ C and C ---+ B) = P(A ---+ C)P(C ---+ B) = 0.9 · 0.946 = 0 . 851 .
P(A ---+ D and D ---+ B) = P(A ---+ D)P(D ---+ B ) = 0 . 75 · 0. 95 = 0.712,
and finally we obtain the desired probability

P(A ---+ B) = 1 - ( 1 - P (A ---+ C and C ---+ B ) ) ( l - P(A ---+ D and D ---+ B))
= 1 - ( 1 - 0.85 1 ) ( 1 - 0. 7 12 )
= 0. 95 7 .
Independent Trials and the Binomial Probabilities
If an experiment involves a sequence of independent but identical stages, we say
that we have a sequence of independent trials . In the special case where there
are only two possible results at each stage, we say that we have a sequence of
independent Bernoulli trials. The two possible results can be anything, e . g . ,
"it rains" or "it doesn't rain," but we will often think in terms of coin tosses and
refer to the two results as "heads" (H) and "tails" (T) .
42 1

Consider an t hat consists of n tosses of a

which the probability of IS p, P is some number between O L In
this means are ... .U.'-.L"" LJ "" .. ...-

dent , w here A i
AI ! ! • • • • An
{ i t h toss is a head } .
=

can by means of a
description , as shown condi tional

outcome
tai ls has probabi l ity

u
n
t h at
sses.
...... ..... ... ..... .... .
of t o
"-"JLU. a ,U. LJ
n-Iong
k from 0
n.

three lnc1lp.np.n-
1 . 16 :
dent tosses of a coi n .
of an
t h e branches of the tree . we record the r- Ar oc' , rl
conditional probabilit ies, and by t h e multiplication rule, t he probability o f ob­
.. ...... .
taining a particular 3-toss sequence is calculated by multiplying t he probabilities
recorded the of t he tree .

p r obab i li ty
p( k ) = P ( k ...............'..... ... come in an n-toss
which will play an important role
of any sequence .. .......... -
k u
showed above
IS (1
the probabi1ity
, so we have

p(k ) == () -
n
k
(1 p)n - k :
Sec. 1.5 Independence 43

where we use the notation

(�) = number of distinct n-toss sequences that contain k heads.

The numbers (�) ( read as "n
choose k" )
are known as the binomial coeffi­
cients, while the probabilities p( k )
are known as the binomial probabilities.
Using a counting argument, to be given in Section 1.6, we can show that

(nk) n!
k! ( n - k ) ! ' k = 0, 1 , , n, ...
where for any positive integer i we have

i! = 1 . 2 · . . (i - 1 ) . i,

and, by convention, O! 1. An alternative verification is sketched in the end-of­

=

chapter problems. Note that the binomial probabilities must add to 1 , thus p( k )
showing the binomial formula

tk=O G) pk (l - p)n- k = 1.

Example 1 .25. Grade of Service. An internet service provider has installed c

modems to serve the needs of a population of n dialup customers. It is estimated
that at a given time, each customer will need a connection with probability
independent of the others. What is the probability that there are more customers
p,
needing a connection than there are modems?
Here we are interested in the probability that more than c customers simul­
taneously need a connection. It is equal to
n

L p( k ) ,
k =c+ l

where

are the binomial probabilities. For instance, if n = 1 00, = 0. 1 , and c = 15, the p
probability of interest turns out to be 0.0399 .
This example is typical of problems of sizing a facility to serve the needs
of a homogeneous population, consisting of independently acting customers. The
problem is to select the facility size to guarantee a certain probability ( sometimes
called grade of service) that no user is left unserved.
 44 Sample Space and Probability Chap. 1

1.6 COUNTING

The calculation of probabilities often involves counting the number of outcomes

in various events. We have already seen two contexts where such counting arises.
( a) When the sample space n has a finite number of equally likely outcomes,
so that the discrete uniform probability law applies. Then, the probability

of any event A is given by

number of elements of A
P A)
( _

- number of elements of n '

and involves counting the elements of A and of n.

( b ) When we want to calculate the probability of an event A with a finite
number of equally likely outcomes, each of which has an already known
probability p. Then the probability of A is given by

P(A) = p. ( number of elements of A) .

and involves counting the number of elements of A. An example of this type

is the calculation of the probability of heads in n coin tosses ( the binomial
k
probabilities ) . We saw in the preceding section that the probability of each
k
distinct sequence involving heads is easily obtained, but the calculation
of the number of all such sequences, to be presented shortly, requires some
thought.
While counting is in principle straightforward, it is frequently challenging;
the art of counting constitutes a large portion of the field of combinatorics. In
this section, we present the basic principle of counting and apply it to a number
of situations that are often encountered in probabilistic models.

The Counting Principle

The counting principle is based on a divide-and-conquer approach, whereby the

counting is broken down into stages through the use of a tree. For example,
consider an experiment that consists of two consecutive stages. The possible
results at the first stage are
stage are bl , b2 , . . . , bn . al, a2 , . . . , am;
the possible results at the second
Then, the possible results of the two-stage experiment
are all possible ordered pairs (ai , bj),
i 1 , . . . , m, j 1 , . . . , n. Note that the
=

number of such ordered pairs is equal to mn. This observation can be generalized
as follows (see also Fig. 1 . 17).
 1 45

1 . 1 7 : I l l ustration of t he basic pri nciple . The co u nt i ng is c arried

(r = 4 i n t h e The fi rst has nl resu lts. Fo r
every res u l t at t he fi rst i - 1 stages , there are nl r es u l t s a t t he
ith s tage . T he number of leaves is n l n2 . . . n \'" . This is the des i red count .

'-''V .. .. ..... ''..,;> '"'..., of r S uppose

(a) are nl possible results at the first
(b) For every possible result at the first stage, there are n2 possible results
t he second stage .

(c) for any the i - I

,,0.,;::;. ..... .;:1 . t h ere are ni
number of possible results of the r-stage process is

Example 1 The Number of Telephone Numbers. A local telephone

n umber is a 7- d igit sequence, b ut t he fi rst d igit has to be d ifferent from 0 or 1 .
How d isti nct telep h o n e numbers are there? We can vis u alize choice of a
seque nce as a U,,", lI. I IJ H:..;, a process ! we a
tot al of 7 .;:1 "' 0.';::;' <;;;;::1 and a of one out of 1 0 for
46 Sample Space and Probability Chap. 1

the first stage where we only have 8 choices. Therefore, the answer is

8 . 10 . 10 · . . 10 = 8 . 106 .
"'-.".-'
6 times

Example 1 .27. The Number of Subsets of an n-Element Set. Consider

an n-element set {Sl , S2 , . . ' , S n } . How many subsets does it have ( including itself
and the empty set ) ? We can visualize the choice of a subset as a sequential process
where we examine one element at a time and decide whether to include it in the set
or not. We have a total of n stages, and a binary choice at each stage. Therefore
the number of subsets is
2 . 2 . . . 2 = 2n .
'-......-"
n times

It should be noted that the Counting Principle remains valid even if each
first-stage result leads to a different set of potential second-stage results, etc. The
only requirement is that the number of possible second-stage results is constant,
regardless of the first-stage result.
In what follows, we will focus primarily on two types of counting arguments
that involve the selection of k objects out of a collection of n objects. If the order
of selection matters, the selection is called a permutation, and otherwise, it is
called a combination. We will then discuss a more general type of counting,
involving a partition of a collection of n objects into multiple subsets.

k-permutations
We start with n distinct objects, and let k be some positive integer, with k � n .
We wish to count the number of different ways that we can pick k out of these
n objects and arrange them in a sequence, i.e. , the number of distinct k-object
sequences. We can choose any of the n objects to be the first one . Having chosen
the first, there are only n - 1 possible choices for the second; given the choice of
the first two, there only remain n - 2 available objects for the third stage, etc.
When we are ready to select the last (the kth) object, we have already chosen
k - 1 objects, which leaves us with n - (k - 1 ) choices for the last one. By the
Counting Principle, the number of possible sequences, called k-permutations,
IS
n(-'---n_ 1 )_·_ . (n _ ) (.:...
+_1...:. ..- -_k----=-)__._
2 1
n( n - 1 ) . . . (n _ k + 1) = _ - -_----=- ' .,.:..._ k _ .,. n_ • • .
_

. .
(n - k) · 2 · 1
n!
(n - k) ! '
In the special case where k = n, the number of possible sequences, simply called
permutations, is
n(n - l ) (n - 2) . . . 2 . 1� = n! .
(Let k = n in the formula for the number of k-permutations, and recall the
convention O! = 1 .)
Sec. 1 . 6 Counting 47

Example 1 .28. Let us count the number of words that consist of four distinct
letters. This is the problem of counting the number of 4-permutations of the 26
letters in the alphabet . The desired number is
n ! = 26! = 26 · 25 · 24 · 23 = 358 , 800 .
(n k) !
_
22 !

The count for permutations can be combined with the Counting Principle
to solve more complicated counting problems.

Example 1.29. You have n l classical music CDs, n2 rock music CDs, and n3
country music CDs. In how many different ways can you arrange them so that the
CDs of the same type are contiguous?
We break down the problem in two stages, where we first select the order of
the CD types, and then the order of the CDs of each type. There are 3! ordered se­
quences of the types of CDs (such as classical/rock/country, rock/country/classical,
etc . ), and there are n l ! (or n 2 ! . or n3!) permutations of the classical (or rock. or
country, respectively) CDs . Thus for each of the 3! CD type sequences, there are
nl ! n2! n3! arrangements of CDs. and the desired total number is 3! n l ! n2 ! n3 ! .
Suppose now that you offer to give ki out of the n . CDs of each type i to a
friend, where ki < ni , i = 1 . 2. 3. What is the number of all possible arrangements
of the CDs that you are left with? The solution is similar, except that the number of
(ni - ki)-permutations of CDs of type i replaces ni l in the estimate, so the number
of possible arrangements is

Combinations
There are n people and we are interested in forming a committee of k. How
many different committees are possible? More abstractly, this is the same as the
problem of counting the number of k-element subsets of a given n-element set.
Notice that forming a combination is different than forming a k-permutation.
because in a combination there is no ordering of the selected elements.
For example, whereas the 2-permutations of the letters A, B, C, and D are

AB. BA, AC , CA, AD, DA, BC. CB, BD. DB, CD, DC,

the combinations of two out of these four letters are

AB, AC, AD, BC, BD. CD.

In the preceding example, the combinations are obtained from the per­
mutations by grouping together "duplicates" ; for example, AB and BA are not
48 Sample Space and Probability Chap. 1

viewed as distinct, and are both associated with the combination AB. This rea­
soning can be generalized: each combination is associated with k! "duplicate"
k-permutations, so the number n!/(n - k) ! of k-permutations is equal to the
number of combinations times k! . Hence, the number of possible combinations,
is equal to
n!
'
k! (n - k) !
Let us now relate the above expression to the binomial coefficient, which
was denoted by G) and was defined in the preceding section as the number of
n-toss sequences with k heads . We note that specifying an n-toss sequence with
k heads is the same as selecting k elements (those that correspond to heads) out
of the n-element set of tosses: i.e. : a combination of k out of n objects. Hence,
the binomial coefficient is also given by the same formula and we have

n!
k! (n - k) ! '

The number of combinations of two out of the four letters A , B.

Example 1.30.
4
C, and D is found by letting = and k = 2. It is
n

(4) _ � _6
2 - 2! 2! - ,

consistent with the listing given earlier.

It is worth observing that counting arguments sometimes lead to formulas
that are rather difficult to derive algebraically. One example is the binomial
formula
t G)P'(1
k=O p)n- -k 1. =

discussed in Section 1 .5. In the special case where p 1/2, = this formula becomes

tk=O G) = 2n ,
and admits the following simple interpretation. Since G) is the number of k­
element subsets of a given n-element subset. the sum over k of (�) counts the

2n.
number of subsets of all possible cardinalities. It is therefore equal to the number
of all subsets of an n-element set. which is

Example 1.31. We have a group of n persons. Consider clubs that consist of a

special person from the group (the club leader) and a number (possibly zero) of
Sec. 1 . 6 Counting 49

additional club members. Let us count the number of possible clubs of this type in
two different ways, thereby obtaining an algebraic identity.
There are n choices for club leader. Once the leader is chosen, we are left
with a set of n - 1 available persons, and we are free to choose any of the 2 n - 1
subsets. Thus the number of possible clubs is n2 n - 1 •

Alternatively, for fixed k, we can form a k-person club by first selecting k out
of the n available persons [there are G) choices] . We can then select one of the
members to be the leader ( there are k choices ) . By adding over all possible club
sizes k, we obtain the number of possible clubs as L�=l k (�) , thereby showing the
identity

Partitions
k
Recall that a combination is a choice of elements out of an n-element set
without regard to order. Thus, a combination can be viewed as a partition of
k
the set in two: one part contains elements and the other contains the remaining
k.
n - We now generalize by considering partitions into more than two subsets.
We are given an n-element set and nonnegative integers nl , n2 , · . . . nr .
whose sum is equal to n. We consider partitions of the set into r disjoint subsets.
with the ith subset containing exactly ni elements. Let us count in how many
ways this can be done.
We form the subsets one at a time. We have C� J ways of forming the
first subset. Having formed the first subset, we are left with n - nl elements.
We need to choose n2 of them in order to form the second subset, and we have
(n�; l ) choices, etc. Using the Counting Principle for this r-stage process, the
total number of choices is

which is equal to

n! (n - n I ) ! ( n - nl - . . . - nr- I ) !
n d (n - n I ) ! n2 ! ( n - nl - n2 ) !

We note that several terms cancel and we are left with

n!

This is called the multinomial coefficient and is usually denoted by

50 Sample Space and Probability Chap. 1

Example 1 .32. Anagrams. How many different words (letter sequences) can be
obtained by rearranging the letters in the word TATTOO? There are six positions
to be filled by the available letters. Each rearrangement corresponds to a partition
of the set of the six positions into a group of size 3 (the positions that get the letter
T) , a group of size 1 (the position that gets the letter A ) , and a group of size 2 (the
positions that get the letter 0). Thus, the desired number is
---.,.
6!
---,--,--
---
1 ! 2! 3!
= · · · ·5·
1 .2.3.4. .6
1 1 2 1 2 3
= 60 .

It is instructive to derive this answer using an alternative argument. (This

argument can also be used to rederive the multinomial coefficient formula: see
the end-of-chapter problems. ) Let us write TATTOO in the form TI AT2 T301 0 2
pretending for a moment that we are dealing with 6 distinguishable objects. These
6 objects can be rearranged in 6! different ways. However, any of the 3! possible
permutations of T1 , T2, and T3 , as well as any of the 2 ! possible permutations of
01 and 02 , lead to the same word. Thus, when the subscripts are removed, there
are only 6!/(3! 2! ) different words.

Example 1 .33. A class consisting of 4 graduate and 1 2 undergraduate students

is randomly divided into four groups of 4. What is the probability that each group
includes a graduate student? This is the same as Example 1 . 1 1 in Section 1 .3, but
we will now obtain the answer using a counting argument.
We first determine the nature of the sample space. A typical outcome is a
particular way of partitioning the 1 6 students into four groups of 4. We take the
term "randomly" to mean that every possible partition is equally likely, so that the
probability question can be reduced to one of counting.
According to our earlier discussion, there are

( 16
4, 4 , 4 , 4
) 16!
4! 4 ! 4! 4!

different partitions, and this is the size of the sample space.

Let us now focus on the event that each group contains a graduate student.
Generating an outcome with this property can be accomplished in two stages:
(a) Take the four graduate students and distribute them to the four groups; there
are four choices for the group of the first graduate student, three choices for
the second, two for the third. Thus, there is a total of 4! choices for this stage.
(b) Take the remaining 1 2 undergraduate students and distribute them to the
four groups (3 students in each) . This can be done in

12!
3! 3! 3! 3!

different ways.
By the Counting Principle, the event of interest can occur in
4! 1 2!
3! 3! 3! 3!
Sec. 1. 7 Summary and Discussion 51

different ways. The probability of this event is

4! 12!
3! 3! 3! 3!
16!
4! 4! 4! 4!

After some cancellations, we find that this is equal to

12 · 8 · 4
15 · 14 · 13 '

consistent with the answer obtained in Example 1 . 1 1 .

Here is a summary of all the counting results we have developed.

Summary of Counting Results

• Permutations of n objects: n!.
• k-permutations of n objects: n!/(n - k)!.

• Combinations of k out of n objects: (�) = �

n
k ! (n k ) ! ·
• Partitions of n objects into r groups, with the ith group having ni
objects:

SUMMARY AND DISCUSSION

A probability problem can usually be broken down into a few basic steps:
(a) The description of the sample space, that is, the set of possible outcomes
of a given experiment.
(b) The (possibly indirect) specification of the probability law (the probability
of each event).
(c) The calculation of probabilities and conditional probabilities of various
events of interest .
The probabilities of events must satisfy the nonnegativity, additivity, and nor­
malization axioms. In the important special case where the set of possible out­
comes is finite, one can just specify the probability of each outcome and obtain
52 Sample Space and Probability Chap. 1

the probability of any event by adding the probabilities of the elements of the
event.
Given a probability law, we are often interested in conditional probabilities,
which allow us to reason based on partial information about the outcome of
the experiment. We can view conditional probabilities as probability laws of a
special type, under which only outcomes contained in the conditioning event can
have positive conditional probability. Conditional probabilities can be derived
from the (unconditional) probability law using the definition P(A I B) = P(A n
B)/P (B) . However, the reverse process is often convenient, that is, first specify
some conditional probabilities that are natural for the real situation that we wish
to model, and then use them to derive the (unconditional) probability law.
We have illustrated through examples three methods for calculating prob­
abilities:
(a) The counting method. This method applies to the case where the num­
ber of possible outcomes is finite, and all outcomes are equally likely. To
calculate the probability of an event, we count the number of elements of
the event and divide by the number of elements of the sample space.
(b) The sequential method. This method applies when the experiment has a
sequential character, and suitable conditional probabilities are specified or
calculated along the branches of the corresponding tree (perhaps using the
counting method) . The probabilities of various events are then obtained
by multiplying conditional probabilities along the corresponding paths of
the tree, using the multiplication rule.
(c) The divide-and-conquer method . Here, the probabilities P (B) of vari­
ous events B are obtained from conditional probabilities P(B I Ai ) , where
the Ai are suitable events that form a partition of the sample space and
have known probabilities P ( A i ) The probabilities P (B) are then obtained
.

by using the total probability theorem.

Finally, we have focused on a few side topics that reinforce our main themes.
We have discussed the use of Bayes' rule in inference, which is an important
application context . We have also discussed some basic principles of counting
and combinatorics, which are helpful in applying the counting method.
Problems 53

PROBLEMS

SECTION 1.1. Sets

Problem 1 . Consider rolling a six-sided die. Let A be the set of outcomes where the
roll is an even number. Let B be the set of outcomes where the roll is greater than 3 .
Calculate and compare the sets o n both sides of D e �lorgan's laws

(A n B) C AC u BC •
=

Problem 2. Let A and B be two sets.

( a ) Show that

( b ) Show that

(c ) Consider rolling a fair six-sided die. Let A be the set of outcomes where the roll
is an odd number. Let B be the set of outcomes where the roll is less than 4.
Calculate the sets on both sides of the equality in part ( b ) , and verify that the
equality holds.

Problem 3. * Prove the identity

Solu tion. If x belongs to the set on the left, there are two possibilities. Either x E A,
in which case x belongs to all of the sets A U BlI • and therefore belongs to the set on
the right. Alternatively. x belongs to all of the sets Bn in which case. it belongs to all
of the sets A U Bn . and therefore again belongs to the set on the right.
Conversely. if x belongs to the set on the right. then it belongs to A u Bn for all
n. If x belongs to A. then it belongs to the set on the left. Otherwise. x must belong
to every set Bn and again belongs to the set on t he left .
Problem 4 . * Cantor's diagonalization argument. Show that the unit interval
[0, 1] is uncountable. i.e., its elements cannot be arranged in a sequence.
Solution. Any number x in [0. 1] can be represented in terms of its decimal expansion.
e.g., 1/3 = 0.3333 · · ·. Note that most numbers have a unique decimal expansion,
but there are a few exceptions. For example, 1 /2 can be represented as 0.5000 · . . or
as 0.49999 · . '. It can be shown that this is the only kind of exception, i.e . . decimal
expansions that end with an infinite string of zeroes or an infinite string of nines.
54 Sample Space and Probability Chap. 1

Suppose, to obtain a contradiction, that the elements of [0, :1.] can be arranged
in a sequence Xl , X2 , X3 , , so that every element of [0, 1] appears in the sequence.
• . .

Consider the decimal expansion of Xn :

where each digit a� belongs to {O, 1 , . . . , 9}. Consider now a number y constructed as
follows. The nth digit of y can be 1 or 2, and is chosen so that it is different from the
nth digit of Xn . Note that y has a unique decimal expansion since it does not end with
an infinite sequence of zeroes or nines. The number y differs from each Xn , since it has
a different nth digit. Therefore, the sequence Xl , X2, does not exhaust the elements
• • •

of [0, 1], contrary to what was assumed. The contradiction establishes that the set [0, 1]
is uncountable.

SECTION 1.2. Probabilistic Models

Problem 5. Out of the students in a class, 60% are geniuses, 70% love chocolate,
and 40% fall into both categories. Determine the probability that a randomly selected
student is neither a genius nor a chocolate lover.

Problem 6. A six-sided die is loaded in a way that each even face is twice as likely
as each odd face. All even faces are equally likely, as are all odd faces. Construct a
probabilistic model for a single roll of this die and find the probability that the outcome
is less than 4.

Problem 7. A four-sided die is rolled repeatedly, until the first time (if ever) that an
even number is obtained. What is the sample space for this experiment?

Problem 8. You enter a special kind of chess tournament, in which you play one game
with each of three opponents, but you get to choose the order in which you play your
opponents, knowing the probability of a win against each . You win the tournament if
you win two games in a row, and you want to maximize the probability of winning.
Show that it is optimal to play the weakest opponent second, and that the order of
playing the other two opponents does not matter.

Problem 9. A partition of the sample space n is a collection of disjoint events

81 , . . . , 8n such that n = U� 1 8i .
(a) Show that for any event A, we have
n
P ( A ) = L P ( A n 8i ) .
i= l

(b) Use part (a) to show that for any events A, B , and C, we have
P ( A ) = P ( A n B) + P(A n C) + P ( A n BC n CC) - P(A n B n C) .

Problem 10. Show the formula

p ( (A n BC) u (AC n B)) = P(A) + P(B) - 2P(A n B),
Problems 55

which gives the probability that exactly one of the events A and B will occur. [Compare
with the formula P(A U B) = P(A) + P(B) - P(A n B), which gives the probability
that at least one of the events A and B will occur. ]

Problem 1 1.* Bonferroni's inequality.

(a) Prove that for any two events A and B, we have
P(A n B) �
P(A) + P(B) 1 . -

(b ) Generalize to the case of n events A I , A2 , . . . , An , by showing that

P(AI n A2 n · · · n An ) � P(AI ) + P(A2) + . . . + P(An ) - ( n - 1 ) .
Solution. We have P(A U B ) = P(A) + P(B) - P(A n B ) and P(A U B ) :::; 1 . which
implies part ( a) . For part ( b ) , we use De Morgan's law to obtain
1 - P(AI n · · · n An) = P ((A I n · · · n An r )
= P(A� U · · · U A� )
:::; P(A� ) + . . . + P(A� )
= ( 1 - P(Ad ) + . . . + ( 1 - P(An ) )
= n P(A I ) - . . . - P(An ).
-

Problem 1 2 . * The inclusion-exclusion formula. Show the following generaliza­

tions of the formula
P(A U B) = P(A) + P(B) - P(A n B).
(a) Let A , B, and C be events. Then,
P(A uBuC) = P(A)+P(B)+P(C) - p (AnB) -p(BnC) - p (AnC)+p(AnBnC).
(b ) Let A l , A2 , . . . , An be events. Let 81 = {i l l :::; i :::; n}, 82 = { (il , i2) l l :::; il <
i2 :::; n}, and more generally, let 8m be the set of all m-tuples (il , . . . , im) of
indices that satisfy 1 ::; i l < i 2 < . . . < im ::; n . Then.
P (U k=I A k ) = L P(At ) - L P(Ai l n Ai 2 )

Solution . ( a)
We use the formulas P(X U Y) = P(X) + P(Y) - P(X n Y) and
(A U B) n C = (A n C) U (B n C) . We have
P(A U B U C) = P(A U B) + P(C) - P ( (A U B) n C )
= P(A U B) + P(C) - P ( (A n C) U ( B n C) )
= P(A U B) + P(C) - P(A n C) - P(B n C) + P(A n B n C)
= P(A) + P(B) - P(A n B) + P(C) - P(A n C) - P(B n C)
+ P(A n B n C)
= P(A) + P(B) + P(C) - P(A n B) - P(B n C) - P(A n C)
+ p(A n B n C) .
56 Sample Space and Probability Chap. 1

(b) Use induction and verify the main induction step by emulating the derivation of
part (a) . For a different approach, see the problems at the end of Chapter 2.
Problem 13. * Continuity property of probabilities.
(a) Let A I , A2 , . . . be an infinite sequence of events, which is "monotonically increas­
ing," meaning that A n C A n + l for every n. Let A = U;:='= IA n . Show that
P(A) = limn_ oo P (An ) . Hint: Express the event A as a union of countably
many disjoint sets.
(b) Suppose now that the events are "monotonically decreasing," i.e. , An+l C An
for every n. Let A n�= I An . Show that P (A ) limn_ oo P (An ) . Hint: Apply
= =

the result of part (a) to the complements of the events.

(c) Consider a probabilistic model whose sample space is the real line. Show that

p ( [O, (0)) = lim

n - oc
P ( [O, nj) , and lim P ( [n, (0 ) )
n -oo
= 0.

Solu tion. (a) Let BI = Al and, for n 2: 2, Bn = A n n A� - l ' The events Bn are
disjoint, and we have Uk= I Bk = An, and Uk= I Bk = A. We apply the additivity axiom
to obtain
00 n
P (A) = � P (Bk ) = lim � P(Bk ) = lim P(Uk= I Bk ) = lim P(An ) .
L
k= l
n - oc L
k= l
n -oo n -oo
(b) Let Cn = A� and C = AC . Since An+l C An , we obtain Cn C Cn + l , and the events
Cn are increasing. Furthermore, C = AC = (n�= I An)C = U�=lA� = U�=lCn . Using
the result from part (a) for the sequence Cn , we obtain

1 - P(A) = P (AC ) = P( C) = nlim P ( Cn ) = lim ( 1 - P (An) ) ,

-x n -oo
from which we conclude that P (A) = limn_ x P(An ) .
(c) For the first equality, use the result from part (a) with A n = [O, n] and A = [0, (0).
For the second, use the result from part (b) with An = [n, (0 ) and A = n� I An = 0 .

SECTION 1.3. Conditional Probability

Problem 14. We roll two fair 6-sided dice. Each one of the 36 possible outcomes is
assumed to be equally likely.
(a) Find the probability that doubles are rolled.
(b) Given that the roll results in a sum of 4 or less, find the conditional probability
that doubles are rolled.
(c) Find the probability that at least one die roll is a 6 .
(d) Given that the two dice land on different numbers, find the conditional probability
that at least one die roll is a 6.

Problem 15. A coin is tossed twice. Alice claims that the event of two heads is at
least as likely if we know that the first toss is a head than if we know that at least one
Problems 57

of the tosses is a head. Is she right? Does it make a difference if the coin is fair or
unfair? How can we generalize Alice's reasoning?

Problem 16. We are given three coins: one has heads in both faces, the second has
tails in both faces, and the third has a head in one face and a tail in the other. We
choose a coin at random, toss it, and the result is heads. What is the probability that
the opposite face is tails?

Problem 17. A batch of one hundred items is inspected by testing four randomly
selected items. If one of the four is defective, the batch is rejected. What is the
probability that the batch is accepted if it contains five defectives?

Problem 18. Let A and B be events. Show that P ( A n B I B) = P ( A I B), assuming

that P(B) > O.

SECTION 1 .4. Total Probability Theorem and Bayes' Rule

Problem 19. Alice searches for her term paper in her filing cabinet. which has several
drawers. She knows that she left her term paper in drawer j with probability Pi > O.
The drawers are so messy that even if she correctly guesses that the term paper is in
drawer i. the probability that she finds it is only di . Alice searches in a particular
drawer. say drawer i . but the search is unsuccessful. Conditioned on this event, show
that the probability that her paper is in drawer j , is given by
Pi Pi ( l - di )
i f j ¥= i . if j = i .
1 - p, di .
Problem 20. How an inferior player with a superior strategy can gain an
advantage. Borb b about to play a two-game che:s:s match with an opponent, and
wants to find the strategy that maximizes his winning chances. Each game ends with
either a win by one of the players, or a draw. If the score is tied at the end of the two
games, the match goes into sudden-death mode, and the players continue to play until
the first time one of them wins a game (and the match) . Boris has two playing styles.
timid and bold, and he can choose one of the two at will in each game. no matter what
style he chose in previous games. With timid play. he draws with probability Pd > 0,
and he loses with probability 1 - Pd . With bold play. he wins with probability pw , and
he loses with probability 1 pw ' Boris will always play bold during sudden death, but
-

may switch style between games 1 and 2.

(a) Find the probability that Boris wins the match for each of the following strategies:
(i) Play bold in both games 1 and 2.
(ii) Play timid in both games 1 and 2.
(iii) Play timid whenever he is ahead in the score. and play bold otherwise.
(b) Assume that pw < 1 /2, so Boris is the worse player, regardless of the playing
style he adopts. Show that with the strategy in (iii) above. and depending on
the values of pw and Pd.
Boris may have a better than a 50-50 chance to win the
match. How do you explain this advantage?

Problem 2 1 . Two players take turns removing a ball from a jar that initially contains
m white and n black balls. The first player to remove a white ball wins. Develop a
58 Sample Space and Probability Chap. 1

recursive formula that allows the convenient computation of the probability that the
starting player wins.

Problem 22. Each of k jars contains m white and n black balls. A ball is randomly
chosen from jar 1 and transferred to jar 2, then a ball is randomly chosen from jar 2
and transferred to jar 3, etc. Finally, a ball is randomly chosen from jar k. Show that
the probability that the last ball is white is the same as the probability that the first
ball is white, i.e. , it is m / ( m + n ) .

Problem 23. We have two jars, each initially containing an equal number of balls.
We perform four successive ball exchanges. In each exchange, we pick simultaneously
and at random a ball from each jar and move it to the other jar. What is the probability
that at the end of the four exchanges all the balls will be in the jar where they started?

Problem 24. The prisoner's dilemma. The release of two out of three prisoners
has been announced. but their identity is kept secret. One of the prisoners considers
asking a friendly guard to tell him who is the prisoner other than himself that will be
released, but hesitates based on the following rationale: at the prisoner's present state
of knowledge, the probability of being released is 2/3, but after he knows the answer,
the probability of being released will become 1 /2, since there will be two prisoners
(including himself) whose fate is unknown and exactly one of the two will be released.
What is wrong with this line of reasoning?

Problem 25. A two-envelopes puzzle. You are handed two envelopes. and you
know that each contains a positive integer dollar amount and that the two amounts are
different. The values of these two amounts are modeled as constants that are unknown.
Without knowing what the amounts are, you select at random one of the two envelopes,
and after looking at the amount inside, you may switch envelopes if you wish. A friend
claims that the following strategy will increase above 1 /2 your probability of ending
up with the envelope with the larger amount: toss a coin repeatedly. let X be equal to
1 /2 plus the number of tosses required to obtain heads for the first time, and switch
if the amount in the envelope you selected is less than the value of X . Is your friend
correct?

Problem 26. The paradox of induction. Consider a statement whose truth is

unknown. If we see many examples that are compatible with it, we are tempted to
view the statement as more probable. Such reasoning is often referred to as induc­
tive inference (in a philosophical, rather than mathematical sense). Consider now the
statement that "all cows are white." An equivalent statement is that "everything that
is not white is not a cow." We then observe several black crows. Our observations are
clearly compatible with the statement. but do they make the hypothesis "all cows are
white" more likely?
To analyze such a situation, we consider a probabilistic model. Let us assume
that there are two possible states of the world, which we model as complementary
events:
A : all cows are white,
A C : 50% of all cows are white.
Let p be the prior probability P(A) that all cows are white. We make an observation
of a cow or a crow, with probability q and 1 - q , respectively, independent of whether
Problems 59

event A occurs or not . Assume t hat 0 < p < 1, 0 < q < 1, and that all crows are black.

(a) Given the event B = {a black crow was observed } , what is p eA I B)?
( b ) G iven the event C = {a white cow was observed } , what i s peA I C)?

Problem 27. Alice and Bob have 2n + 1 coins, each coin with probability of heads
equal to 1 /2. Bob tosses n + 1 coins, while Alice tosses the remaining n coins. Assuming
independent coin t osses , show that the probability that after all coins have been tossed ,
Bob will have gotten more heads than Alice is 1 /2.

Problem 28. * Conditional version of the total probability theorem. Let

Cl , • • . , Cn be disjoint events that form a partition of the state space. Let also A and
B be events such that PCB n Ci ) > 0 for all i. Show t hat

P (A I B) = I: P (Cd B)p(A I B n CI ) .
i=l

Solution. We have
n

p(A n B ) = L P ( A n B) n Ci ) ,
i=l

and by using the multiplication rule,

Combining these two equations, dividing by PCB), and using the formula peA I B) =

peA n B)/P(B), we obtain the desired result .

Problem 29. * Let A and B be events with peA) > 0 and PCB) > O. We say that
an event B suggests an event A if p e A I B ) > peA), and does not suggest event A if
peA I B) < p eA) .
(a) Show t hat B suggests A if and only if A suggests B.
( b ) Assume that P(BC ) > O. Show t hat B suggests A if and only if BC does not
suggest A.
( c ) We know that a treasure i s located in one o f two places, with probabilities f3 and
1 - f3, respectively, where 0 < f3 < 1 . We search the first place and if the treasure
is there, we find it with probability p > O. Show that the event of not finding the
treasure in the first place suggests that the treasure is in the second place.

Solu tion. (a) We have peA I B) = peA n B ) /P ( B ) , so B suggests A if and only if

peA n B) > P(A)P(B), which is equivalent to A suggesting B, by symmetry.
(b) Since PCB) + P(BC) = 1, we have

P ( B)P(A) + P ( BC )P(A) = p eA) = P (B ) P (A I B) + P(BC ) P (A I BC),

which implies that

P ( BC ) (P(A) - peA I B e») = P ( B) ( P ( A I B) - peA ») .

 Probability 1

P(A I > P(A) (8 if P (A) > n ot

s uggest A ) .

A B eve nts

A = treasure i s

B = { we t reasure i n

Using the probabi l ity

P(B) = ( AC ) P (B I + (B I A ) = t1( 1 - + (1 - ,6) !

( ___1____ _1 1
so
_A n >
=
{3
= =
____

+ ( 1 - /3 )
B) - _- _
=
__-,- ( A) .
P (B) .8 ( 1 - 1 - .8p .

It event

Problem A hu nter has two hu nti n g dogs . One d ay, o n the t rail of some ani m a l ,
d i ve rges i nto two p at h s . He t hat
p.
h u nter d og a pat h ,
if t hey d isagree , to randomly pick a pat h .
t h e two decide o n a pat h ?

a no isy A sou rce a

- p,
message ( a str ing of symbols) through a noisy commu n i cation channel. Each symbol is
o or 1 with probabi lity p 1 is recei ved i ncorrectly prob-
£0 (I ! in
are i ndep endent .

o o

1
1 -

1 . 18: Error probabi l i ties in a communication channeL

( a) the

(b) t he

(c) is
 ... .0 ...'.0 " . ,.0 ... as a 0

(d )

is a

can
or the movies ) is

1 . 19: A the sub-

tha t starts at
,,", " ::"'< ".C I l I

A power utility can

i with
62 Sample Space and Probability Chap. 1

(a) Suppose that any one plant can produce enough electricity to supply the entire
city. What is the probability that the city will experience a black-out?
(b) Suppose that two power plants are necessary to keep the city from a black-out.
Find the probability that the city will experience a black-out.

Problem 37. A cellular phone system services a population of n l "voice users" (those
who occasionally need a voice connection) and n2
"data users" (those who occasionally
need a data connection) . We estimate that at a given time, each user will need to be
connected to the system with probability P I (for voice users) or P2
(for data users) ,
rl
independent of other users. The data rate for a voice user is bits/sec and for a data
user isr2 bits/sec. The cellular system has a total capacity of c bits/sec. What is the
probability that more users want to use the system than the system can accommodate?

Problem 38. The problem of points. Telis and Wendy play a round of golf ( 18
holes) for a $10 stake, and their probabilities of winning on any one hole are P and
1 - p, respectively, independent of their results in other holes. At the end of 10 holes,
with the score 4 to 6 in favor of Wendy, Telis receives an urgent call and has to report
back to work. They decide to split the stake in proportion to their probabilities of
winning had they completed the round, as follows. If PT and pw are the conditional
probabilities that Telis and Wendy, respectively, are ahead in the score after 18 holes
given the 4-6 score after 10 holes, then Telis should get a fraction PT / (PT + pw ) of the
stake, and Wendy should get the remaining pw /(PT + pw ). How much money should
Telis get? Note: This is an example of the, so-called, problem of points, which played
an important historical role in the development of probability theory. The problem
was posed by Chevalier de Mere in the 17th century to Pascal, who introduced the
idea that the stake of an interrupted game should be divided in proportion to the
players' conditional probabilities of winning given the state of the game at the time of
interruption. Pascal worked out some special cases and through a correspondence with
Fermat , stimulated much thinking and several probability-related investigations.

Problem 39. A particular class has had a history of low attendance. The annoyed
n
professor decides that she will not lecture unless at least k of the students enrolled
in the class are present. Each student will independently show up with probability
Pg if the weather is good, and with probability Pb if the weather is bad. Given the
probability of bad weather on a given day, obtain an expression for the probability that
the professor will teach her class on that day.

Problem 40. Consider a coin that comes up heads with probability P and tails with
probability 1 - p. Let qn be the probability that after independent tosses, there have
n
been an even number of heads. Derive a recursion that relates qn to qn - I , and solve
this recursion to establish the formula

qn = ( 1 + ( 1 - 2Pt ) /2.

Problem 41. Consider a game show with an infinite pool of contestants, where
at each round i, contestant i obtains a number by spinning a continuously calibrated
wheel. The contestant with the smallest number thus far survives. Successive wheel
spins are independent and we assume that there are no ties. Let N be the round at
which contestant 1 is eliminated. For any positive integer n, find peN = n ) .
Problems 63

Problem 42. * Gambler's ruin. A gambler makes a sequence of independent bets.

In each bet, he wins $1 with probability p, and loses $ 1 with probability 1 - p. Initially,
the gambler has $k, and plays until he either accumulates $n or has no money left.
What is the probability that the gambler will end up with $n?
Solution. Let us denote by A the event that he ends up with $n, and by F the event
that he wins the first bet. Denote also by Wk the probability of event A, if he starts
with $k. We apply the total probability theorem to obtain

Wk = P (A I F)P(F) + P(A I FC)P(FC) = pP(A I F) + qP(A I FC ). 0 < k < n,

where q = 1 - p. By the independence of past and future bets, having won the first bet
is the same as if he were just starting now but with $(k+ l ) , so that P(A I F) = Wk+ l
and similarly P(A I FC) = Wk - l . Thus, we have W k = PWk+ l + qwk- l , which can be
written as
0 < k < n,
where r = q/p. We will solve for Wk in terms of p and q using iteration, and the
boundary values Wo = 0 and Wn = 1 .
We have Wk+ l - Wk = r k (wl - wo ) , and since Wo = 0,
Wk+ l = Wk + r k WI = Wk - l +r
k-l k
WI + r WI =
k
WI + rWI + . . . + r WI .

The sum in the right-hand side can be calculated separately for the two cases where
r = 1 (or p = q) and r =J. 1 (or p =J. q). We have

if p =J. q .
if p = q.
Since Wn = 1 , we can solve for WI and therefore for Wk :
-r
if p =J. q,
1 - rn '
WI =
1
-. if p = q,
rn
so that k
c-r •
if p =J. q,
1 - rn
Wk =
if p = q .
k
-,
n

Problem 43. * Let A and B be independent events. Use the definition of indepen­
dence to prove the following:
(a) The events A and Be are independent.
(b) The events AC and BC are independent.
Solution. (a) The event A is the union of the disjoint events A n BC and A n B. Using
the additivity axiom and the independence of A and B, we obtain

P(A) = P(A n B) + P(A n BC) = P(A)P(B) + P(A n BC).

64 Sample Space and Probability Chap. 1

It follows that
P (A n Be) = P (A) (1 - P(B) ) = P (A)P(BC ) .
so A and BC are independent.
(b) Apply the result of part (a) twice: first on A and B. then on BC and A.
Problem 44. * Let A. B. and C be independent events, with P( C) > O. Prove that
A and B are conditionally independent given G.
Solution. We have
P (A n B n G)
P(A n B I G) =
P (G)
P(A)P(B)P(G)
P (G)
= P(A)P (B)
= P(A I G)P(B I G),
so A and B are conditionally independent given G. In the preceding calculation, the
first equality uses the definition of conditional probabilities; the second uses the as­
sumed independence; the fourth uses the independence of A from G, and of B from G .
Problem 45. * Assume that the events A I , A2 , A3 , A4 are independent and that
P(A3 n A4 ) > O. Show that

Solution. We have

We similarly obtain P (A2 1 A3 n A4 ) = P(A2) and P (A 1 n A2 1 A3 n A4 ) = P(A 1 n A2 ) ,

and finally,

P(AI U A2 1 A3 n A 4 ) = P(A 1 I A3 n A 4 ) + P(A2 1 A3 n A4 ) - P(AI n A2 1 A3 n A 4 )

= P(A t ) + P (A2) - P(A1 n A2 )
= P( A 1 U A2 ).

Problem 46. * Laplace's rule of succession. Consider m + 1 boxes with the kth
box containing k red balls and m - k white balls, where k ranges from 0 to m. We
choose a box at random (all boxes are equally likely) and then choose a ball at random
n
from that box, successive times (the ball drawn is replaced each time, and a new ball
n
is selected independently). Suppose a red ball was drawn each of the times. What
is the probability that if we draw a ball one more time it will be red? Estimate this
probability for large m.
Solution. We want to find the conditional probability P (E I Rn ) , where E is the event
of a red ball drawn at time + 1 , and Rn is the event of a red ball drawn each of the
n n
preceding times. Intuitively, the consistent draw of a red ball indicates that a box with
Problems 65

a high percentage of red balls was chosen, so we expect that P ( I E Rn)

is closer to I
than to O. In fact, Laplace used this example to calculate the probability that the sun
will rise tomorrow given that it has risen for the preceding 5,000 years. (It is not clear
how serious Laplace was about this calculation, but the story is part of the folklore of
probability theory. )
We have
P(E I Rn ) = p(EP(RnnRn)) '
and by using the total probability theorem, we obtain

P(Rn) t p(kth box chosen) (!) n = m � I t (!) n ,

=

m ( k )1l + 1
k=O k=O

Rn) = R71 + d = m + I L m
P(E n P(
I
k=O

For large m, we can view P(Rn) as a piecewise constant approximation to an integral:

P(
I m ( k )n I 1 m
Rn ) = m + I L m :::::: (m + I ) mn x dx = (m + I ) mn . n ++1I :::::: n +I I .
n I
0
m n
k=O

Similarly,
P(E n Rn ) = ( Rn+ d :::::: n +I 2 '
P

so that
P (E I R71) :::::: nn ++ 2I .
Thus, for large m, drawing a red ball one more time is almost certain when n is large.
Problem 47. * Binomial coefficient formula and the Pascal triangle.
(a) Use the definition of G) as the number of distinct n-toss sequences with k
heads, to derive the recursion suggested by the so called Pascal triangle, given in
Fig. 1 . 20.
(b) Use the recursion derived in part (a) and induction, to establish the formula

()
n
k
-
n!
k! (n - k)! '

Solution. (a) Note that n-toss sequences that contain k heads ( for a < k < n) can be
obtained in two ways:
( 1 ) By starting with an (n - I )-toss sequence that contains k heads and adding a tail
at the end. There are ( n k1)
different sequences of this type .
(2) By starting with an (n - I )-toss sequence that contains k - 1 heads and adding
a head at the end. There are G=�) different sequences of this type.
 1

(0 )
) ( 1 )
2
( ) ( �) (2 )
3 3
) () ( ) ()
(� ) ( ) ( ) ( � ) (�� ) 1

1 .20: .... ... ...." ' ... 'n f- binomial co��m4C le:m:.s
Each term (�) in the
and i n t he array on the its two in the
row above it ( exce p t for the boundary terms with k = 0 or k = n , w h ich are equal
to 1 ) .

if k = 1, . . . j n - 1.

if k = n.

This is form u l a correspond i n g to the Pascal triangle calculat i o n , gi ven in 1

(b) now use (a) � to

=
(�) k ! (n - k ) ! '

by on n . we t he (�) = (i) = 1 , so n = 1
above formula is seen to ho l d as l o n g as we use the convention O ! = 1. If t he formula
holds for each i n dex up to n - 1 \ we have for k = 1 , 2 , . , n - 1j ..
n
( k)
n
(k - l
- 1) (n - )+ k
1

-------�-
=
(k -
= -k . + -- . ---- ---
n-k n!
n k! ( n - k) ! 11,
k ! (n - k ) !

k! ( n - k ) ! '

and the i nd uction is complete.

48 . *
ith t r i a l is some positive n u m b e r P i .
Consi der a n infinite sequence o f trials.
The probability of success at N b e t he
Problems 67

event that there is no success, and let

of successes.
I be the event that there is an infinite number
2:: :': 1 PI =
I
(a) Assume that the trials are independent and that
P (N) = 0 and P ( ) = 1 .
00. Show that

(b) Assume that 2:: : 1 PI < 00 . Show that P( I) = O.

Solution. (a) The event N is a subset of the event that there were no successes in the
first n trials, so that
n
P(N) :::; II ( 1 - pd ·
i= 1
Taking logarithms,
n n
i=1 i= 1
Taking the limit as n tends to infinity, we obtain log P (N) = - 00 . or P (N) = O.
Let now Ln be the event that there is a finite number of successes and that the
last success occurs at the nth trial. We use the already established result P (N) = 0,
and apply it to the sequence of trials after trial n, to obtain P(Ln ) = O. The event
(finite number of successes) is the union of the disjoint events Ln, n 2: 1 . and N, so
Ie
that
I
P ( C ) = P (N) + L P(Ln ) = 0,
=

n= 1
and P(I ) = 1 .
(b) Let Si b e the event that the ith trial is a success.
Fix some number n and for every
i > n, let Pi be the event that the first success after time n occurs at time i. Note
that Pi C Si. Finally, let An be the event that there is at least one success after time
I
n. Note that C An , because an infinite number of successes implies that there are
successes subsequent to time n. Furthermore, the event An is the union of the disjoint
events Pi , i > n. Therefore,

P(I) :s P(An ) = P (�Q , Fi ) - � P (F; ) i� ' P(Si ) ,�,

i '
:s = Pi ·

We take the limit of both sides as n ---+ 00. Because of the assumption 2:: : 1 Pt < 00 ,

the right-hand side converges to zero. This implies that P(I ) = O.

SECTION 1.6. Counting

Problem 49. De Mere's puzzle. A six-sided die is rolled three times independently.
Which is more likely: a sum of 1 1 or a sum of 1 2? (This question was posed by the
French nobleman de Mere to his friend Pascal in the 1 7th century.)

Problem 50. The birthday problem. Consider n people who are attending a
party. We assume that every person has an equal probability of being born on any day
68 Sample Space and Probability Chap. 1

during the year. independent of everyone else, and ignore the additional complication
presented by leap years (i.e. , assume that nobody is born on February 29) . What is
the probability that each person has a distinct birthday?

Problem 51. An urn contains m red and n white balls.

(a) We draw two balls randomly and simultaneously. Describe the sample space and
calculate the probability that the selected balls are of different color, by using
two approaches: a counting approach based on the discrete uniform law, and a
sequential approach based on the multiplication rule.
(b) We roll a fair 3-sided die whose faces are labeled 1 ,2,3, and if k comes up. we
remove k balls from the urn at random and put them aside. Describe the sample
space and calculate the probability that all of the balls drawn are red. using a
divide-and-conquer approach and the total probability theorem.

Problem 52. We deal from a well-shuffled 52-card deck. Calculate the probability
that the 13th card is the first king to be dealt.

Problem 53. Ninety students, including Joe and Jane, are to be split into three
classes of equal size, and this is to be done at random. What is the probability that
Joe and Jane end up in the same class?

Problem 54. Twenty distinct cars park in the same parking lot every day. Ten of
these cars are US-made. while the other ten are foreign-made . The parking lot has
exactly twenty spaces. all in a row. so the cars park side by side. However. the drivers
have varying schedules. so the position any car might take on a certain day is random.
(a) In how many different ways can the cars line up?
(b) What is the probability that on a given day, the cars will park in such a way
that they alternate (no two US-made are adjacent and no two foreign-made are
adjacent)?

Problem 55. Eight rooks are placed in distinct squares of an 8 x 8 chessboard, with
all possible placements being equally likely. Find the probability that all the rooks are
safe from one another, i.e . . that there is no row or column with more than one rook.

Problem 56. An academic department offers 8 lower level courses: { L 1 , L2 , . . . . Ls }

and 10 higher level courses: { H 1 . H2 • H 10 } . A valid curriculum consists of 4 lower
. . • •

level courses. and 3 higher level courses.

(a) How many different curricula are possible?
(b) Suppose that { H1 . . . . , H5 } have L 1 as a prerequisite, and { H6 , ' " HlO } have L2
and L3 as prerequisites. i.e . . any curricula which involve, say, one of {H 1 , . . . , H5 }
mllst also include L1 . How many different curricula are there?

Problem 57. How many 6-word sentences can be made using each of the 26 letters
of the alphabet exactly once? A word is defined as a nonempty (possibly jibberish)
sequence of letters.
Problems 69

Problem 58. We draw the top 7 cards from a well-shuffled standard 52-card deck.
Find the probability that:
(a) The 7 cards include exactly 3 aces.
(b) The 7 cards include exactly 2 kings.
(c) The probability that the 7 cards include exactly 3 aces. or exactly 2 kings, or
both.

Problem 5 9. A parking lot contains 100 cars, k of which happen to be lemons. We

select m of these cars at random and take them for a test drive. Find the probability
that n of the cars tested turn out to be lemons.

Problem 60. A well-shuffled 52-card deck is dealt to 4 players. Find the probability
that each of the players gets an ace.

Problem 6 1. * Hypergeometric probabilities. An urn contains balls, out of n

which m are red. We select k of the balls at random. without replacement (Le., selected
balls are not put back into the urn before the next selection) . What is the probability
that i of the selected balls are red?
Solution. The sample space consists of the (�) different ways that we can select k out
of the available balls. For the event of interest to occur. we have to select i out of the
m red balls, which can be done in (7) ways, and also select k i out of the n - m balls
that are not red, which can be done in ( nk�7) ways. Therefore, the desired probability
-

IS

for i � 0 satisfying i � m, i � k, and k - i � n - m. For all other i, the probability is

zero.
Problem 62. * Correcting the number of permutations for indistinguishable
objects. When permuting n objects, some of which are indistinguishable, different
permutations may lead to indistinguishable object sequences, so the number of distin­
guishable object sequences is less than nL For example, there are six permutations of
the letters A, B, and C:
ABC. ACB, BAC, BCA, CAB, CBA,
but only three distinguishable sequences that can be formed using the letters A, D,
and D:
ADD, DAD. DDA.

n
(a) Suppose that k out of the objects are indistinguishable. Show that the number
of distinguishable object sequences is n! / kL
r
(b) Suppose that we have types of indistinguishable objects, and for each i, ki
objects of type i. Show that the number of distinguishable object sequences is
n!
70 Sample Space and Probability Chap. 1

Solution. (a) Each one of the n! permutations corresponds to k! duplicates which are
obtained by permuting the k indistinguishable objects. Thus, the n! permutations can
be grouped into n! /k! groups of k! indistinguishable permutations that result in the
same object sequence. Therefore, the number of distinguishable object sequences is
n!/kL For example, the three letters A, D, and D give the 3! = 6 permutations

ADD, ADD, DAD, DDA, DAD, DDA ,

obtained by replacing B and C by D i n the permutations of A, B, and C given earlier.

However, these 6 permutations can be divided into the n!/k! = 3!/2! = 3 groups

{ADD, ADD } , {DAD, DAD } , { DDA, DDA},

each having k! = 2! = 2 indistinguishable permutations.

(b) One solution is to extend the argument in (a) above: for each object type i, there are
ki ! indistinguishable permutations of the ki objects. Hence, each permutation belongs
to a group of kd k2 ! . . . kr ! indistinguishable permutations, all of which yield the same
object sequence.
An alternative argument goes as follows. Choosing a distinguishable object se­
quence is the same as starting with n slots and for each i, choosing the ki slots to be
occupied by objects of type i. This is the same as partitioning the set { I , . . . , n} into
groups of size kl ' . . . , kr , and the number of such partitions is given by the multinomial
coefficient.