CHAPTER 1: PROBABILITY

January 12, 2024

1.1 Properties of Probability

Definition. The collection of all possible distinct outcomes of an experiment is called the
sample space, denoted by S.
Example. The sex of the first two newborns in town tomorrow.

• S = {BB, BG, GB, GG}

Example. Let each of 10 persons taste a cup of instant coffee and a cup of percolated
coffee. Record how many people prefer the instant coffee.

• S = {0, 1, 2, · · · , 10}

Example. Administer an antibiotic to patients suffering from a viral infection until one has
an adverse reaction.

• S = {R, N R, N N R, N N N R, · · · }, R for“adverse reaction”, N for “no adverse reac-

tion”.

Definition. A collection of outcomes in S is called an event. An event is a subset of the

sample space S. We say that event A has occurred if the outcome of the experiment is
in A.

Definition (Operations with events).

• ∅ denotes the empty set.

• A ⊂ B means A is a subset of B (not necessarily a proper subset, that is, A may be

equal to B).

• Union: A ∪ B = {x : x ∈ A or x ∈ B}

• Intersection: A ∩ B = {x : x ∈ A and x ∈ B}.

1
 • Complement: A0 = {x : x 6∈ A}.

Properties.

• Commutative Laws:
A ∪ B = B ∪ A, A ∩ B = B ∩ A.

• Associative Laws:
A ∪ (B ∪ C) = (A ∪ B) ∪ C, A ∩ (B ∩ C) = (A ∩ B) ∩ C.

• Distributive Laws:
A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C), A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C)

• De Morgan’s Laws:
(A ∪ B)0 = A0 ∩ B 0 , (A ∩ B)0 = A0 ∪ B 0

Definition. Two events A and B are disjoint (mutually exclusive) if A ∩ B = ∅ (in

words, A and B can not both occur at the same time).
Definition. The events A1 , A2 , · · · , An are mutually exclusive if Ai ∩ Aj = ∅, ∀i 6= j.
Definition. If the events A1 , A2 , · · · , An are mutually exclusive and ∪ni=1 Ai = S (called
exhaustive), then the collection A1 , A2 , · · · , An forms a partition of S.

Definition (Naive definition of probability). If a sample space consists of k outcomes that

are all equally likely to occur and an event A consists of m of these k elements, then
m
P (A) = .
k
Note that for this way of computing probability, we need the special assumption of finite
sample space and equally likely elements. A sample space possessing such properties is
called a uniform probability model.
Example. Tossing a balanced die.
Definition (General definition of probability). Probability is a function, defined on events,
that satisfies the following conditions:

(i) For all events A, 0 ≤ P (A) ≤ 1.

(ii) P (∅) = 0, P (S) = 1.

(iii) If A1 , A2 , A3 , . . . are events such that Ai ∩ Aj = ∅, ∀ i 6= j, then

P (A1 ∪ A2 ∪ A3 ∪ · · · ) = P (A1 ) + P (A2 ) + P (A3 ) + · · ·

2
Note that everything below follows from this definition of probability.

Theorem.

• P (A0 ) = 1 − P (A).

• If A ⊂ B, then P (A) ≤ P (B).

• (Addition Rules) P (A ∪ B) = P (A) + P (B) − P (A ∩ B).

P (A∪B ∪C) = P (A)+P (B)+P (C)−P (A∩B)−P (A∩C)−P (B ∩C)+P (A∩B ∩C).

1.2 Methods of Enumeration

Multiplication Principle: When an experiment consists of two parts such that the first part
can have n1 distinct results, and if with each result of the first part there can be n2 distinct
results of the second part, then the total number of possible results of the experiment is
n1 × n2 . This generalizes to k parts with n1 , n2 , . . . , nk distinct results in each part and
the total number of possible results is n1 n2 · · · nk .
Example. When preparing a lawsuit on a patent infringement, a company has a choice of
three lawyers and also a choice of one of two courts in which to initially try the case. Then
there are totally six possibilities.

Permutations. The number of different orderings or arrangements that can be formed with
r objects selected from a group of n distinct objects (1 ≤ r ≤ n) is denoted by n Pr , and
can be computed by the multiplication principle as
n!
n Pr = n(n − 1)(n − 2) · · · (n − r + 1) = .
(n − r)!

This is the same as the number of ways to sample r from n distinct objects without
replacement, when the order matters. (By convention, n P0 = 1.)

Combinations. The number of unordered subsets of size r of n distinct objects (0 ≤ r ≤ n)

is denoted by n Cr or nr , read as “n choose r”. It can be computed as

n Pr n!
n Cr = = .
r! r!(n − r)!

This is because each unordered subset is counted r! times in the permutation.

Moreover, we have n Cr =n Cn−r .
By convention, if r < 0 or r > n, then n Pr =n Cr = 0.

3
Example. There are 15 people competing in a bicycle race. In how many ways can the
first, second, and third prizes be awarded to persons participating in the race? (Answer:
15 × 14 × 13)
Example. An urn contains 12 articles, 8 of which are good (marked G1 , G2 , · · · , G8 ), and
4 of which are defective (marked D1 , D2 , D3 , D4 ). Suppose that 3 articles are to be drawn
from the urn one after another without replacement, and the results of the individual draws
are to be recorded.

• Keeping track of the order of selections, how many ordered samples are possible?
(Answer: 12 × 11 × 10)

• How many ordered samples are possible under the condition that the first two draws
produce defective articles and the third yields a good piece? (Answer: 4 × 3 × 8)

• If the articles are draw at random, what is the probability of obtaining defectives in
4×3×8
the first two draws and a good piece in the third draw? (Answer: 12×11×10 )

1.3 Conditional Probability

The probability of an event A needs to be modified after information is obtained as to
whether or not a related event B has taken place.
Definition. Whenever P (B) > 0, the conditional probability of A given B is defined
as
P (A ∩ B)
P (A|B) =
P (B)

The following proposition is a direct consequence of the above definition.

Proposition (Multiplication rule). We have, provided that P (B) > 0,

P (A ∩ B) = P (A|B)P (B).

Moreover, provided that P (B ∩ C) > 0,

P (A ∩ B ∩ C) = P (A|B ∩ C)P (B|C)P (C).

1.4 Independent Events

The following relation between events are of particular interest.
Definition. Two events A and B are independent if

P (A ∩ B) = P (A) × P (B).

4
Equivalently, if A and B are independent and P (B) > 0, then we have

P (A|B) = P (A),

that is, information about the occurrence of B has no bearing on the assessment of the
probability of A.
Definition. We say three events A, B and C are mutually independent if

P (A ∩ B) = P (A)P (B), P (A ∩ C) = P (A)P (C), P (B ∩ C) = P (B)P (C),

and
P (A ∩ B ∩ C) = P (A)P (B)P (C).
Fact. If A and B are independent, then A and B 0 are also independent.
Example. Two doctors (A and B) test all patients for the same disease. Doctor A diagnoses
10% of the patients as positive. Doctor B diagnoses 17% of the patients as positive. Both
doctors diagnose 8% of the patients as positive.
• Are the events
A = {Dr. A makes a positive diagnosis}
and
B = {Dr. B makes a positive diagnosis}
independent? (Answer: No.)

• What is the probability that Dr. A gives a positive diagnosis given Dr. B has given
8%
a positive diagnosis? (Answer: 17% )

1.5 Bayes’ Theorem

To prepare for Bayes’ Theorem, we first need the following result:
Theorem (Law of Total Probability). We have, if P (B) > 0 and P (B 0 ) > 0, then

P (A) = P (A|B)P (B) + P (A|B 0 )P (B 0 ).

In general, let B1 , B2 , . . . , Bn be a partition of the sample space such that P (Bi ) > 0 for
all i, and let A be any event, then
n
X
P (A) = P (A|Bi )P (Bi )
i=1

Example. The rate of diabetes among Caucasians is 7%, among African-Americans is 10%,
Hispanics 12% and Asian-Americans is 5% The ethnic distribution in an American town

5
is 30% Caucasian, 25% African-American, 40% Hispanic and 5% Asian-American. What
is the rate of diabetes among the population of the American town?
(Answer: 7% × 30% + 10% × 25% + 12% × 40% + 5% × 5%)

Now we are ready to state Bayes’ Theorem.

Theorem (Bayes’ Theorem). We have, if P (B) > 0 and P (B 0 ) > 0, then

P (A|B)P (B)
P (B|A) = .
P (A|B)P (B) + P (A|B 0 )P (B 0 )

In general, let B1 , B2 , · · · , Bn be a partition of the sample space such that P (Bi ) > 0 for
all i, and let A be any event, then

P (A|Bi )P (Bi )
P (Bi |A) = Pn
j=1 P (A|Bj )P (Bj )

Example. 84% of people with high blood pressure are diagnosed as having high blood
pressure by a machine. 23% of people with normal blood pressure are diagnosed as having
high blood pressure by a machine. 20% of people in the general population have high blood
pressure. Compute P (B|A),where

B = {having high blood pressure},

A = {diagnosed as having high blood pressure by the machine}.

P (A|B)P (B) 84%×20%
(Answer: P (B|A) = P (A|B)P (B)+P (A|B 0 )P (B 0 ) = 84%×20%+23%×80% )

Example (The game show problem). (This is more challenging and is optional material.)
https://www.youtube.com/watch?v=XXcv3RHuROU

• Define events
A = {The car is behind door 1},
B = {The car is behind door 2},
C = {The car is behind door 3},
W = {The game host opens door 3}.

• Try to find P (A|W ), P (B|W ) and P (C|W ).