INTRODUCTION TO PROBABILITY AND STATISTICS

MAT 222
TOPIC 3

Elementary probability theory

3.0 Introduction
Statistical inference is based on probability theory. The aim of this unit is to introduce to you
basic elements of probability. In the unit you will learn about elementary probability rules. You
will actually learn about multiplication and addition rule. This will be followed by law of total
probability and the Bayes theorem. The Bayes rule provides the starting point for Bayesian
inference where in estimation of parameters there is inclusion of prior knowledge.

3.1 Unit Objectives

By the end of this unit you should be able to:

a) define basic probability terms

b) apply basic probability rules e.g. compliment of an event rule to find probability of an
event.
c) apply multiplication rule to find probability of an event
d) use addition rule to find probability of an event
e) use the Bayes rule to find probability of an event

Key terms
• Probability
• Conditional probability
• Bayes rule

Page 1 of 14
[email protected]
3.2 Basic probability terms

Random Experiment: Is the process of observing the outcome of a chance

event.
Elementary Outcomes: Are all possible results of the random experiment.
Sample Space: Is the set or collection of all the elementary outcomes.

Example
In throwing a coin, sample space is the set S={H,T} where H means head and T means tail.
In throwing a die sample space is the set S={1,2,3,4,5,6}

Example
A family has three children. Write the sample space.

Solution
We may be helped by the tree diagram:

Fig 3.1: Sample space of three children

The sample space consists of eight possibilities. {BBB, BBG, BGB, BGG, GBB, GBG, GGB,
GGG}. The possibility BGB, for example, indicates that the first born is a boy, the second born
a girl, and the third a boy.

Page 2 of 14
[email protected]
Example
Two dice are rolled. Write the sample space.

Solution
We assume one of the dice is red, and the other green. We have the following 36 possibilities.
S=

The entry (2, 5), for example, indicates that the red die shows a two, and the green a 5.

Event: An event is a subset of a sample space or it is an outcome of interest. For example if

you have thrown a coin twice, and you are looking at probability that you have two heads (HH),
then your event is the two heads (HH).

n( E )
Probability: P ( E )
n( S )

Activity3.1
A coin and a die are thrown, write the sample space.

3.3 Basic probability rules

Suppose P(A) is the probability of some event defined by A. P(A) is always in the interval
[0, 1] i.e 0 ≤ 𝑃(𝐴) ≤ 1. Now P(A) = 1 means the event occurs with certainty and P(A) = 0
means that it will certainly not occur. The probability of all events put together must add up to
1, so long as we don’t double-count by including events that overlap. For example, consider a
sample space of an experiment of throwing a coin, . Now p(H)+p(T)=1. The complement of
event A also known as not event A is denoted by ~A (or in other books A/ ). This is the
probability that A will not occur. P(~A) = 1 – P(A), and P(A) + P(~A) = 1.

Page 3 of 14
[email protected]
 Activity 3.2
The metrological department says that probability that it will rain in central region
today is 0.8. What do you think is the probability that it will not rain today?

3.4 Intersection and union of events

The intersection of two events is when two events both happen.

Fig 3.2: Union and intersection of events

We denote union of event A and B by 𝐴 ∪ 𝐵 , sometimes A or B, and we denote intersection

by A ∩ B, sometimes A and B. Probability of the union can be found using this formula:
P(𝐴 ∪ 𝐵) = P(A) + P(B) – P(A ∩ B) Why do we subtract P(A ∩ B) ? To keep us from double-
counting. What if we have union of three events as follows (shaded area):

Fig 3.3: Union of three events

𝑃(𝐴 ∪ 𝐵 ∪ 𝐶 ) = 𝑝(𝐴) + 𝑝(𝐵) + 𝑝(𝐶 ) − 𝑝(𝐴 ∩ 𝐵) - 𝑝(𝐴 ∩ 𝐶 ) − 𝑝(𝐵 ∩ 𝐶 ) + 𝑝(𝐴 ∩ 𝐵 ∩ 𝐶)

In general if we have events 𝐴1 , 𝐴2, 𝐴3 , … … 𝐴𝑛 , then

p(𝐴1 ∪ 𝐴2 ∪ 𝐴3 ∪ … … ∪ 𝐴𝑛 ) = ∑𝑛𝑖=1 𝐴𝑖 − ∑1≤𝑖1≤𝑖2≤𝑛 ,(𝐴𝑖1 ∩ 𝐴𝑖2 ) −

Page 4 of 14
[email protected]
If the two events A and B are mutually exclusive, in the equation
P(A U B) = P(A) + P(B) – P(A ∩ B) meaning they cannot happen at the same time, then the
formula simplifies to P(A U B) = P(A) + P(B).
Note that if event A and B are mutually exclusive their intersection is 0, i.e A ∩ B=0,
see below

Fig 3.4: Mutually exclusive events.

In a more general case if for i=1, 2, 3,.., n are mutually exclusive i.e
their intersection is zero as shown below:

Fig 3.5: Mutually exclusive events

p(𝐴1 ∪ 𝐴2 ∪ 𝐴3 ∪ … … ∪ 𝐴𝑛 ) = 𝑝(𝐴1 ) + 𝑝(𝐴2 ) + 𝑝(𝐴3 ) + ⋯ … + 𝑝(𝐴𝑛 )
,since the intersections are impossible
= ∑𝑛𝑖=1 𝑝(𝐴𝑖 ) , in compact form.
This is a general form of addition rule of probability.

Example
In a class there are 210 students, 60 take Mathematics and Statistics, 40 take Mathematics only,
and 50 take Statistics only, 60 take neither Mathematics nor Statistics. Find

(a) Probability of picking a Mathematics or statistics student at random

(b) Probability of picking a Mathematics student or student who neither take mathematics nor
statistics

Page 5 of 14
[email protected]
Solution
Sketch of events:

(a) Let the event of picking mathematics student be M and that of picking statistics student
be S, then probability of picking Mathematics or statistics student is denoted as p(M or S)
sometimes p(M U S) which is defined as
𝑝( 𝑀 ∪ 𝑆 ) = 𝑝 ( 𝑀 ) + 𝑝( 𝑆 ) − 𝑝( 𝑀 ∩ 𝑆 )
100 110 60
= + −
210 210 210
5
=
7
(b) Let event of neither M nor S be N, we need p(M or N) also denoted as p(M U N) defined
as p(M U N)=p(M)+p(N)-p(M and N)
=p(M)+p(N), because event M and N is impossible
100 60
= 210 + 210

16
=
21

Page 6 of 14
[email protected]
 Activity 3.3
Let E1 be an event a cow gives birth to male cow and E2 be an event a cow gives birth
to a female cow, how can we find 𝑝(𝐸1 ∪ 𝐸2)

3.5 Conditional Probabilities

A conditional probability tells you the probability of some event given that you already know
that another event has occurred. The formula for conditional probability of A given B is:
𝑝( 𝐴 ∩ 𝐵 )
𝑃 (𝐴 |𝐵 ) =
𝑃 (𝐵 )

What about conditional probability of B given A denoted as P(B|A)?

𝑃(𝐵 ∩ 𝐴)
𝑃 (𝐵 |𝐴 ) =
𝑃 (𝐴 )

Example

Consider situation below

Find probability of picking a mathematics pupil given that he also take statistics
Solution
𝑃 (𝑀 ∩ 𝑆 ) 60 110
𝑃 (𝑀|𝑆) = = ÷
𝑃 (𝑆 ) 210 210
6
=
11
Page 7 of 14
[email protected]
Example
The following table shows the distribution by gender of students at DCE Campus who use public
transport and the ones who drive to school.

Male(M) Female(F) Total

Public Transportation(P) 8 13 21
Drive(D) 39 40 79
Total 47 53 100

The events M, F, P, and D are self explanatory. Find the following probabilities.
a. P(D | M) b. P(F | D) c. P(M | P)
Solution

𝑃(𝐸∩𝐹)
We use the conditional probability formula 𝑃(𝐸 |𝐹 ) = 𝑃(𝐹)

𝑃(𝐷∩𝑀) 39 47
a. 𝑃 (𝐷|𝑀) = = 100 ÷ 100
𝑃(𝑀)

39
= 47

𝑃(𝐹∩𝐷) 40 79
b. P(F | D) = = 100 ÷ 100
𝑃(𝐷)

40
= 79 .

𝑃(𝑀∩𝑃) 8 21
c. P(M | P) = = 100 ÷ 100
𝑃(𝑃)
8
= 21 .

Page 8 of 14
[email protected]
Multiplication Rule
Events are called independent if the occurrence of one does not affect the probability of the other.
That is, P(A|B) = P(A).

𝑃(𝐴∩𝐵)
So P(A | B) = = 𝑃(𝐴)
𝑃(𝐵)
𝐵𝑢𝑡 𝑃(𝐴 ∩ 𝐵) = 𝑃(𝐴) ⋅ 𝑃(𝐵)
∴ P(A | B) = P(A) for independent events

This would mean, P(A ∩ B) = P(A) ∙ P(B). Thus the multiplication rules says that if A and B
are independent events, then you can multiply the probabilities together to get the joint
probability (the probability of the intersection). That is, P(A ∩ B) = P(A) ∙ P(B)

In general if events A_i for i=1, 2, 3,….,n are independent, then

p(𝐴1 ∪ 𝐴2 ∪ 𝐴3 ∪ … … ∪ 𝐴𝑛 ) = 𝑝(𝐴1 ) × 𝑝(𝐴2 ) × 𝑝(𝐴3 ) × … … × 𝑝(𝐴𝑛 )

Example
A die and a coin are thrown together once, what is the probability of getting an even number on
a die and a head on a coin.

Solution
Let event of even on die be denoted as E and head on coin as H. Then we need event E and H
also denoted as E ∩ H. But these two events are independent, that is, can not
influence each other since are occurring on two different objects. Thus
P(E ∩ H)=P(E)×P(H)
=3/6×1/2
1
=4
Example
The probability that Jaime will visit his aunt in Mzuzu this year is 0.30, and the probability that
he will go river rafting on Rukuru river is 0.50. If the two events are independent, what is the
probability that Jaime will do both?

Solution:
Let A be the event that Jaime will visit his aunt this year, and R be the event that he will go
river rafting. We are given P(A) = 0.30 and P(R) = 0.50, and we want to find P(A R). Since
we are told that the events A and R are independent, P(A R) = P(A)× P(R) = (0.30)(0.50) =
0.15.

Page 9 of 14
[email protected]
Example
Given P(B | A) = 0.4. If A and B are independent, find P(B).

Solution
If A and B are independent, then by definition P(B | A) = P(B). Therefore, P (B) =0 .4

Activity 3.4
A cow gives birth twice in ten years. How can you get probability of a female birth at second
birth given a male birth at first birth?

3.6 Law of total probability and Bayes rule

3.6.1 Law of total probability

Let the sample space S be partitioned by events A1, A2 , . . . An. i. e

S = A 1 ∩ A2 ∩.......∩ An and Ai ∩ Aj =∅ for i ≠ j. Let B, be an event in S such that B ∩

Ai ≠ ∅ , that is,

Fig 3.6: Partition of S by events

We have p(B)=p(B|A1)p(A1)+p(B|A2)p(A2)+……+p(B|An)p(An)

Page 10 of 14
[email protected]
Prove
𝑃(𝐵) = p(B|A1)p(A1)+p(B|A2)p(A2)+……+p(B|An)p(An),
proof
Note 𝐵 = (𝐵 ∩ 𝐴1 ) ∪ (𝐵 ∩ 𝐴2 ) ∪ (𝐵 ∩ 𝐴3 ) ∪ … .∪ ( 𝐵 ∩ 𝐴𝑛 )
𝑃(𝐵) = 𝑝 (𝐵 ∩ 𝐴1 ) + 𝑝(𝐵 ∩ 𝐴2 ) + 𝑝 (𝐵 ∩ 𝐴3 ) + ⋯ . . + p( 𝐵 ∩ 𝐴𝑛 )
𝑃(𝐵∩𝐴)
𝑏𝑢𝑡 𝑃 (𝐵|𝐴) = 𝑃(𝐴)

𝑃(𝐵 ∩ 𝐴) = 𝑃(𝐵|𝐴) ⋅ 𝑃(𝐴)

𝑡ℎ𝑒𝑟𝑒𝑓𝑜𝑟𝑒 𝑤𝑒 ℎ𝑎𝑣𝑒
𝑃(𝐵) = p(B|A1)p(A1)+p(B|A2)p(A2)+……+p(B|An)p(An),
. =∑𝑛𝑖=1 𝑝(𝐵|𝐴𝑖 )𝑝 (𝐴𝑖 ) in compact form
In law of total probability the total probability of an event B depends on probability of all other
events that results into its occurrence.

Example
The following contingency table gives the results of operations in a hospital according to the
complexity of the operation.
Simple Complex Total
Successful 1990 950 2940
Unsuccessful 10 50 60
Total 2000 1000 3000
Solution
Let A be the event that an operation is simple and B the event that an operation is successful
and C be the event that the operation s unsuccessful
P (A)=P(A|B)×P(B)+P(A| C )×p(C) , because for A to occur its either due to occurrence of B
or not B.
𝑃 (𝐴) = 𝑃(𝐴|𝐵) × 𝑃 (𝐴) + 𝑃(𝐴|𝐶) × 𝑃(𝐶)
1990 2940 10 60
= × + ×
3000 3000 3000 3000

1219
=
1875

Page 11 of 14
[email protected]
3.6.2 Bayes rule
The Bayes rule of probability was coined by Thomas Bayes (1761).
Now B may be occur after occurrence of A1, A2, A3,….., or An and thus by law of total probability
P(B)= p(B|A1)p(A1)+p(B|A2)p(A2)+……+p(B|An)p(An).
Now suppose B has occurred, after Ak occurred, we want probability of any of the event Ak for
k=1,2,3,…,n that made B to occur given that B has occurred, denoted as p(Ak|B). This probability is
found by the Bayes Theorem:
𝑃 (𝐵 |𝐴 𝐾 )𝑃 (𝐴 𝑘 )
𝑃 (𝐴 𝑘 |𝐵 ) =
𝑃 (𝐵 )
𝑃(𝐵|𝐴𝑘 )𝑃(𝐴𝑘 )
= 𝑛
∑𝑖=1 𝑃(𝐵|𝐴𝑖 )𝑃(𝐴𝑖 )

Note p(Bt) has to be total probability of B. The Bayes theorem seeks to find the probability of
cause (𝐴𝑘 ) given the effect (B).

Example
In a binary communication system a zero and a one are transmitted with probability 0.6 and 0.4
respectively. Due to error in the communication system a zero becomes a one with a probability
0.1 and a one becomes a zero with a probability 0.08. Determine the probability
(i) of receiving a one
(ii) that a one was transmitted when the received message is one.

Solution
Let S be the sample space corresponding to binary communication. Suppose T0 be event of transmitting
0 and T1 be the event of transmitting 1 and R0 and R1 be corresponding events of receiving 0 and 1
respectively.
Given:

𝑃(𝑇0 ) = 0.6 , 𝑃 (𝑇1 ) = 0.4 , 𝑃(𝑅1 |𝑇0 ) = 0.1 𝑃(𝑅0 |𝑇1 ) = 0.08
𝑃(𝑅0 |𝑇0 ) = 0.9 , 𝑃(𝑅1 |𝑇1 ) = 0.92
By using total probability
i 𝑃(𝑅1 ) =, 𝑃(𝑅1 |𝑇0 )𝑃(𝑇0 ) + 𝑃(𝑅1 |𝑇1 )𝑃(𝑇1 )
= 0.1 × 0.6 + 0.92 × 0.4
= 0.428
by using Bayes rule
ii P(Sent |received)
𝑃 (𝑅1 |𝑇1 ) × 𝑃(𝑇1 )
𝑃(𝑇1 |𝑅1 ) =
𝑃 (𝑅1 )

Page 12 of 14
[email protected]
 0.92 × 0.4
=
0.428
= 0.8598

Example
In an electronics laboratory, there are identically looking capacitors of three makes
A1, A2 and A3 in the ratio 2:3:4. It is known that 1% of A1,1.5% of A2 and 2% of A3 are defective.
a. What percentage of capacitors in the laboratory are defective?
b. If a capacitor picked at defective is found to be defective, What is the probability it is of
make A3 ?
Solution
Let D be the event that the item is defective. Here we have to find :
𝑃(𝐷) and 𝑃(𝐴3 |𝐷)
2 3 1 4
We are given 𝑃(𝐴1 ) = 9 , 𝑃(𝐴2 ) = 9 = 3 , 𝑃(𝐴3 ) = 9
𝑃(𝐷|𝐴1 ) = 0.01
𝑃(𝐷|𝐴2 ) = 0.015
𝑃(𝐷|𝐴3 ) = 0.02
a. 𝑃(𝐷) = 𝑃(𝐷|𝐴1 ) × 𝑃(𝐴1 ) + 𝑃(𝐷|𝐴2 ) × 𝑃(𝐴2 ) + 𝑃(𝐷|𝐴3 ) × 𝑃(𝐴3 )
2 1 4
= 0.01 × + 0.015 × + 0.02 ×
9 3 9
= 0.0161
𝑃(𝐷|𝐴3 )×𝑃(𝐴3 )
b. 𝑃(𝐴3 |𝐷) =
𝑃(𝐷)
4
0.02 ×
= 9
0.0161
= 0.552

example
a factory uses 3 machines 𝑥 , 𝑦, 𝑎𝑛𝑑 𝑧 to produce lighting bulbs. Suppose
machine x produces 50% of the bulbs which are 3 % defective, machine y
produces 30% of the bulbs which are 4% defective and machine z produces 20%
of the bulbs which are 5% defective.
a. Find the probability that a randomly selected bulb is defective.
b. Find the probability that a given defective bulb found was produced
by machine y

Page 13 of 14
[email protected]
 Given 𝑃(𝑥 ) = 0.5, 𝑃(𝑦) = 0.3, 𝑃(𝑧) = 0.2
𝑃(𝐷|𝑥 ) = 0.03, 𝑃(𝐷|𝑦) = 0.04, 𝑃(𝐷|𝑧) = 0.05
a. 𝑃(𝐷 ) = 𝑃(𝐷|𝑥 ) × 𝑃(𝑥 ) + 𝑃(𝐷|𝑦) × 𝑃 (𝑦) + 𝑃(𝐷|𝑧) × 𝑃(𝑧)
= 0.03 × 0.5 + 0.04 × 0.3 + 0.05 × 0.2
= 0.037
𝑃(𝐷 |𝑦 )×𝑃(𝑦)
b. 𝑃(𝑦|𝐷 ) =
𝑃(𝐷)

0.04 × 0.3
=
0.037

= 0.324

Activity 3.5
A chicken has birds’ flue in Malawi if it is from Madagascar or
Indonesia. It is known that 1% chickens are from Madagascar and 15%
chickens are from Indonesia. The probability that a chicken has flue
that it is from Madagascar is 0.2 and that it has flue given it is from
Indonesia is 0.06. Find probability that chicken has flue.

Practice activity
1. In a recent survey in a Statistics class, it was determined that only 60% of the students
attend class on Thursday. From past data it was noted that 98% of those who went to
class on Thursday pass the course, while only 20% of those who did not go to class on
Thursday passed the course.
(a) What percentage of students is expected to pass the course?
(b) Given that a student passes the course, what is the probability that he/she attended
classes on Thursday?

2. A prisoner has just escaped from jail. There are three roads leading away from the jail. If the
prisoner selects road A to make good her escape, the probability that she succeeds is 0.25. If
she selects road B, the probability that she succeeds is 0.2. If she selects road C, the
probability that she succeeds is (1/6). Furthermore, the probability that she selects each of
these roads is the same. It is (1/3). If the prisoner succeeds in her escape,
What is the probability that she made good escape by using road B?

Page 14 of 14
[email protected]