Asossa University

College of Engineering
Electrical and Computer Engineering Department

Post-Graduate (PG) Program

Probability and Stochastic Processes

Review of Basic Probability Theory

 Basic Concepts of Probability Theory
Outline
 Introduction
 Sample Space and Events
 Basic Set Operations
 Axioms and Properties of Probability
 Conditional Probability
 Independence of Events

2 January 2023 Lamessa .D 2

 Introduction
 Probability is the study of randomness and uncertainty.
 Probability theory provides powerful mathematical tools to
explain, model and design an efficient, reliable and cost
effective real world physical systems with some degree of
uncertainty.
 Some application areas of probability theory include:
• Data communication systems
• Wireless communication systems
• Control and power systems, etc…

 Events can in general be classified as:

• Random-the event may or may not occur
• Certain- the occurrence of the event is inevitable
2 January
• 2023 Lamessa .D 3
Impossible-the event will never occur
Sample Space and Events
i. Random Experiment
 A random experiment is an experiment in which the
outcome varies in an unpredictable manner when the
experiment is repeated under the same conditions.
Examples:
• Tossing a coin
• Rolling a die
• Picking a card from a deck
ii. Sample Space
 The sample space is the set of all possible outcomes of
a random experiment.

2 January 2023 Lamessa .D 4

Sample Space and Events Cont’d….

 The sample space is denoted by Ω and the possible

i
outcomes are represented by
  { 1 , 2 , ...... n }

iii. Event
 An event is any subset of the sample of the sample
space, Ω
 Events can be represented by A, B, C, ……
Example-1:
Consider a random experiment of rolling a die once.
i. Sample
 Space
{ 1, 2, 3, 4, 5, 6}

2 January 2023 Lamessa .D 5

Sample Space and Events Cont’d….
ii. Some possible events
 An event of obtaining even numbers
A  { 2, 4, 6}

 An event of obtaining numbers less than 4

B  { 1, 2, 3}

Example-2:
Consider a random experiment of flipping a fair coin
twice.
i. Sample space
  { HH , HT, TH, TT }

2 January 2023 Lamessa .D 6

Sample Space and Events Cont’d….

ii. Some possible events

 An event of getting exactly one head
A { HT , TH }

 An event of getting at least one tail

B { HT , TH, TT }

 An event of getting at least one head

C { HH , HT , TH }

2 January 2023 Lamessa .D 7

 Basic Set Operations
 We can combine events using set operations to obtain other
events.
1. Union
The union of two events A and B is defined as the set of
outcomes that are either in A or B or both and is
A denoted
B. by

A  B  {  :   A or   B }  A  B { :   A   B}
Ω
A B

E F

AB

2 January 2023 Lamessa .D 8

 Basic Set Operations Cont’d…..
2. Intersection
The intersection of two events A and B is defined as the set
of outcomes that are common to both A and B and is
denoted
A  B . by

A  B  {  :   A and   B }  A  B { :   A   B}

A B

AB

2 January 2023 Lamessa .D 9

Basic Set Operations Cont’d…..
3. Complement
The complement of an event A is defined as the set of all
outcomes that are not in A and is denotedA by
.

A { :   and   A}  A  {  :       A}
Ω

EA A

2 January 2023 Lamessa .D 10

 Basic Set Operations Cont’d…..
4. Mutually Exclusive (Disjoint) Events
Two events A and B are said to be mutually exclusive or
disjoint if A and B have no elements in common, i.e.,
A B 

A B

A B 

5. Equal Events
Two events A and B are said to equal if they contain the
same outcomes and is denoted by A=B.

2 January 2023 Lamessa .D 11

 Some Properties of Set Operations
1. Elementary Properties
i.  v. A A 
ii .  vi . A A 
iii .  A   vii . A  A
iv .  A  A

2. Commutative Properties
A B  B A
A B  B A
3. Associative Properties
A (B C )  ( A  B) C
A (B C )  ( A  B) C

2 January 2023 Lamessa .D 12

Some Properties of Set Operations Cont’d…..
4. Distributive Properties
A (B C )  ( A  B) ( A C )
A (B C )  ( A  B) ( A C )
5. DeMorgan’s Rules
(A B)  A B
(A B)  A B

 The union and intersection operations can be repeated

for an arbitrary number of events as follows.
n

A
i 1
i
 A1  A 2  ....  A n
n

A
i 1
i
 A1  A 2  ...  A n

2 January 2023 Lamessa .D 13

Axioms and Properties of Probability
 Probability is a rule that assigns a number to each event A
in the sample space, Ω.
 In short , the probability of any event A is given by
n( A)
P ( A) 
n ( )
where

n ( A ) - is the number of elements in the event A

n ( ) - is the number of elements in the sample space 

2 January 2023 Lamessa .D 14

 Axioms and Properties of Probability Cont’d…..

 The probability of an event A is a real number which

satisfies the following axioms.
1. Probability is a non-negative number, i.e.,
P ( A) 0
2. Probability of the whole set is unity, i.e.,

1 P ( )
From axioms (1) and (2), we obtain

0  P ( A) 1

2 January 2023 Lamessa .D 15

Axioms and Properties of Probability Cont’d…..

3. Probability of the union of two mutually exclusive

(disjoint) events is the sum of the probability of the
events,
If A i.e.,
B  , then P ( A  B )  P ( A )  P ( B )

 We can generalize axiom (3) for n pairwise mutually

exclusive (disjoint) events.
 If A1, A2, A3, …, An is a sequence of n pairwise mutually
Aexclusive
i
(disjoint)
 A j  , for i  j , thenevents in the sample space Ω such
that  n  n
P   A i    P ( A i )
 i 1  i 1
2 January 2023 Lamessa .D 16
Axioms and Properties of Probability Cont’d…..
 By using the above probability axioms, other useful
properties of probability can be obtained.

1. P ( A) 1  P ( A)
Proof:

A  A    P ( A  A )  P ( A )  P ( A ), but A A 
 P ( )  P ( A )  P ( A ) , P ( )  P ( A  A)
 1  P ( A )  P ( A ), P ( ) 1
P ( A) 1  P ( A)

2 January 2023 Lamessa .D 17

Axioms and Properties of Probability Cont’d…..
 We can decompose the events A, B and AUB as unions of
mutually exclusive (disjoint) events as follows.

A B

A B A B A B

We can see that A B, A B and A B are disjoint events.

2 January 2023 Lamessa .D 18

Axioms and Properties of Probability Cont’d…..

 From the above Venn diagram, we can write the following

relations.
i. A  (A  B) (A  B) ii . B  (A  B) (A  B)

 P ( A)  P(A B) P(A B)  P(B)  P( A  B)  P( A  B)

 P( A  B)  P(B)  P( A  B)
 P(A B)  P ( A) P(A B)

iv . A  B  B (A  B)
iii . A B  A (A  B)
 P( A  B)  P(B)  P( A  B)
 P(A B)  P ( A) P(A B)

v. A  B  (A  B) (A  B) (A  B)
 P(A B)  P(A B) P(A B) P(A B)

2 January 2023 Lamessa .D 19

Axioms and Properties of Probability Cont’d…..

2. P(A  B )  P ( A)  P (B )  P ( A  B )

Proof:
P(A  B )  P ( A)  P ( A  B )
But, P ( A  B)  P(B)  P( A  B)
P ( A  B )  P ( A)  P (B )  P ( A  B )

 We can generalize the above property for three events A, B

and C as follows.
P(A  B  C )  P ( A )  P ( B )  P (C )  P ( A  B )  P ( A  C )  P ( B  C )
 P( A  B C )

2 January 2023 Lamessa .D 20

Axioms and Properties of Probability Cont’d…..

 For n events A1, A2, A3,…,An the above property can be

generalized as:
 n  n n

P   A i    P ( A j )   P ( A j  A k )  ....  ( 1) P ( A1  A 2  ...  A n )

n

 i 1  j 1 j k

3. P(A  B )  P ( A)  P (B )
Proof:
P(A  B )  P ( A)  P (B )  P ( A  B )
But, P ( A B) 0
P ( A  B )  P ( A)  P (B )

2 January 2023 Lamessa .D 21

Axioms and Properties of Probability Cont’d…..
Example-1:
A box contains 10 identical balls numbered 0, 1, 2,…,9. A
single ball is selected from the box at random. Consider the
following events.
A: number of ball selected is odd
B: number of ball selected is multiple of 3
C: number of ball selected is less than 5
Find the following probabilities.
a. P ( A) d. P ( A B)
b. P(B) e. P(A  B C )
c. P (C )

2 January 2023 Lamessa .D 22

Axioms and Properties of Probability Cont’d…..
Solution:
 The sample space and the events are given by:

  { 0, 1, 2, 3, 4, 5, 6, 7, 8, 9} C  { 0, 1, 2, 3, 4}

A  { 1, 3, 5, 7, 9} A  B  { 3, 9}
B  { 3, 6, 9} A  B  C  { 0, 1, 2, 3, 4, 5, 6, 7, 9}

 The number of elements in the sample space and events

are:
n ( )  10 n (C )  5
n( A) 5 n( A B)  2
n( B ) 3 n( A  B  C )  9

2 January 2023 Lamessa .D 23

Axioms and Properties of Probability Cont’d…..
 Thus, probabilities of the given events are given by:
n( A) 5 1 B) 2 1
n( A
a. P ( A )    d. P(A B)   
n ( ) 10 2 n ( ) 10 5
n( B ) 3 n( A  B  C ) 9
b. P ( B )   e. P( A  B C )  
n ( ) 10 n ( ) 10
n (C ) 5 1
c. P ( C )   
n ( ) 10 2
Example-2:
Given P ( A)  0 .9, P(B)  0 . 8 and P(A  B )  0 . 75 , find :

a. P ( A B) c. P ( A B) e. P ( A B)

b. P ( A B) d. P(A B) f . P(B)

2 January 2023 Lamessa .D 24

Axioms and Properties of Probability Cont’d…..
Solution:
a. P(A B)  P ( A)  P (B )  P ( A  B ) d. P(A B)  P ( A  B ) 1 P ( A  B )
 P ( A  B )  0 . 9  0 . 8  0 . 75  P ( A  B )  1  0 . 75
 P ( A  B )  0 . 95  P ( A  B )  0 . 25

b. P(A B)  P ( A)  P ( A  B ) e. P(A B) 1  P ( A)  P ( A  B )

 P ( A  B )  0 . 9  0 . 75  P ( A  B )  1  0 . 9  0 . 75
 P ( A  B )  0 . 15  P ( A  B )  0 . 85

c. P(A  B )  P ( A  B ) 1 P ( A  B ) f. P(B) 1 P (B )

 P ( A  B )  1  0 . 95  P ( B )  1  0 .8
 P ( A  B )  0 . 05  P ( B )  0 .2

2 January 2023 Lamessa .D 25

Axioms and Properties of Probability Cont’d…..
Exercise:
1 . If A  B  , then show that P ( A )  P ( B ).
2 . If P ( A )  P ( B )  P ( A  B ) , then show that

P [( A  B )  ( A  B )]  0 .
3 . If P ( A )  P ( B )  1, then show that P ( A  B )  1.
4 . If P ( A )  0 . 9 and P(B)  0 . 8 , then show that P ( A  B )  0 .7

2 January 2023 Lamessa .D 26

Conditional Probability ----some correction made
 The conditional probability of an event A given B, denoted
by P(A/B), is defined as:
P(A B)
P(A /B)  , P(B) 0 (1)
P(B)

 Similarly, the conditional probability of an event B given A,

denoted by P(B/A), is given by

P(A B)
P (B / A)  , P ( A) 0 (2)
P ( A)

 From equations (1) and (2), we will get

P(A  B)  P( A /B)P(B)  P(B / A)P ( A) (3)

2 January 2023 Lamessa .D 27

Conditional Probability Cont’d…….
 Then using equation (3), we will get

P (B / A)P ( A) P( A /B)P(B)
P(A /B)  OR P (B / A)  (4)
P(B) P ( A)

 We know that
P(B)  P ( A  B )  P ( A  B)
 P(B)  P(B / A)P ( A)  P(B / A)P ( A) (5)

 Substituting equation (5) into equation (4), we will get

P (B / A)P ( A)
P(A /B)  (6)
P (B / A)P ( A)  P(B / A)P ( A)

2 January 2023 Lamessa .D 28

Conditional Probability Cont’d…….
 Similarly,
P( A /B)P(B)
P (B / A)  (7)
P( A /B)P(B)  P( A /B)P(B)

 Equations (6) and (7) are known as Baye’s Rule.

 Baye’s Rule can be extended for n events as follows.
 Let events A1, A2, A3, …, An be pairwise mutually exclusive
(disjoint ) events and their union be the sample space Ω, i.e.
n

Ai  Aj   and A
i 1
i


 n  n
 P   A i    P ( A i )
 i 1  i 1
2 January 2023 Lamessa .D 29
Conditional Probability Cont’d…….
 Let B be any event in Ω as shown below.

A2 .....
A1 A n 1

A3 An
.....

B  B  ( A1  A 2  ....  A n )

 B  ( B  A1 )  ( B  A 2 )  ...  ( B  A n )

But, A i  A j    ( B  Ai )  ( B  A j )  

2 January 2023 Lamessa .D 30

Conditional Probability Cont’d…….
 The eventsB  A i
and B  Aj are mutually exclusive events.
 P ( B )  P ( B  A1 )  P ( B  A 2 )  ...  P ( B  A n )

 P(B)  P(B / A1 ) P ( A1 )  P(B / A2 ) P ( A2 )  ...  P ( B / An ) P ( An ) (8)

 In short,
n n

P(B)   P ( B  Ai )   P ( B / Ai ) P ( Ai ) (9)
i 1 i 1

 Then using equation (7), we will obtain

P ( B / Ai ) P ( Ai )
P ( Ai / B )  n
(10)

P(B / A )P( A )
i 1
i i

2 January 2023 Lamessa .D 31

Conditional Probability Cont’d…..
Example-1:
Show that P ( A / B ) 1 P ( A /B )

Solution:

P(B)  P(A B) P(A B)

 P(B)  P( A /B)P(B)  P( A /B)P(B)

Dividing both sides by P ( B ) , we obtain

1  P(A /B)  P ( A /B )

 P ( A /B )  1  P ( A / B )

2 January 2023 Lamessa .D 32

Conditional Probability Cont’d…..
Example-2: correction has made to it
Let A and B be two events such that P(A)=x, P(B)=y and
P(B/A)=z. Find the following probabilities in terms of x, y
and z.
a. P ( A / B )

b. P(A B)
c. P(A /B)

Solution:
P(A  B)  P(B / A)P ( A)  xz
P(A B) xz
a. P(A /B)  
P(B) y

b. P(A  B )  P ( A  B )  1  P ( A  B )  1  xz
P(A B) P(B) P(A B) xz
c. P(A /B)   1
P(B) P(B) y

2 January 2023 Lamessa .D 33

Conditional Probability Cont’d…..
Example-3:
A box contains two black and three white balls. Two balls
are selected at random from the box without replacement.
Find the probability that
a. both balls are black
b. the second ball is white

Solution:
First let us define the events as follows:

B 1 : the outcome in the first selection is a black ball

2 January 2023 Lamessa .D 34

Conditional Probability Cont’d…..

B 2 : the outcome in the second selection is a black ball

W 1 : the outcome in the first selection is a white ball

W 2 : the outcome in the second selection is a white ball

P ( B1 )  2 /5 P ( B 2 / B1 ) 1/4 P (W 2 / B 1 )  3 /4
P (W 1 )  3 /5 P ( B 2 /W 1 )  2 /4 P (W 2 / W 1 )  2 /4

a. P ( B1  B2)  P ( B 2 /B 1 ) P ( B 1 )  (1 / 4 )( 2 / 5 )
 P ( B 1  B 2 )  1 /10
b. P (W 2 )  P (W 2  B 1 )  P (W 2  W 1 )
 P (W 2 / B 1 ) P ( B 1 )  P (W 2 / W 1 ) P (W 1 )

 ( 3 / 4 )( 2 / 5 )  ( 2 / 4 )( 3 / 5 )
 P (W 2 )  3 / 5
2 January 2023 Lamessa .D 35
Conditional Probability Cont’d…..
Example-4:
Box A contains 100 bulbs of which 10% are defective.
Box B contains 200 bulbs of which 5% are defective. A
bulb is picked from a randomly selected box.
a. Find the probability that the bulb is defective
b. Assuming that the bulb is defective, find the
probability that it came from box A.

2 January 2023 Lamessa .D 36

Conditional Probability Cont’d…..

Solution:
First let us define the events as follows.
A : Box A is selected


P ( A)  P(B) 1/2
B : Box B is selected P (D / A)  1 /10
D : Bulb is defective P(D /B)  1 / 20

a. P(D )  P (D / A)P ( A)  P (D / B )P (B )
 (1 /10 )( 1 / 2 )  (1 / 20 )( 1 / 20 )
 P ( D )  3 / 40
P (D / A)P ( A) 1 / 20
b. P(A /D )    (1 / 20 )( 40 / 3)
P(D ) 3 / 40
 P ( A / D )  2 /3

2 January 2023 Lamessa .D 37

Conditional Probability Cont’d…..
Example-5:
One bag contains 4 white and 3 black balls and a second
bag contains 3 white and 5 black balls. One ball is drawn
from the first bag and placed in the second bag unseen
and then one ball is drawn from the second bag. What is
the probability that it is a black ball?
Solution:
First let us define the events as follows.
B 1 : black ball is drawn from the first bag
W 1 : white ball is drawn from the first bag
2 January 2023 Lamessa .D 38
Conditional Probability Cont’d…..
B 2 : black ball is drawn from the second bag
W 2 : white ball is drawn from the second bag

Then, we will have:

P ( B1 )  3 /7 P ( B 2 / B1 )  6 /9 P ( B 2 /W 1 )  5 /9

P (W 1 )  4 /7 P (W 2 / B 1 )  3 /9 P (W 2 / W 1 )  4 /9

P (B2)  P ( B 2  B1 )  P ( B 2 W 1 )

 P ( B 2 )  P ( B 2 / B 1 ) P ( B 1 )  P ( B 2 / W 1 ) P (W 1 )

 P ( B 2 )  ( 6 / 9 )( 3 / 7 )  ( 5 / 9 )( 4 / 7 )

 P ( B 2 )  28 / 63

2 January 2023 Lamessa .D 39

Conditional Probability Cont’d…..
Exercise:
1. For three events A, B and C, show that:
a. P [( A  B ) / C ]  P [ A /( B  C )] P ( B /C )

b. P(A  B  C )  P [ A /( B  C )] P ( B /C ) P (C )

2. Box A contains 3 white and 2 red balls while another box

B contains 2 red and 5 white balls. A ball drawn at random
from one of the boxes turns out to be red. What is the
probability that it came from box A?

2 January 2023 Lamessa .D 40

 Conditional Probability Cont’d…..
3. In a certain assembly plant, three machine A, B and C
make 30%, 45% and 25% of the products respectively. It is
known that 2%, 3% and 5% of the products made by each
machine, respectively, are defective. Suppose that a
finished product is randomly selected.
a. What is the probability that it is defective?
b. If the product is known to be defective, what is the
probability that it is made by machine A?

2 January 2023 Lamessa .D 41

Independence of Events
 Two events A and B are said to be statistically
independent if and only if
P(A B)  P ( A)P (B )

 Similarly, three events A, B and C are said to be

statistically independent if and only if
P(A  B C )  P ( A ) P ( B ) P (C )

 Generally, if A1, A2, …, An are a sequence of independent

events, then
 n  n
P   A i    P ( A i )
 i 1  i 1

2 January 2023 Lamessa .D 42

Independence of Events Cont’d……
 If A and B are independent, then we have
P(A B) P ( A)P (B )
i. P(A /B)    P ( A)
P(B) P(B)
 P ( A / B )  P ( A)
P(A B) P ( A)P (B )
ii. P (B / A)    P(B)
P ( A) P ( A)
 P(B / A)  P(B)

Example-1:

If A and B are independen t, then show that A and B are also

independen t.

2 January 2023 Lamessa .D 43

Independence of Events Cont’d……
Solution:
P ( A)  P(A B) P(A B)
 P ( A  B )  P ( A)  P ( A  B )  P ( A)  P ( A)P (B )
 P ( A  B )  P ( A )[ 1  P ( B )]  P ( A ) P ( B )
 By the definition of independen t events, A and B are independen t.

Example-2:
The probability that a husband and a wife will be alive 20
years from now are given by 0.8 and 0.9 respectively. Find
the probability that in 20 years
a . both will be alive c . at least one will be alive
b . neither wi ll be alive
2 January 2023 Lamessa .D 44
Independence of Events Cont’d……
Solution:
• First let us define the events as follows.

H : Husband will be alive

W : Wife will be alive

• Then we will have

P(H )  0 .8  P ( H )  1  P ( H )  1  0 .8  0 .2
P (W )  0 . 9  P (W )  1  P (W )  1  0 . 9  0 . 1
• The two events can be considered as independent.

2 January 2023 Lamessa .D 45

Independence of Events Cont’d……
Solution:
a . P ( both )  P ( H W )  P ( H ) P (W )

 P ( both )  P ( H  W )  ( 0 . 8 )( 0 . 9 )

 P ( both )  P ( H  W )  0 . 72

b . P ( neither )  P ( H W )  P ( H ) P (W )

 P ( neither )  P ( H  W )  ( 0 . 2 )( 0 . 1)

 P ( neither )  P ( H  B )  0 . 02

c . P ( at least one)  1  P ( neither )

 P ( at least one)  1  0 . 02

 P ( at leat one)  0.98

2 January 2023 Lamessa .D 46
Assignment-I

1. Show that the probability that exactly one of the events

or occurs is given by:
P ( A)  P (B )  2P ( A  B )
2. Box A contains 10 white and 3 black balls. Another box B
contains 3 white and 5 black balls. Two balls are drawn
at random from box A and placed in box B and then one
ball is taken at random from the latter. What is the
probability that it is a white ball?
3. A child mixes 10 good and three dead batteries. To find
the dead batteries, his father tests them one-by-one and
without replacement. If the first four batteries tested are
all good, what is the probability that the fifth one is dead?

2 January 2023 Lamessa .D 47

2 January 2023 Lamessa .D 48