Theorem of Total Probability

• Let B1, B2, …, Bn be a pairwise mutually exclusive (disjoint) events

(i.e., Bk ∩ Bj = φ, where k ≠ j ; k,j = 1, 2,…,n) and they form a
partition of sample space S such that B1 ∪ B2 ∪ . . . . . ∪ Bn = S.
•Then probability of any other event A from this sample space can
Pbe
( A) given
P( A | Bin terms of probabilities of these disjoint events as:
1 ) P ( B1 )  P ( A | B2 ) P ( B2 )  .....  P ( A | Bn ) P ( Bn )

(
P( A) = ∑ i =1 P A Bi P (Bi )
n
)
• This is called divide and conquer approach to find the probability
of complex event A by identifying all different causes Bi for A.

03/01/2024 Dr. Jigar Shah (Asst. Prof., ICT, PDEU, 1

Gandhinagar)
 Bayes’ Theorem
Let B1, B2, …, Bn be a pairwise mutually exclusive (disjoint) events and form a
partition of sample space S as defined earlier. Let A be an event with P(A)>0.
Then for i=1,2,….,n,

(
P A Bi P (Bi ) )
P (Bi A) =
∑
n

i =1
P A Bi P ( (
Bi
) )
likelihood  prior
posterior 
evidence
Bayes’ theorem is an important tool for determining the
relative likelihood of a particular cause to an event.
03/01/2024 Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh 2
inagar)
 Example
• A communication system encounters one of three possible
interference waveforms: F1, F2 or F3. The probability of each
interference is 0.8,0.16 and 0.04 respectively. The communication
system fails with probabilities 0.01,0.1 and 0.4 when it encounters F1,
F2 and F3 respectively. Given that the system had failed, find the
probability that the failure is a result of F1, F2 or F3 respectively.
P(A | F1 ) P ( F1 )
• P(F1)=0.8, P(F2)=0.16, P(F1 | A)  3
 0 .2
 P(A | F ) P(F )
i i

P(F3)=0.04 i 1

P(A | F2 ) P ( F2 )
P(F2 | A)  3
 0 .4
• A=event of system failure.  P(A | F ) P(F )
i 1
i i

P(A | F3 ) P ( F3 )
• P(A|F1)=0.01, P(A|F2)=0.1, P(A| P(F3 | A)  3
 0 .4
 P(A | F ) P(F )
i i

F3)=0.4 i 1

• P(F1|A), P(F2|A), P(F3|A)=?

03/01/2024 Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh 3
inagar)
 Random Variable (R.V.):
• Characterizes the random experiment in terms of real
numbers by a rule which assigns a numerical value to
each possible outcomes.
• Example: If the experiment is of tossing a coin. Then a
random variable X can be defined as :
x1 Head X=1

x2 Tail X = -1

• Types of Random Variables:

– Discrete Random Variables
• The random variable can only take a finite number of values in any finite observation interval
• E.g. Outcome of rolling dice, tossing coins, drawing cards from deck etc.
– Continuous Random Variables
• The random variable can take any value in a finite observation interval; infinite number of possible
values
03/01/2024 Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh 4
• E.g. Temperature at some location, Noise signal amplitude at particular instant of time
inagar)
Discrete Random Variable (D.R.V.): Notation
S
S1 S Sn S

x1 x2 xn R

-3 -2 -1 0 1 2 3
• An upper-case letter will represent the name of the
random variable, usually X.
• Its lower-case counterpart, xi,(i=1,2,..,n) will represent
the value of the random variable for outcome i.
• The probability that the random variable X will equal
xi is: P(X = xi) or PX (xi)
• Let X = number of heads in 10 flips of coin
• P(X = 5) = PX (5)Dr.inagar)
03/01/2024 = Shah
Jigar probability of
(Asst. Prof., ICT, PDEU, 5 heads in 10 flips
Gandh 5
 Discrete Random Variable (D.R.V.): Example

• Two dice are thrown. The sum of the points

appearing on the two dice is an R.V. X. Find the
values taken by X, and corresponding
probabilities.
• Number of possible outcomes=6*6=36
• R.V. X can take values: 2,3,…,12
X =xi 2 3 4 5 6 7 8 9 10 11 12
PX(xi) 1/36 2/36 3/36 4/36 5/36 6/36 5/36 4/36 3/36 2/36 1/36

= 1/36 1/18 1/12 1/9 5/36 1/6 5/36 1/9 1/12 1/18 1/36

P
i
X ( xi )  1

03/01/2024 Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh 6

inagar)
• Probability Mass Function (PMF) for D.R.V.
– P(X = xi) or PX (xi) is also called PMF for D.R.V.
P
i
X ( xi )  1

• Cumulative Distribution Function (CDF)

= P( X  x ) i

• Properties of CDF ：
1. 0  FX ( xi )  1
2. FX ()  1, FX ()  0
3. FX is a nondecreasing function
i.e. FX ( xi )  FX ( x j ) for xi  x j
• For D.R.V. the CDF is always staircase type function and PMF is
always impulse train
03/01/2024
as cleared in next example.
Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh 7
inagar)
 Example: CDF of Discrete R.V.
• Consider an experiment of rolling of two dice.
Assign the random variable X to the sum of
numbers on two dice. Find PMF and CDF.
• X = sum of two numbers after rolling two dice.
Following table specifies all possible outcomes
along with their probabilities.
X =xi 2 3 4 5 6 7 8 9 10 11 12
PX(xi) 1/36 2/36 3/36 4/36 5/36 6/36 5/36 4/36 3/36 2/36 1/36

FX(xi) 1/36 3/36 6/36 10/36 15/36 21/36 26/36 30/36 33/36 35/36 36/36

03/01/2024 Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh 8

inagar)
• PMF and CDF plots are shown in figure.
• In general for any D.R.V. the CDF is staircase (sequence of steps)
and PMF is train of impulses specifying probability for different
values of D.R.V.
• If a D.R.V. X takes values x1,x2,…..,xn with probabilities a1,a2,
…,an respectively, then
• 03/01/2024 and
Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh
inagar)
9
Discrete Random Variables: Joint PMF
The concept of PMF can be extended to two
variables.
• PX,Y (xi, yj) = P(X =xi, Y= yj): Joint PMF for two
D.R.V. X and Y
 P
i j
X ,Y ( xi , y j )  1

• It can be extended to multiple D.R.V.

03/01/2024 Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh 10

inagar)
 Discrete Random Variables: Independent D.R.V.

For independent outcomes, R.V. are also independent

and PX,Y (xi,yj)= PX (xi)PY (yj)
• Example: An experiment of tossing a coin twice:
• X= R.V. for tossing a coin first time; x1=1, x2=-1
• Y= R.V. for tossing a coin second time ; y1=1, y2=-1
• Outcomes of two tosses are independent, So, X and
Y are independent R.V.
• PX,Y (1,1)= PX (1)PY (1) = ½*½ = 1/4.
• Similary, P X,Y (1,-1)=
PX,Y (-1, 1)=PX,Y (-1, -1)= 1/4
03/01/2024 Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh
inagar)
11
 Discrete Random Variables: Conditional and
Marginal (total) PMF
• Conditional PMF and Bayes’ Rule
– PX|Y (xi|yj) = P(X = xi ; given Y = yj) = PX,Y (xi,yj)/PY (yj)
– PY|X (yj| xi)= P(Y = yj ; given X = xi) = PX,Y (xi,yj)/PX(xi)
– PX,Y (xi,yj)= PX|Y (xi|yj) PY (yj)=PY|X (yj| xi) PX(xi)
– PX|Y (xi|yj) =PY|X (yj| xi) PX(xi)/PY (yj)

∑ 𝑃 𝑋∨𝑌 (𝑥𝑖∨𝑦 𝑗)=1𝑎𝑛𝑑 ∑ 𝑃𝑌∨ 𝑋(𝑦 𝑗∨𝑥𝑖)=1

• Also sum of probability over sample space is 1; So we can have
𝑖 𝑗

𝑃𝑌 ( 𝑦 𝑗)=∑ 𝑃𝑌∨𝑋 (𝑦 𝑗∨𝑥𝑖)𝑃 𝑋 (𝑥𝑖)

• From this equation and Bayes’ rule we can write:

𝑖
03/01/2024 Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh 12
• This is called marginal (total) PMF.
inagar)
Continuous Random Variables (C.R.V.) and Distribution Functions:

• For C.R.V. meaningful quantity is not the

probability at any particular value, X=xi, but the
probability within small interval x<X≤x+∆x.
• The (CDF) defined for discrete R.V. is also valid
for continuous R.V.
P( X  x )
• ≜
• All properties of CDF of D.R.V. are also valid for
C.R.V.

03/01/2024 Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh 13

inagar)
• Probability Density Function (PDF):

• ≜ F ( x)   f (u )du ; u is dummy variable

dFX ( x) x
i.e. X X
dx 

• Properties of PDF ：
1. PDF is always non-negative f X ( x )  0
2. Total area under PDF curve is unity

 f ( x)dx  F ()  F ()  1

X X X

3. Relationship between probability and PDF:

Probability of X in (x1,x2]=Area of PDF curve
between x=x1, x2 :
x2
P( x1  X  x2 )  FX ( x2 )  FX ( x1 )   f X ( x)dx
x1
03/01/2024 Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh 14
inagar)
Nature of Distribution Functions for C.R.V.:
• Both CDF and PDF are continuous function for C.R.V.

03/01/2024 Dr. Jigar Shah (Asst. Prof., ICT, PDEU, Gandh 15

inagar)