Probability

Classification of Processes
• Based on the nature of the processes, they can be classified as:
 Deterministic
• No uncertainty about the time dependence of the process at any
instant of time
• Conservation laws 2m
m
 Stochastic/Random
• Processes depend upon number of unknown factors, and hence,not
completely predictable
• Most real life processes – throwing of dice, detection of an
encoded signal at receiver, weather forecasting,etc.

So how to analyze random processes??

— The concept ofProbability
Why Probability/Randomness?

Approximations

Imperfections
Disturbances

• No mathematical model is perfect

• Dynamic systems are driven by inputs as well as disturbances

• Sensors do not provide perfect data

Random Processes

• Cannot be completely mathematically described

• Any real life experiment – a random process

• Outcome(s) of experiment – random variable(s)

• Measuring temperature – random process

• Temperature reading from sensor(s) – random variable(s)

• Thus, random processes can be viewed as data (random

variables) generating phenomena
7
Randomness

• In many situations one will know what are all the possible outcomes,
but one cannot be certain about the exact outcome
• e.g. You Buy a Lottery ticket
• You may win, or may not win – Possibilities are fixed, known
• Will you win, or not win – Exact outcome is uncertain
• e.g. You roll a dice
• You may get 1,2,3,4,5, or 6 - Possibilities are fixed, known
• Will you get 1,2,3,4,5, or 6 - Exact outcome is uncertain
• Both the examples above has an element of randomness that governs
the outcome at any instance
 Quantifying Randomness
• Lets take the example of rolling the dice again
• If two people experiment (with 10 trials) to understand what is the proportion
of the outcome being “6”, we may get something like this..
Roll Your Cumulative Cumulative
(Trial) Roll Proportion of 6's My roll Proportion of 6's
1 6 1/1 1 0/1
2 5 1/2 2 0/2
3 1 1/3 1 0/3
4 6 2/4 1 0/4
5 6 3/5 3 0/5
6 2 3/6 2 0/6
7 3 3/7 4 0/7
8 1 3/8 5 0/8
9 4 3/9 3 0/9
10 4 3/10 6 1/10

You may have vastly different predictions with small number of trials – Expected of a random phenomenon

At the end of 250 trials, proportion of 6’s – 50/250, and at the end of 1000 trials – 170/1000
Quantifying Randomness - Probability
• If the number of trials are large (or as they say “In the Long
Run”),the proportion of 6’s will be closer to 1/6.

• We are able to put a number to the outcome.

• Probability – The probability of a particular outcome is the

proportion of times that particular outcome would occur in a
long run of observations

• Experiment, trial, outcome, sample space, favourable

outcome, probability
Probability Distribution Functions – By Example
Throwing two dice together
Each die can turn up with any one of 6 values {1,2,3,4,5,6}. For two
dice, there are a total of 6x6 = 36 combinations {Sample Space}

The sum of the two die values cannot be 0, or 1, and also cannot be
greater than 12

We can get a sum of 2, through only one combination: {1,1}. Hence probability of sum = 2, is 1/36
We can get a sum of 6, through the following combinations: {1,5}, {5,1}, {2,4}, {4,2}, {3,3}.Hence probability of sum = 6, is 5/36

Sum 2 3 4 5 6 7 8 9 10 11 12 Total
Probability 1/36 2/36 3/36 4/36 5/36 6/36 5/36 4/36 3/36 2/36 1/36 1

A plot of the probabilities, is called the probability distribution function

For discrete random variables such plot , is called the probability mass
function (pmf)
For continuous random variables such plot , is called the probability
density function (pdf)

You can describe such functions as appropriate mathematical equations

Commonly encountered distribution functions
1  1 ( x   )2 
f ( x)  exp  
Normal Distribution  2  2  2


Binomial
1.20E-01
1.00E-01

Binomial Distribution 8.00E-02

6.00E-02
4.00E-02
2.00E-02
0.00E+00
0 20 40 60
-2.00E-02

1
Uniform Distribution f ( x)  ,a  x  b
ba
Sampling Distribution - Proportions

• Toss of a coin: Pr(H) = p = 0.5 : Pr(T) = q = 0.5

• Let’s say we toss the coin 5 times (n=5), Can we estimate the number of times heads will
turn up?
The probabilities for a given number of
Number of Heads Freq Prob heads(x) turning up can be computed by
0 1 0.03125 the binomial distribution function formula
1 5 0.15625
2 10 0.3125
3 10 0.3125
4 5 0.15625
5 1 0.03125 The binomial distribution function can also be
Total 32
well approximated by a normal curve, with a
mean of µ = np, and a variance of σ2 =npq
Why do we talk about coin tosses ,cards and dice in
Probability
Passes a quality test/Fails a quality test

A random Dichotomous qualitative variable Gender: Man/Women

Customer: Buys/Doesn’t buy

Education Level of an employee (many levels)

A random Polytomous qualitative variable
Colour of mobile phones (Many colours)

Brand of car (Many options)

Number of durables owned

A random discreet quantitative variable
Number of children at home

Number of promotions a employee is given

 Sample Space and Events
• Sample Space is the set of all possible outcomes in a random phenomenon
• e.g. Let us say, we toss a coin three times:
• The possible outcomes are: {HHH, HHT, HTT, HTH, THH, THT, TTH, TTT}
• The sample space is: {HHH, HHT,HTT,HTH,THH,THT, TTH,TTT}

• An Event is a subset of the sample space and some examples are:

• Event A: The first toss is Heads: {HHH, HHT,HTT,HTH}

• Event B: There at least two heads in three tosses: {HHH, HHT, HTH,THH}
• Event C: The first toss is tails: {THH, THT, TTH, TTT}
Types of Events: Exclusive and Independent Events

Mutually exclusive events Independent events

• Two events are mutually exclusive • Two events are independent if

if occurrence of one prevents the occurrence of one has no influence
occurrence of the other on occurrence of other
• For example: In a three coin toss • For example: In a three coin toss
experiment, events {The first toss experiment, the occurrence of
is Heads} and {The first toss is head in second and third tosses is
tails} are mutually exclusive independent of occurrence of
• If there are no common outcomes head or tail in first or second
between two event sets, then they toss
are mutually exclusive events
Probabilities for an Event

• The Probability of each individual outcome is between 0 and 1

• 0 ≤ P(HHH) ≤ 1, and so on
• The total of all individual probabilities equals 1.
• The Probability of event A, is
•
• If, Event A: The first toss is Heads: {HHH, HHT,HTT,HTH}
• Sample space is: {HHH,HHT,HTT,HTH,THH,THT, TTH,TTT}
• P(First toss is Heads) = 4/8 = 0.50
Mutually Exclusive Events – Addition Rule

• Mutually Exclusive events are events which do not have any common
outcomes:
• Say, in a three coin toss,
• Event A = {First toss is heads}
• Event C = {First Toss is Tails}
• Event A = {HHH, HHT, HTH, HTT}, P(A) = 4/8
• Event C = {THH,THT, TTH, TTT}, P(C) = 4/8
• If we define a new event D, as either Event A or Event C,
• P(D) = P(A or C) = P(A)+P(C) = 4/8+4/8 = 1
• i.e. – is the sum rule for mutually exclusive events
 Mutually non-exclusive events – Addition Rule
• Mutually Non-Exclusive events are events that do have some common
outcomes:
• Say, in a three coin toss,
• Event A = {First toss is heads, and there are two heads in total} {HHH, HHT, HTH}, P(A) = 3/8
• Event B = {First Toss is Heads, and the last is not tails} {HHH, HTH}, P(B) = 2/8
• If we define a new event C, as either Event A or Event B
• P(C) = {P(A or B) = P(A)+P(B) – Outcomes that are double counted}
= 3/8+2/8 -2/8 = 3/8
• i.e.
• – is the sum rule for mutually non-exclusive events
Independent Events – Product Rule
• Two events are independent, if occurrence of one event does not
influence the occurrence of the other event
• e.g. Suppose we want to know the probability of event {HHT}, in a three coin toss..
• First Event = P(Heads in the first toss) = ½
• Second Event = P(Heads in the second toss) = ½
• Third Event = P ( Tails in the third toss) = ½
• The outcome of the first event does not affect the outcome of the second event,
and so forth.
• Therefore, P(HHT) = ½ * ½ * ½ = 1/8
• i.e. x P(C) – Is the Product rule for independent
events
Joint, Marginal, and Conditional Probabilities
Quality Check Plant A Plant B Plant C Total
Pass 9000 4000 4500 17500
Fail 1000 1000 500 2500
Total 10000 5000 5000 20000

Converting the numbers to probabilities

Quality Check Plant A Plant B Plant C Total

Pass 0.45 0.20 0.225 0.875
Fail 0.05 0.05 0.025 0.125
Total 0.5 0.25 0.25 1.0

Quality Check Plant A Plant B Plant C Total

Pass P(Pass∩Plant A) P(Pass∩Plant B) P(Pass∩Plant C) P(Pass)
Fail P(Fail ∩Plant A) P(Fail ∩Plant B) P(Fail ∩Plant C) P(Fail)
Total P(S: Plant A) P(S: Plant B) P(S: Plant C) 1.0
Joint, Marginal, and Conditional Probabilities
Quality Check Plant A Plant B Plant C Total
The highlighted numbers
Pass 0.45 0.20 0.225 0.875 are Joint Probabilities
Fail 0.05 0.05 0.025 0.125 Joint Probability =
Probability of two events
Total 0.5 0.25 0.25 1.0 occurring together

Quality Check Plant A Plant B Plant C Total The highlighted numbers

Pass 0.45 0.20 0.225 0.875 are Marginal Probabilities
Fail 0.05 0.05 0.025 0.125 Marginal Probability =
Probability of an event
Total 0.5 0.25 0.25 1.0 (unconditionally)

Quality Check Plant A Plant B Plant C Total

Pass P(Pass∩Plant A) P(Pass∩Plant B) P(Pass∩Plant C) P(Pass)
Fail P(Fail ∩Plant A) P(Fail ∩Plant B) P(Fail ∩Plant C) P(Fail)
Total P(Plant A) P(Plant B) P(Plant C) 1.0
Conditional Probability
Supposing you want to determine, what is the probability of a unit failing the test, if you know for
sure that the unit is sampled from Plant A

- Conditional probability defined thus:

A AC
F∩ A) F F
Joint, Marginal, and Conditional Probabilities
Quality Check Plant A Plant B Plant C Total
Pass 9000 4000 4500 17500 P(Test = Fail | Plant = A)
Fail 1000 1000 500 2500
Total 10000 5000 5000 20000 = P(Test =Fail ∩ Plant = A)/P(Plant =A)

From the contingency table, we

Quality Check Plant A Plant B Plant C Total can quickly compute this to be
Pass 0.45 0.20 0.225 0.875 = 0.05/0.5 =0.1

Fail 0.05 0.05 0.025 0.125

Total 0.5 0.25 0.25 1.0 We can also quickly compute this
from the original data as
= 1000/10000 =0.1
In your data table, if you take column percentages (Instead of % of grand total), the cells
would represent conditional probabilities conditional on the column variable value, and
vice-versa with the row percentages

Just like we computed P(Test = Fail | Plant = A), we can also compute P(Plant = A | Test = Fail), from the
contingency table as = P(Pant =A ∩ Test = Fail)/P(Test = Fail) = 0.05/0.125 = 0.4 (or as 1000/2500)
Joint, Marginal, and Conditional Probabilities
Quality Check Plant A Plant B Plant C Total
Pass 0.45 0.20 0.225 0.875
Fail 0.05 0.05 0.025 0.125
Total 0.5 0.25 0.25 1.0

We have two equations, for two different conditional probabilities as below:

P(Test = Fail | Plant = A) = P(Test =Fail ∩ Plant = A)/P(Plant =A) 1

P(Plant = A | Test = Fail) = P(Plant = A∩ Test = Fail)/P(Test = Fail) 2

Rearranging the two equations we get:

3
P(Fail | A) *P(A) = P(A | Fail)*P(Fail)

The above formula relates two conditional probabilities, and two marginal probabilities, and is the
Bayes Theorem
Bayes Theorem

Bayes theorem is formulated as

It is interpreted commonly as

is referred to as prior probability

𝑷 𝑬𝒗𝒆𝒏𝒕 𝑬𝒗𝒊𝒅𝒆𝒏𝒄𝒆 𝒊𝒔 𝒓𝒆𝒇𝒆𝒓𝒓𝒆𝒅 𝒕𝒐 𝒂𝒔 𝑷𝒐𝒔𝒕𝒆𝒓𝒊𝒐𝒓 𝒑𝒓𝒐𝒃𝒂𝒃𝒊𝒍𝒊𝒕𝒚

Bayes Theorem Example
A patient is diagnosed with a rare disease. The doctor informs that the test is very accurate.
90% of those who have the disease are correctly tested positive, and 95% of those who don’t
have the disease are tested negative. He also says that the disease is normally found in only 0.1
% of the population. Should the patient be worried?
Out of every thousand people tested only one has the disease (0.1% of population)
90% of the patients who have the disease test positive, and 10% of those who have the
disease test negative.
95% of those who don’t have the disease test negative, and 5% of those who don’t have the
disease test positive
The only worry for the patient is whether he really has the disease, given that the test has
returned positive

What he should be computing is

Bayes Theorem – An Example

The above are a set of marginal and conditional probabilities, which we compute by
constructing the contingency table

Test +ve Test -ve Total

Has Disease
No Disease
Total

Test +ve Test -ve Total

Has Disease P(Disease ∩ Positive) P(Disease ∩ Negative) P(Disease)
No Disease P(No Disease ∩ Positive) P(No Disease ∩ Negative) P(No Disease)
Total P(Positive) P(Negative) Total

P ( A  B )
P ( A B )  ( ∩ )
P (B ) P(Positive | Disease) =
( )
Bayes Theorem – An example
A patient is diagnosed with a rare disease. The doctor informs that the test is very accurate.
90% of those who have the disease are correctly tested positive, and 95% of those who
don’t have the disease are tested negative. He also says that the disease is normally found
in only 0.1 % of the population. Should the patient be worried?

Let us say, 100000 people were tested in total – 100 had the disease in total (0.1%)

90% of those who had the disease tested positive, and 10% tested negative

95% of those who did not have the disease tested negative, and 5% tested positive
Test +ve Test -ve Total = 90/100 = 0.9
Has Disease 90 10 100
No Disease 4995 94905 99,900
Total 5085 94915 1,00,000

.
Using the Bayes Equation we compute
.
Reference
• A first course in probability by Sheldon M. Ross
• Introduction to probability and statistics for engineers and scientists
by Sheldon M. Ross