Probability Distributions

Lecture - 4
Probability theory

• The field of “probability theory” is a branch of mathematics

concerned with describing the likelihood of different
outcomes from uncertain processes.

• Probability theory is the cornerstone of the field of Statistics,

which is concerned with assessing the uncertainty of
inferences drawn from random data samples.
Probability theory Example
• Suppose the probability of obtaining a number 4 on rolling a fair dice
needs
to be established.

• The number of favorable outcomes is 1.

• The possible outcomes of the dice are {1, 2, 3, 4, 5, 6}. This

implies that there are a total of 6 outcomes.

• Thus, the probability of obtaining 4 on a dice roll, using probability

theory, can be computed as 1 / 6 = 0.167.
Probability Theory Basics
Random Experiments

• Random experiments can be defined as experiments that can be performed

many times under the same conditions and their outcome cannot be predicted
with complete certainty.

• All the possible outcomes are known, however, its exact

outcome cannot be precisely predicted in advance.
There are certain terms associated with random experiments

1. Sample space
2. Event
3. Outcome
4. Trial
Probability Theory Basics
Random Experiments Example

• Suppose a coin is tossed. The two possible outcomes are getting a head or a tail.
The outcome of this experiment cannot be predicted before it has been performed.
Furthermore, it can be conducted many times under the same conditions. Thus,
tossing a coin is an example of a random experiment.

• Another random experiment example is that of rolling a dice. There can be 6

possible outcomes {1, 2, 3, 4, 5, 6}. However, none of the outcomes can be
exactly predicted.
1.
Probability Theory Basics
2.Events
• Events in probability can be defined as a set of outcomes of a random
experiment.

• The sample space indicates all possible outcomes of an experiment.

• Thus, events in probability can also be described as subsets of the sample

space.
Example
• Suppose a fair die is rolled.
• The total number of possible outcomes will form the sample space and are given by {1, 2, 3,
4, 5, 6}.
• Let an event, E, be defined as getting an even number on the die.
• Then E = {2, 4, 6}. Thus, it can be seen that E is a subset of the sample space and is an
outcome of
the rolling of a die.
Probability Theory Basics
Probability of an Event
Let P(A) denote the probability that event A occurs.
Any probability is a number between 0 and 1.
For any event A,
0 ≤ P(A) ≤ 1
It is trivial to see that
P(φ) = 0 and P(S) = 1

If the sample space S consists of a finite (or countable infinite) number of outcomes, assign a
probability to
each outcome
• The sum of all probabilities equals to 1.
• If there is k (finite) outcomes in the sample space S, all equally likely, then each individual
outcome
has probability 1/k

The Probability of Occurrence of an Event?

P(E) = Number of Favourable Outcomes/ Total Number of Outcomes

Probability Theory Basics
Types of Events in Probability
• Impossible and Sure Events

• Simple Event and Compound

Event

• Dependent and Independent

Events

• Mutually Exclusive Events

• Exhaustive Events

• Equally Likely Events

Probability Theory Basics
Different Probability
Formulas
a. Probability formula with addition rule:
Whenever an event is the union of two other events, say A and B, then

P(A 𝖴 B) = P(A) + P(B) - P(A∩B)

P(A or B) = P(A) + P(B) - P(A∩B)

b. Probability formula with the complementary rule:

Whenever an event is the complement of another event, specifically, if A is an event, then
P(not A) = 1 - P(A) or P(A') = 1 - P(A).
P(A) + P(A′) = 1.
c. Probability formula with the conditional rule:
When event A is already known to have occurred, the probability of event B is known as
conditional probability and is given by:
P(B∣A) = P(A∩B)/P(A)

d. Probability formula with multiplication rule:

Whenever an event is the intersection of two other events, that is, events A and B need
to occur simultaneously. Then

P(A ∩ B) = P(A)⋅P(B) (in case of independent events)

P(A∩B) = P(A)⋅P(B∣A) (in case of dependent events)
Probability Theory Basics
A. Random Variable

B. Discrete Probability
Distributions

C. Bernoulli Distribution

D. Binomial Distribution

E. Geometric Distribution

F. Poisson Distribution
Discrete Probability Distributions
A. Random Variable
• In probability, a random variable is a real-valued function whose domain is the sample space of
the random
experiment.

• It means that each outcome of a random experiment is associated with a single real number, and
the single real number may vary with the different outcomes of a random experiment.

• Hence, it is called a random variable and it is generally represented by the letter “X”.

For example, let us consider an experiment for tossing a coin two times.

Hence, the sample space for this experiment is S = {HH, HT, TH, TT}

If X is a random variable and it denotes the number of heads obtained, then the values are represented as follows:

X(HH) = 2, X(HT) = 1, X(TH) = 1, X(TT) = 0.

Similarly, we can define the number of tails obtained using another variable, say Y.
(i.e) Y(HH) = 0, Y(HT) = 1, Y(TH) = 1, Y(TT)= 2. Dr. Selva Kumar S
(SCOPE)
Discrete Probability Distributions
Random Variable
A random variable can be either discrete or
continuous
 A.Discrete random variables have a countable number of outcomes
Examples: Dead/alive, treatment/placebo, dice, counts,etc.

 B.Continuous random variables have an infinite continuum of possible values.

Examples: blood pressure, weight, the speed of a car, the real numbers from 1
to 6.
Discrete Probability Distributions
A.Discrete Random Variables
• The two types of discrete random variables most commonly used in machine
learning are
binary and categorical.
•Binary Random Variable: x in {0, 1}

•Categorical Random Variable: x in {1, 2, …, K}.

• A binary random variable is a discrete random variable where the finite set of
outcomes is in
{0, 1}.

• A categorical random variable is a discrete random variable where the finite set of
outcomes
is in {1, 2, …, K}, where K is the total number of unique outcomes.
Discrete Probability Distributions roll of a die

A.Discrete Random Variables

• Each outcome or event for a discrete random variable has a probability.

• The relationship between the events for a discrete random variable and their
probabilities is called the discrete probability distribution or probability mass
function (PMF).

• For outcomes that can be ordered, the probability of an event equal to or less than
a given
value is defined by the cumulative distribution function (CDF).

• The inverse of the CDF is called the percentage-point function and will give the
discrete
outcome that is less than or equal to a probability.
Discrete Probability Distributions
Probability Mass
Function

• PMF: Probability Mass Function, returns the probability of a given outcome.

• The returned probability lies in the range [0, 1] and the sum of all probabilities for every state
equals
one. • The relationship between the events for a
discrete random variable and their probabilities
is called the discrete probability distribution or
probability mass function (PMF).
Discrete Probability Distributions
Probability Mass Function(PMF)
• Let is dice is rolled then the probability of getting a number equal to 4 is an
example of probability mass function. The sample space for the given event is
S={1, 2, 3, 4, 5, 6} and X be the random variable for getting a 4.

• The probability mass function evaluated for X = 4 is 1/6. x p(x)

1 p(x=1)=1/6
2 p(x=2)=1/6

3 p(x=3)=1/6
4 p(x=4)=1/6
5 p(x=5)=1/6
6 p(x=6)=1/6
1.0
Discrete Probability Distributions
There are many common discrete probability distributions

1. Binary Random Variable: Bernoulli Distribution

2. Sequence of a Binary Random Variable: Binomial

Distribution

3. Geometric Distribution

4. Poisson Distribution
Discrete Probability Distributions
1.Bernoulli Distribution
• The Bernoulli distribution is a discrete probability distribution of a single binary
random
variable, which either takes the value 1 or 0.

• The Bernoulli distribution as a model giving the set of possible outcomes for a single
experiment, that can be answered with a simple yes-no question.

• Suppose, we toss a fair coin once.

• The probability of obtaining heads is P(Heads) = 0.5

The Bernoulli Distribution models only a single

trial.
Discrete Probability Distributions
2.Binomial Distribution
• The binomial distribution describes the discrete probability distribution of the
number of successes in a sequence of n independent trials, each with a binary
outcome.

• The success or failure is given by the probability p or (1-p) respectively.

• The success of k is given by the probability p to the power of k, whereas the

probability of failure is defined by (1-p) to the power of n minus k, which is
basically the number of trials minus the one trial where we get k.
Discrete Probability Distributions
2.Binomial Distribution
• Example:

we are going to flip the fair coin three times, while

being interested in the random variable describing the number
of heads obtained

• If we want to compute the probability of the coin coming

up as heads two times.
Discrete Probability Distributions
3.Geometric Distribution
• Suppose, we are interested in the number of times we have to flip a coin until it
comes up
heads for the first time.

• The geometric distribution gives the probability of the first success occurrence,
requiring n independent trials, with a success probability of p.

The following assumptions need to be true, to calculate the

geometric
distribution:

• Independence
• For each trial, there are only two possible outcomes
• The probability of success is the same for every trial
Discrete Probability Distributions
4.Poisson Distribution
• It is usually used in scenarios where we are counting the occurrences of certain
events in an interval of time or space. In practice, it is often an approximation of a
real-life random variable.

Example:
• The number of emails that I get in a weekday can be modeled by a Poisson distribution with an
average
of 0.2 emails per minute.

• What is the probability that I get no emails in an interval of length 5 minutes?

Let X be the number of emails that I get in the 5 -minute interval. Then, by the
assumption X is
a Poisson random variable with parameter λ=5(0.2)=1 ,
Continuous Random Variable
Continuous Random Variable
• A continuous random variable can be defined as a random variable
that can take on an infinite number of possible values.
• The probability that a continuous random variable will take on an
exact value
is 0.
• The cumulative distribution function (CDF) and the probability density
function (PDF) are used to describe the characteristics of a continuous
random variable.
Cumulative Distribution Function
(CDF)
• The cumulative distribution function of a continuous
random variable can be determined by integrating the
probability density function.

• It can be defined as the probability that the random variable, X, will take on a value
that is
lesser than or equal to a particular value, x.

• The formula for the cdf of a continuous random variable, evaluated between two points
a and b,
is given below:

CDF Properties
• Non-decreasing

• FX (x) tends to 1, as x → ∞
Probability Density
Function(PDF)
• The probability density function of a continuous random variable can be defined as a
function that gives the probability that the value of the random variable will fall between a
range of values.

• Let X be the continuous random variable, then the formula for the pdf, f(x), is given as follows:

For the pdf of a continuous random variable to be valid, it must satisfy the following conditions.

This means that the total area under the graph of the pdf must be equal to 1.

• f(x) > 0. This implies that the probability density function of a continuous random variable
cannot be negative.
Mean and Variance of Continuous Random Variable
Mean of Continuous Random Variable
• The mean of a continuous random variable can be defined as the weighted average value of
the random variable, X.

• It is also known as the expectation of the continuous random variable.

Variance of Continuous Random Variable

• The variance of a continuous random variable can be defined
as the expectation of the squared differences from the mean.
Continuous Random Variable Types

• Uniform Random Variable

• Normal Random Variable

• Exponential Random
Variable
Continuous Random Variable Types
Uniform Random Variable
• A continuous random variable that is used to describe a uniform distribution is known as a
uniform random variable.

• Such a distribution describes events that are equally likely to occur.

Continuous Random Variable Types
Normal Random Variable
• A continuous random variable that is used to model a normal distribution is known as a
normal random
variable.

• If the parameters of a normal distribution are given as X ∼ N (μ,σ2) then the formula for the
pdf.

• A normal distribution where μ = 0 and σ2 = 1 is known as a

standard
normal distribution.

• Thus, a standard normal random variable is a continuous

random
variable that is used to model a standard normal distribution.
Continuous Random Variable Types
Exponential Random Variable
• Exponential distributions are continuous probability distributions
that model processes where a
certain number of events occur continuously at a constant average rate, λ ≥ 0 .

• A continuous random variable used to describe such a distribution is called an exponential

random variable.
Continuous Random Variable Types
Exponential Random Variable
• Exponential distributions are continuous probability distributions
that model processes where a
certain number of events occur continuously at a constant average rate, λ ≥ 0 .

• A continuous random variable used to describe such a distribution is called an exponential

random variable.
The Gaussian distribution, also known as the normal distribution, plays a
fundamental role in machine learning. It is a key concept used to model the
distribution of real-valued random variables and is essential for understanding
various statistical methods and algorithms.
The Gaussian distribution, also known as the normal distribution, is a continuous
probability distribution that is widely used in statistical modeling and
Machine Learning. It is a bell-shaped curve that is symmetrical around its mean
and is characterized by its mean and standard deviation.
•The Gaussian distribution is important because it is a common and
useful model for data, and because it allows for efficient
calculations in statistical analysis.