Lec 1: Introduction to Probability

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering,

National Institute of Technology Andhra Pradesh,
Tadepalligudem, Andhra Pradesh
India

Jan 10, 2025

Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

References

A. Papoulis and S. Pillai, Probability, Random Variables and

Stochastic Processes, Fourth Edition, McGraw-Hill.

Peyton Z. Peebles, Probability, Random Variables, and Random

Signal Principles, Fourth Edition, McGraw-Hill.

Dimitri P. Bertsekas and John N. Tsitsiklis., Introduction to

Probability , Athena Scientific.

Geoffrey Grimmett and David Stirzaker, Probability and Random

Processes, Fourth Edition, Oxford University Press.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Outline of the Lecture

Introduction

Basic Set Theory

Cardinality and Countability

Probability Space

Conditional Probability

Bayes’ Rule

Independence

Combined Experiments

Bernouli Trials
V. Prakash Singh, PhD
Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Outline

1 Introduction

2 Mathematical Preliminaries

3 Axiomatic theory of Probability

4 Conditional Probability

5 Independence

6 Combined Experiments

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Introduction

Probability theory is the study of random phenomenon.

It provides a mathematical framework for quantifying uncertainty and

making predictions about events that might happen in the future.

Probability theory has its roots in gambling and games of chance but
has grown into a fundamental aspect of modern mathematics and
science.

Jacob Bernoulli and Pierre-Simon Laplace have made significant

contributions to the probability theory in the 17th and 18th centuries.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Classical definition of Probability

The classical definition of probability, also known as the ”Laplace

definition” after the mathematician Pierre-Simon Laplace, is one of
the earliest formal definitions of probability. This definition is based
on the assumption that all outcomes of a random experiment are
equally likely.

The probability P(A) of an event A is determined a priori without

actual experimentation: It is given by the ratio
NA
P(A) =
N
where N is the number of possible outcomes and NA is the number of
outcomes that are favorable to the event A.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Relative frequency definition of Probability

The relative frequency definition of probability, also known as the

frequentist interpretation, is based on the idea of conducting repeated
trials of an experiment and observing how often a particular event
occurs.

The probability P(A) of an event A is determined a priori without

actual experimentation. It is given by the ratio
nA
P(A) = lim
n→∞ n
where nA is the number of occurrences of the event A and n is the
number of trials..

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Classical definition of Probability

The classical definition of probability can be questioned on many

grounds and it is applicable only to a limited class of problems.

The classical definition assumes that all outcomes are equally likely,
which is not always the case in real-world scenarios. For example:
biased coins or loaded die may not give correct probabilities using
classical definition.

For experiments with an uncountable sample space, this definition is

inadequate. We illustrate the resulting difficulties with an example
known as the Bertrand paradox in the next few slides.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Bertrand Paradox

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Bertrand Paradox

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Bertrand Paradox

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Bertrand Paradox

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Outline

1 Introduction

2 Mathematical Preliminaries

3 Axiomatic theory of Probability

4 Conditional Probability

5 Independence

6 Combined Experiments

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Basic Set Theory

Set: A set is a collection of objects. The objects are called the
elements of the set.

A set is denoted by a capital letter while the elements are denoted by

small letters.

If an object a is an element of set A we write a ∈ A. If a is not an

element of A, we write a ̸∈ A.

A set can be specified as a list of all elements enclosed in a curly

bracket.
A = {1, 2, 3, 4, 5}
A set can also be specified in terms of the properties that every
element of the set satisfy.
A = {x|P(x)} = {x|x ≤ 5, x ∈ N }
V. Prakash Singh, PhD
Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Basic Set Theory

Subset: A set B is said to be a subset of another set A if every

element of B is also an element of A. This is denoted as B ⊆ A.

B is called a proper subset of A if B is a subset of A and there is at

least one element in A, which does not belong to B. This is denoted
as B ⊂ A.

Two sets A and B are said to be equal (A = B) if

B⊆A and B ⊆ A

It means every element of A is an element of B and vice versa.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Basic Set Theory

Universal Set: An all encompassing set of objects under discussion in

a given situation is called the universal set. It is denoted by S or U.

All sets including U are subsets of the universal set.

Null Set: A set which has no element is called a null set. It is denoted
by Φ or {}.

Null set is a subset of every set including itself.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Set Operations

Complement: The context for the complement of a set is provided by

the universal set. The complement of a set, denoted as, Ac is defined
as
Ac ≜ {x | x ̸∈ A, x ∈ U}
Union: The union of two sets A and B is the set of all elements that
belong to A or B or both and is denoted as A ∩ B.

A ∪ B ≜ {x | x ∈ A or x ∈ B}

Intersection: The intersection of two sets A and B is the set of all

elements that belong to both A and B and is denoted as A ∩ B.

A ∩ B ≜ {x | x ∈ A and x ∈ B}

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Set Operations
We can generalize the union and intersections to infinitely many sets
∞
[
Ai = A1 ∪ A2 ∪ · · · = {x | x ∈ Ai for some i}
i=1
\∞
Ai = A1 ∩ A2 ∩ · · · = {x | x ∈ Ai for all i}
i=1

Two sets A and B are called mutually exclusive or disjoint if their

intersection is empty.
A∩B ≜Φ
Partition of a Set: A partition of a set S is a collection of mutually
exclusive subsets Ai of S such that their union is set S.

A1 ∪ A2 ∪ · · · ∪ An = S and Ai ∪ Aj = Φ for all i ̸= j

V. Prakash Singh, PhD
Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Set Operations

Union and Intersection operations follow commutative law.

A∪B = B ∪A
A∩B = B ∩A

Union and Intersection operations follow associative law.

(A ∪ B) ∪ C = A ∪ (B ∪ C ) = A ∪ B ∪ C
(A ∩ B) ∩ C = A ∩ (B ∩ C ) = A ∩ B ∩ C

Union and Intersection operations follow distributive law.

(A ∪ B) ∩ C = (A ∩ C ) ∪ (B ∩ C )
(A ∩ B) ∪ C = (A ∪ C ) ∩ (B ∪ C )

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Set Operations
The distributive law for union and intersection operations can be
generalized to infinitely many sets as
[ ∞  ∞ 
[ 
Ai ∩ B = Ai ∩ B
i=1 i=1
∞
\  \∞  
Ai ∪B = Ai ∪ B
i=1 i=1

De Morgan Law : Complement of the union of two sets A and B is

equal to the intersection of the complements of the sets.
(A ∪ B)c = Ac ∩ B c
De Morgan Law : Complement of the intersection of two sets A and
B is equal to the union of the complements of the sets.
(A ∩ B)c = Ac ∪ B c
V. Prakash Singh, PhD
Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Venn Diagrams

Sets and the set operations can be easily visualized using Venn
Diagrams

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Ordered Pairs and Product Set

An ordered pair is a pair of elements written in a specific order,

typically as (a, b), where a is the first element and b is the second
element. The order of the elements is crucial in distinguishing one
ordered pair from another. For example, (a, b) is different from (b, a).

Consider two arbitrary sets A and B. The set of all ordered pairs (a, b)
where a ∈ A and b ∈ B is called the product, or Cartesian product, of
A and B. This product is denoted as A × B and read as A cross B.

A × B = {(a, b)|a ∈ A, b ∈ B}

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Function
Function : A function f : A → B is a rule that associate every
element of A with a unique element of B.

f (x) is the image of element x under this function and x is the

preimage of f (x). Set A is called the domain of the function and the
set B is called the co-domain of the function.

The set of all elements of B that are covered by the function is called
the range of the function.

R = {y ∈ B|∃x ∈ A, f (x) = y }
V. Prakash Singh, PhD
Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Types of Functions

Injective Function : A function f is called an injective function if

every element in R (range of f) has a unique pre-image in A. It is also
called a one to one function.

Surjective Function : A function f is called a surjective function if the

co-domain and range of the function is identical. The function does
not leave any element in B.

Bijective Function : A function, which is both injective and surjective,

is called a bijective function. For a bijective function, f −1 is also a
valid function.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Cardinality of a Set

Cardinality : The cardinality of a set is the number of elements in the

set. It is used to compare the sizes of two sets.

For finite sets, we can count the number of elements in the sets and
compare their sizes. For infinite sets, we can use the concept of
bijective functions to compare their sizes.

George Cantor, a German mathematician, extended the definition of

cardinality to sets with infinitely many elements.

For infinite sets, we use the concept of bijective functions to compare

their sizes.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Cardinality of a Set

Sets A and B are said to be equicardinal if there exist a bijective

function between A and B. We denote it as |A| = |B|

Sets B has cardinality greater than or equal to that of set A if there

exist an injective function from A to B. We write |B| >= |A|. This
means that there is at least as many elements in B as in A.

Sets B has cardinality strictly greater than that of set A if there exist
an injective function from A to B but no bijective function from A to
B. We denote it as |B| > |A|.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Countable Sets

A set is said to be countably infinite if it is equicardinal with the set

of natural number N i.e. there exist a bijective function between A
and N .

A set is a countable set if it is either finite or countably infinite.

The set of natural numbers N , the set of integers Z, the set of

positive integers Z + , the set of negative integers Z − , the set rational
numbers Q, the set of prime numbers etc are all countably infinite
(countable) sets.

To show a given set is countably infinite, we just need to put the

elements of the set in a list and every element is covered in that list.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Countable Sets
Example : Show that the set of rational numbers between 0 and 1 i.e.
A = Q ∩ [0, 1] is a countable set.

Solution : Since the given set is not finite, we need to prove that the
set is countable infinite. In other words, we need to show that a
bijective function exist between the set and the set of natural
numbers. We need to put the elements of the set as a list and the list
should cover all the elements of the set.
 
1 1 2 1 3 1
A = 0, 1, , , , , , , . . .
2 3 3 4 4 5

The list covers every rational number between 0 and 1. Therefore A

is a countable set.

A countable union of a countable sets is countable. Using this we can

show that the set of irrational numbers is a countable set.
V. Prakash Singh, PhD
Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Uncountable Sets

A set is uncountable if it is not countable. A set A is uncountable if it

has cardinality strictly larger than that of set of natural numbers.
|A| > |N |.

In other words, we need to show that there exist an injective function

from N to A but no bijective function exist from N to A. Then the
set A is an uncountable set.

The set of real numbers R, the set of irrational numbers R\Q, any
interval on real line i.e. [0, 1], the set of all infinite length binary
strings {0, 1}∞ , power set of N are all uncountable sets.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Outline

1 Introduction

2 Mathematical Preliminaries

3 Axiomatic theory of Probability

4 Conditional Probability

5 Independence

6 Combined Experiments

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Axiomatic Theory of Probability

Andrei Kolmogorov, a Russian mathematician, is credited with laying

the modern foundation of probability theory through his work in the
early 20th century.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Probabilistic Models

Probabilistic model is a mathematical framework to study a

phenomenon or an experiment with uncertain outcomes.

We start with a random experiment whose outcomes can not be

predicted with certainty. Every time we perform the experiment ( a
trial), there will be an outcome of the experiment. We denote this
outcome as ω

The source of uncertainty or randomness in the experiment is that we

can not tell with certainty what this outcome ω will be in a trial.

Probabilistic model of a random experiment consists of two elements

(1) Sample space and (2) Probability measure.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Probabilistic Models

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Sample Space

Set of all possible outcomes of an experiment is called the sample

space of the experiment. It is denoted by Ω. Outcomes are called
elements of the sample space.

The elements of the sample space should be mutually exclusive and

collectively exhaustive.

In a trial of a random experiment, there must be one and only one

outcome (element) from the sample space.

Sample space of an experiment is not unique and the preferred

sample space depends on the problem of interest.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Examples of Sample Space

Tossing a Coin once and we are interested in determining whether it
shows head or Tail.
Ω = {H, T } Sample space is finite
Rolling a die and we are interested in determining what number it
shows on the face.
Ω = {1, 2, 3, 4, 5, 6} Sample space is finite
Tossing a coin n times and we are interested in all the faces that show
up.
Ω = {H, T }n Sample space is finite
Tossing a coin infinitely many times and we are interested in all the
faces that show up.
Ω = {H, T }∞ Sample space is Uncountable
V. Prakash Singh, PhD
Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Examples of Sample Space

Toss a coin until a head appears, and we are interested in counting

the number of times the coin is tossed.

Ω = {1, 2, 3, 4, 5, . . . } Sample space is countably infinite

Throwing a dart on a unit square and we are interested in the

coordinates of the landing point of dart

Ω = {(x, y )|0 ≤ x, y ≤ 1} Sample space is uncountable

Tossing a pair of dice and we are interested in both the numbers that
show up.

Ω = {(1, 1), (1, 2), (1, 3), . . . , (6, 5), (6, 6)} Sample space is finite

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Events

The subsets of the sample space are called events. At each trial of an
experiment, we observe an outcome ωi . We say that the event A has
occurred during this trial if ωi ∈ A.

In this lecture, we will assume that all subsets of Ω are events. This is
not true in general. Only a class F of subset of Ω are called events.

The main reason, however, for not including all subsets of Ω in the
class F of events is of a mathematical nature. In certain cases
involving sets with infinitely many elements, it is impossible to assign
probabilities to all subsets satisfying the probability axioms.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Events

Example : Consider an experiment of rolling a die once and we are

interested in the number obtained on the face. The sample space is
given as Ω = {1, 2, 3, 4, 5, 6}
We define two events A : the number is even 2), B : the number is
greater than 3.

A = {2, 4, 6}
B = {4, 5, 6}

In a trial of an experiment, we gets the outcome ωi = 2. Since

wi ∈ A we say that event A has occurred and ωi ∈ / B, the event B
has not occurred in this trial.

Two events A and B are called mutually exclusive or disjoint if there

is no common elements in set A and B i.e. A ∩ B = Φ.
V. Prakash Singh, PhD
Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Probability Law
The probability law assigns a number P(A) to every event A, called
the probability of the event. The probability of an event is a measure
of how likely the event is to occur when we run the experiment. P(A)
must satisfy the following Probability axioms.

Non Negativity : P(A) ≥ 0 for every event A

Normalization : P(Ω) = 1 Probability of a certain event is 1.

Finite Additivity : For two disjoint events A and B

P(A ∪ B) = P(A) + P(B)
For a countably infinite sample space we can write the countable
additivity axiom for a sequence of disjoint events A1 , A2 , . . .
[∞  X ∞
P Ai = P(Ai )
i=1 i=1
V. Prakash Singh, PhD
Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Few Results using the Probability Axioms

The following results can be proved using the probability axioms and
basic set theory.

For any event A

P(A) ≤ 1
P(Ac ) = 1 − P(A)
The probability of null event is 0 P(Φ) = 0

If A ⊆ B This implies
P(A) ≤ P(B)
For two events A and B
P(A ∪ B) = P(A) + P(B) − P(A ∩ B)
P(A ∪ B) ≤ P(A) + P(B) Union Bound
V. Prakash Singh, PhD
Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Few Results using the Probability Axioms

For three events A, B and C

P(A ∪ B ∪ C ) = P(A) + P(Ac ∩ B) + P(Ac ∩ B c ∩ C )

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Probability Law for Finite Sets

If the sample space Ω is finite, then the probability law is specified by

the probability of the events consisting of a single element P({ωi }).
An event A consists of k elements A = {ω1 , ω2 , . . . , ωk }, the
probability of this event is computed as

P(A) = P({ω1 , ω2 , . . . , ωk }) = P({ω1 }) + P({ω1 }) + · · · + P({ωk })

If the sample space Ω is finite with n outcomes and every outcome is

equally likely, then the probability of events consisting of a single
element P({ωi }) = n1 ∀ i. An event A consists of k elements
A = {ω1 , ω2 , . . . , ωk }, the probability of this event is computed as

k
P(A) = P({ω1 , ω2 , . . . , ωk }) = P({ω1 })+P({ω1 })+· · ·+P({ωk }) =
n

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Example

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Example

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Example

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Outline

1 Introduction

2 Mathematical Preliminaries

3 Axiomatic theory of Probability

4 Conditional Probability

5 Independence

6 Combined Experiments

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Conditional Probability

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Conditional Probability

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Conditional Probability
The conditional probability of an event A given that the event B has
occurred is denoted as P(A|B) and computed as

P(A ∩ B)
P(A|B) =
P(B)

We assume that P(B) > 0

In other words, we want to find what fraction of the total probability

of event B is assigned to the outcomes under which event A also
occurs.
In the example in the last slide, we can find the conditional probability
of A given that event B has occurred as
P(A ∩ B) 2/12
P(A|B) = = = 1/3
P(B) 6/12
V. Prakash Singh, PhD
Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Conditional Probability

The conditional probability is a valid probability law and satisfies all

the axioms of probability.

Non Negativity : P(A|C ) ≥ 0 for every event A and C

Normalization : P(Ω|C ) = 1 Probability of a certain event

conditioned on event C is 1.

Finite Additivity : For two disjoint events A and B

P((A ∪ B)|C ) = P(A|C ) + P(B|C )

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Multiplication Rule

The probability that two events occur is equal to the product of the
probability that first event occurs and the probability that the second
event occurs given that the first event has occurred. The order of the
events can be changed.

P(A ∩ B) = P(A)P(B|A) = P(B)P(A|B)

The multiplication rule can be generalized for a sequence of events

A1 , A2 , . . . , An as
n
Y
P(A1 ∩ A2 ∩ · · · ∩ An ) = P(A1 ) P(Ai |A1 ∩ A2 ∩ · · · ∩ Ai−1 )
i=1

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Multiplication Rule

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Total Probability Rule

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Total Probability Rule

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Total Probability Rule

We can generalize the total probability rule as

n
X
P(B) = P(Ai )P(B|Ai )
i=1

where Ai are the disjoint events that form a partition of the sample
space and B is any event.

The probability that an event occurs is a weighted average of the

probability that the event occurs under different scenarios where the
weights are the probability of the different scenarios.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Bayes’ Rule

Bayes’ Rule provides a systematic approach for using new information

about a subsequent event to revise the probability of the initial events

A prior probability is an initial probability value originally assigned

before any additional information is obtained.

A posterior probability is a probability value that has been revised by

using additional information that is later obtained.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Bayes’ Rule
We start with a probabilistic model where we know the prior
probabilities of different scenarios Ai i.e. P(Ai ). Given that the event
B has occurred in a trial and we wish to use this information to revise
the probabilities of Ai . These revised probabilities P(Ai |B) are known
as posterior probabilities.

P(Ai ∩ B)
P(Ai |B) =
P(B)
P(Ai )P(B|Ai )
=
P(B)
P(Ai )P(B|Ai )
= Pn
j=1 P(Aj )P(B|Aj )

where last equality follows from the total probability rule and events
Aj j = 1, 2, . . . , n form a partition of the sample space.
V. Prakash Singh, PhD
Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Examples

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Examples

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Examples

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Examples

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Examples

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Examples

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Examples

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Examples

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Outline

1 Introduction

2 Mathematical Preliminaries

3 Axiomatic theory of Probability

4 Conditional Probability

5 Independence

6 Combined Experiments

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Independence of two events

If occurrence of an event A does not give any new information about

event B or vice versa, these events are independent and we can write

P(B) = P(B|A)
P(A) = P(A|B)

In other words, we say that the two events A and B are (statistically)
independent if the probability that event A and B occur is equal to
the product of the probabilities of event A and B.

P(A ∩ B) = P(A)P(B)

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Independence and Disjointness

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Conditional Independence

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Independence of a collection of events

A collection of events are independent if information about some of

the events does not change the probabilities related to the remaining
events.

A collection of events A1 , A2 , . . . An are called independent if

P(Ai ∩ Ai+1 ∩ · · · ∩ Am ) = P(Ai )P(Ai+1 ) . . . P(Am )

for all i and m such that 1 ≤ i < m ≤ n.

Three events A, B and C are independent if the following holds

P(A ∪ B) = P(A)P(B), P(B ∪ C ) = P(B)P(C )

P(A ∪ C ) = P(A)P(C ), P(A ∩ B ∩ C ) = P(A)P(B)P(C )

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Pairwise Independence

A collection of events A1 , A2 , . . . ,An are called pair-wise independent

if the following holds for all possible combinations of i and j.

P(Ai ∩ Aj ) = P(Ai )P(Aj )

Three events A, B and C are called pair-wise independent if

P(A ∪ B) = P(A)P(B)
P(B ∪ C ) = P(B)P(C )
P(A ∪ C ) = P(A)P(C )

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Pair-wise Independence

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Example

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Example

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Example

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Outline

1 Introduction

2 Mathematical Preliminaries

3 Axiomatic theory of Probability

4 Conditional Probability

5 Independence

6 Combined Experiments

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Combined Experiments

Two or more experiments can be suitably combined to form a single

experiment. Consider two experiments with sample spaces Ω1 and Ω2
respectively. The sample space Ω of the combined experiment is the
product set of Ω1 and Ω2 .

Ω = Ω1 × Ω2 = {(ω1 , ω2 )|ω1 ∈ Ω1 , ω2 ∈ Ω2 }

If Ω1 has M elements and Ω2 has N elements, the sample space Ω of

the combined experiment will have MN elements.

Consider an event A defined on the first experiment and event B

defined on the second experiment then the product set C = A × B
will be an event defined on the combined experiment.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Combined Experiments

Consider a combined experiment of two sub experiments. The

probability law for a combined experiment is defined as follows. We
consider that both the experiments are independent and for an event
A × B on Ω
P(A × B) = P(A)P(B)
For independent experiments, probability law for a combined
experiment is completely specified in terms of the probability laws of
individual experiments.

For N independent experiments, we can generalize as

P(A1 × A2 × · · · × An ) = P(A1 )P(A2 ) . . . P(An )

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Combined Experiments (Example)

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Combined Experiments (Example)

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Counting

The calculation of probabilities often involves counting the number of

elements in the event set. When the sample space has finite number
(say n) of equally likely outcomes and the event A consist of k
elements, we can write P(A) = kn .

Permutations There are n distinct objects and k ≤ n is a positive

integer. The total number of ways we can pick k objects and arrange
them in a sequence (permutation) is given as

n!
n(n − 1) . . . (n − k + 1) =
(n − k)!

When k = n, the number of possible sequences of n objects is

n.(n − 1) . . . 3.2.1 = n!

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Counting

Combinations : There are n distinct objects and k ≤ n is a positive

integer. The total number of combinations of k objects is given as
 
n n!
=
k k!(n − k)!

In a combination, order of selected objects is not important.

The number of k-permutations is equal to the number of

k-combinations multiplied by the number of ways in which k objects
can be ordered.

The number of k-combinations out of n objects is same as the

number of k-element subsets of a given set with n elements.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Bernoulli Trial

An experiment for which there are only two outcomes 1) Success and
2) Failure in a trial, is called a Bernoulli experiment. For example :
flipping a coin, passing or failing in an exam etc. Any of the two
outcome can be used as success or failure.

Fixed number of independent Bernoulli trials is known as a Binomial

experiment. Example: n independent tosses of a biased coin in which
the probability of head is p.

The Binomial experiment can also be though as a combined

experiment of n sub experiments where each sub experiment is a
Bernoulli experiment.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Binomial Probabilities

Let us consider n independent Bernoulli trials where in each trial the

probability of success is p.

Now let us consider the probability

p(k)
= P(k successes in n independent trials)
 
n k
= p (1 − p)n−k
k

The numbers kn are known as binomial coefficients and the

probabilities p(k) are called binomial probabilities.

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability
Introduction Mathematical Preliminaries Axiomatic theory of Probability Conditional Probability Independence Combined Experiments

Example

V. Prakash Singh, PhD

Department of Electronics and Communication Engineering, National Institute of Technology Andhra Pradesh, Tadepalligudem, Andhra Pradesh India
Lec 1: Introduction to Probability