Introduction to

Probability
INSTRUCTOR: SHAHROKH FARAHMAND
Basic definitions
Probability space:

 Consists of sample space, sigma-field of events, probability function

 Sample space: All possible outcomes of a particular experiment
 Basic Event: Each member of which can be an outcome of the
experiment
 Event: A collection of basis events and a subset of sample space, after an
experiment is carried out an event occurs if the experiment yields one of the
basic events that are present in that particular event
 Example: finite sample space
 Number of distinct E=
 Sure event E=
 Empty event=

SLIDES
SLIDES BY:BY SHAHROKH
SHAHROKH FARAHMAND
FARAHMAND 2
Examples
Ex1. One coin toss
Ex2. Two coin tosses

Ex3. Number of hairs on a person’s head

Ex4. Age of married couples

Ex5. Power across a resistor

SLIDES BY: SHAHROKH FARAHMAND

SLIDES BY: SHAHROKH FARAHMAND 3
Various scenarios for sample space
Sample space is a finite set
 No need to define sigma-fields concept
 Use all possible subsets of as the sigma field of events
 Probability can be assigned in a consistent manner to all events (clarified
later on)
Sample space is an infinite countable set
 Need sigma-field concept
 Probability can be assigned in a consistent manner to al events
Sample space is infinite uncountable
 Need sigma-field concept
 Probability can not be assigned in a consistent manner to all sets (Cantor set)
 Sets of importance to us are consistent
SLIDES BY:BY:
SLIDES SHAHROKH FARAHMAND
SHAHROKH FARAHMAND 4
Set Algebra
Consider E,F to be two sets
Well-known properties

DeMorgan Laws:

Generalization:

SLIDES BY: SHAHROKH FARAHMAND 5

SLIDES BY: SHAHROKH FARAHMAND
Sigma-field for a sample space
Consider a sample space and a collection of its subsets as
is a field if
1)
2)
3)
Note that set of all subsets for a finite sample space is a field
Furthermore, is a sigma-field if:

SLIDES BY: SHAHROKH FARAHMAND 6

SLIDES BY: SHAHROKH FARAHMAND
Kolmogorov probability axioms
Remember
P[.] is a function that assigns to every event a number P[E]
called the probability of E such that

SLIDES BY: SHAHROKH FARAHMAND 7

SLIDES BY: SHAHROKH FARAHMAND
Some properties resulting from axioms

SLIDES BY: SHAHROKH FARAHMAND 8

SLIDES BY: SHAHROKH FARAHMAND
Examples for probability spaces
Ex1. One coin toss

Ex2. Tossing of a fair die

SLIDES BY: SHAHROKH FARAHMAND 9

SLIDES BY: SHAHROKH FARAHMAND
Examples (continued)
Ex3. Picking at random a ball from balls numbered 1-12 from the urn

Ex4. Toss of two fair dies

SLIDES BY: SHAHROKH FARAHMAND 10

SLIDES BY: SHAHROKH FARAHMAND
 Motivation for union bound

Question. Relationship between independent sets and disjoint sets?

Exact solution is complicated and of limited use. Apply union bound instead
SLIDES BY: SHAHROKH FARAHMAND 11
SLIDES BY: SHAHROKH FARAHMAND
Union bound

Bound is tight when sets are disjoint

Note that RHS should be less than one for non-triviality
Wide applications in performance analysis

SLIDES BY: SHAHROKH FARAHMAND 12

SLIDES BY: SHAHROKH FARAHMAND
Union bound application
Ex. Digital communications: 4-QAM (4-PSK) through AWGN

Complex model but I,Q components are real

Suppose is transmitted

SLIDES BY: SHAHROKH FARAHMAND 13

SLIDES BY: SHAHROKH FARAHMAND
Application (continued)

Union bound leads to a simple PoE expression

When number of constellation points increases
 It becomes considerably more complex to apply the exact formula
 However, union bound remains almost as simple as 4-PSK

SLIDES BY: SHAHROKH FARAHMAND 14

SLIDES BY: SHAHROKH FARAHMAND
Joint probabilities
Frequency definition of probability
 Repeat independent experiments for a large number of times say n
 Count the number of times an event A has occurred
 We will have (approximately)

 Note that RHS is random but LHS is deterministic

 This observation will become rigorous later on via law of large numbers (LLN)
Joint probability
 Repeat the experiment n times but observe the number of times that both
events A and B have occurred

SLIDES BY: SHAHROKH FARAHMAND 15

SLIDES BY: SHAHROKH FARAHMAND
Joint probability example
Ex. Want to compute the probability for following events
 A: temperature equals or exceeds 10 C
 B: precipitation equals or exceeds 5 mm
 C: both A and B happen
We observed the weather for n=1000 days

Now, we can compute

SLIDES BY: SHAHROKH FARAHMAND 16

SLIDES BY: SHAHROKH FARAHMAND
Conditional probability
We want to know that given A has occurred what is the probability
that B occurs?
Use the same frequency interpretation: run n experiments, count
the number of occurrences of A or , count the number of
occurrences of both A and B

Hence, we define

SLIDES BY: SHAHROKH FARAHMAND 17

SLIDES BY: SHAHROKH FARAHMAND
 Examples
Ex1. Weather forcast.

Ex2. Binary Symmetric Channel

(BSC)
 For example a B-PAM with hard
Thresholding
 Best & worst values instead of 0.9?

SLIDES BY: SHAHROKH FARAHMAND 18

SLIDES BY: SHAHROKH FARAHMAND
Examples: BSC

SLIDES BY: SHAHROKH FARAHMAND 19

SLIDES BY: SHAHROKH FARAHMAND
Body building contest
 N contestants with different scores
 Contestants pass in front of jury sequentially and randomly
Judges must decide on the spot if the current contestant is best
If they let him pass, they can not elect them as the winner later
Question. What is the best strategy to make sure best contestant is
selected? What is the corresponding probability of success?
Formulate a similar problem. N unknown numbers (contestant score is
summarized into one overall number) are placed in a bag and are drawn
out randomly
First, we limit ourselves to the following strategy:
 We let first contestants pass but record the largest score =
 Then we select the first contestant that appears with a score >

SLIDES BY: SHAHROKH FARAHMAND 20

SLIDES BY: SHAHROKH FARAHMAND
Body building contest 2

Define X to be the draw with the largest number coming out

To make the correct decision, one should have

SLIDES BY: SHAHROKH FARAHMAND 21

SLIDES BY: SHAHROKH FARAHMAND
Body building contest 3

SLIDES BY: SHAHROKH FARAHMAND 22

SLIDES BY: SHAHROKH FARAHMAND
Law of total probability
Consider mutually exclusive
events meaning

Then

SLIDES BY: SHAHROKH FARAHMAND 23

SLIDES BY: SHAHROKH FARAHMAND
Proof

SLIDES BY: SHAHROKH FARAHMAND 24

SLIDES BY: SHAHROKH FARAHMAND
Independence
Two events A and B are called independent if and only if (iff)

This means that occurrence of B does not have an impact on probability of

event A and vice versa
Ex. Tossing of two fair die
Question1. What about tossing of a fair and unfair die?
Question2. Relation between being disjoint and independent?
Independence also means

SLIDES BY: SHAHROKH FARAHMAND 25

SLIDES BY: SHAHROKH FARAHMAND
Independence 2

SLIDES BY: SHAHROKH FARAHMAND 26

SLIDES BY: SHAHROKH FARAHMAND
Example 1
Urn containing black/white, heavy/light, even/odd balls

A: picking a black ball

B: picking a light ball
C: picking an even numbered ball

SLIDES BY: SHAHROKH FARAHMAND 27

SLIDES BY: SHAHROKH FARAHMAND
Example 1 (continued)
Are these events independent?

SLIDES BY: SHAHROKH FARAHMAND 28

SLIDES BY: SHAHROKH FARAHMAND
Example 2: BSC
Are two events indep?

Question. Do you think this is good or bad?

SLIDES BY: SHAHROKH FARAHMAND 29
SLIDES BY: SHAHROKH FARAHMAND
Example 2: BSC
Instead of (0.9,0.1)
Consider (0.5,0.5)
Are two events indep?

Question. Do you think this is good or bad?

SLIDES BY: SHAHROKH FARAHMAND 30
SLIDES BY: SHAHROKH FARAHMAND
Bayes theorem

Proof

Then apply law of total probability

SLIDES BY: SHAHROKH FARAHMAND 31

SLIDES BY: SHAHROKH FARAHMAND
Example. BSC
What is the following probability?

Aposteriori vs. apriori probabilities

SLIDES BY: SHAHROKH FARAHMAND 32

SLIDES BY: SHAHROKH FARAHMAND
Example. Cancer test
A= the test states that tested person has cancer
B= person really has cancer

Need to evaluate E[B|A]:

SLIDES BY: SHAHROKH FARAHMAND 33

SLIDES BY: SHAHROKH FARAHMAND
 Example. Guess the door

Bayes theorem offers the rigorous answer

SLIDES BY: SHAHROKH FARAHMAND 34

SLIDES BY: SHAHROKH FARAHMAND
 Example. Guess the door 2
Lets say award is positioned behind a door at random

Similarly

Note that it is not guaranteed to win the award. Remember the frequency
interpretation of probability SLIDES BY: SHAHROKH FARAHMAND 35
SLIDES BY: SHAHROKH FARAHMAND
Review of combinatorics
Ordered sample sets of size r (ex. CEO, VCEO, Secretary)
Suppose there are n numbered balls inside an urn and we take out r of
them at random.
How many cases are possible?
 Sampling with replacement

1 2 r
 Sampling without replacement

1 2 r

SLIDES BY: SHAHROKH FARAHMAND 36

SLIDES BY: SHAHROKH FARAHMAND
Review of combinatorics 2
Non-ordered samples of size r (Ex. Three board members)
Suppose there are n numbered balls inside an urn and we take out r of
them at random without replacement. How many cases are possible?
 Consider an unordered set of size r, how many ordered sets can be made
from that?
 Answer: r!
 Hence, number of unordered sets equals the number of ordered sets
divided by r!

SLIDES BY: SHAHROKH FARAHMAND 37

SLIDES BY: SHAHROKH FARAHMAND
Review of combinatorics 3

How many cases?

Ex. Tossing of 12 dies (E:= each face appears twice)

SLIDES BY: SHAHROKH FARAHMAND 38

SLIDES BY: SHAHROKH FARAHMAND
Occupancy problems
Problem1. Distributing randomly a set of r numbered balls into n
boxes. How many different cases?

Ex. r=3, n=2

Problem2. Distributing randomly a set of r similar balls into n boxes.

How many different cases?
Ex. r=3, n=2

SLIDES BY: SHAHROKH FARAHMAND 39

SLIDES BY: SHAHROKH FARAHMAND
Occupancy problems 2
Let us consider n boxes with n+1 lines:
Each ball is a *
Problem becomes number of possible orderings of r *’s and n-1 |’s

With fixed 2 lines at the ends, we want to select r members out of

n+r-1 elements

SLIDES BY: SHAHROKH FARAHMAND 40

SLIDES BY: SHAHROKH FARAHMAND
Example
For the case of r > n, determine the number of distinguishable cases
such that no box is empty?
To ensure n boxes are not empty, n *’s must be placed between each
two lines. Then, we need to select the position of r-n remaining balls
from (r-n)+n-1=r-1 positions, yielding

SLIDES BY: SHAHROKH FARAHMAND 41

SLIDES BY: SHAHROKH FARAHMAND
Bernoulli trials
Each experiment has two outcomes:
Success (s) with probability p
Failure (f) with probability q=1-p

Ex1. Flip of a coin

Ex2. Toss of a die with
Example. successive independent Bernoulli trials
3 Tosses of a biased coin: 8 possible outcomes

SLIDES BY: SHAHROKH FARAHMAND 42

SLIDES BY: SHAHROKH FARAHMAND
Example (continued)
What is the probability of two heads?
One of these three disjoint equiprobable events should happen

How to determine the number of events? Choose 2 out of 3

Sample space

SLIDES BY: SHAHROKH FARAHMAND 43

SLIDES BY: SHAHROKH FARAHMAND
Binomial law
N independent repetitions of a Bernoulli trial

Remember
Hence

Intuition: We have 0 s, or 1 s, or 2 s,…, or n s

SLIDES BY: SHAHROKH FARAHMAND 44

SLIDES BY: SHAHROKH FARAHMAND
Example1. Binary pulses

SLIDES BY: SHAHROKH FARAHMAND 45

SLIDES BY: SHAHROKH FARAHMAND
Example 2. Selecting an umpire

SLIDES BY: SHAHROKH FARAHMAND 46

SLIDES BY: SHAHROKH FARAHMAND
Example 3. Umpire selection time

SLIDES BY: SHAHROKH FARAHMAND 47

SLIDES BY: SHAHROKH FARAHMAND
Example 4. Missile strike

SLIDES BY: SHAHROKH FARAHMAND 48

SLIDES BY: SHAHROKH FARAHMAND
 Poisson law

Proof:

SLIDES BY: SHAHROKH FARAHMAND 49

SLIDES BY: SHAHROKH FARAHMAND
Evaluation of RHS limit

SLIDES BY: SHAHROKH FARAHMAND 50

SLIDES BY: SHAHROKH FARAHMAND
Example. Computer failure

SLIDES BY: SHAHROKH FARAHMAND 51

SLIDES BY: SHAHROKH FARAHMAND
Example. Random points on a line

SLIDES BY: SHAHROKH FARAHMAND 52

SLIDES BY: SHAHROKH FARAHMAND
Properties of Poisson law

SLIDES BY: SHAHROKH FARAHMAND 53

SLIDES BY: SHAHROKH FARAHMAND
Example. Switchboard saturation

Note the relation between queuing theory and this example

SLIDES BY: SHAHROKH FARAHMAND 54

SLIDES BY: SHAHROKH FARAHMAND
Example. Repetition coding
Consider the BSC with P0=P1=0.5
Instead of a 0 Tx 000
Instead of a 1 Tx 111
Majority voting at the Rx
Note that X only has two states but
Y has 8 states

SLIDES BY: SHAHROKH FARAHMAND 55

SLIDES BY: SHAHROKH FARAHMAND
Example. Repetition coding 2
We arrive at this equivalent
channel

Note that the channel quality

improves threefold:
Remember
Question. What is the cost paid?
Note that BSC passes bits independently. But, if transmitted bits have a
correlation, then they should be decoded jointly
SLIDES BY: SHAHROKH FARAHMAND 56
SLIDES BY: SHAHROKH FARAHMAND
Example. Erasure channel
Consider the BSC with P0=P1=0.5
We Tx two independent bits but
add a third odd parity bit

If the received bits have odd parity, then they are decoded. If it has even parity, an
erasure is declared meaning that there is certainly an error in the received bits.
Two options for erasure: 1) Ask for a retransmission
2) Use a longer outer code to correct the error
SLIDES BY: SHAHROKH FARAHMAND 57
SLIDES BY: SHAHROKH FARAHMAND
Example. Erasure channel 2

SLIDES BY: SHAHROKH FARAHMAND 58

SLIDES BY: SHAHROKH FARAHMAND
 Example. Erasure channel 3
From Bayes theorem

By symmetry, we get the same result for other probabilities of correct decoding
Note that this solution is a compromise between no coding and repetition coding:
1) We get better performance than no coding but worse than repetition coding
2) We get a better rate than repetition coding (1/3) but worse than no coding (1)
SLIDES BY: SHAHROKH FARAHMAND 59
SLIDES BY: SHAHROKH FARAHMAND
Thank You!

SLIDES BY: SHAHROKH FARAHMAND 60

SLIDES BY: SHAHROKH FARAHMAND