University of Benghazi

Faculty of Engineering
Electrical and Electronics Engineering Department

Probability and Random Process Course

EE277

Salma Elkawafi
[email protected]
Goals
Understand the following:

Probability Bounds
• The Union Bound and Extension
• Markov and Chebyshev Inequalities

Limit Theorems
• Low of large number
• Central Limit Theorem
Probability Bounds
Probability bounds: These inequalities usually apply to a
general scenario. There are several scenarios in which we
resort to inequalities. Sometimes we do not have enough
information to calculate a desired quantity (such as the
probability of an event or the expected value of a random
variable). In other situations, the problem might be
complicated and exact calculation might be very difficult. In
other scenarios, we might want to provide a result that is
general and applicable to wide range of problems.
Probability Bounds
The Union Bound and Extension:
The union bound or Boole's inequality is applicable when you
need to show that the probability of union of some events is
less than some value. Remember that for any two events A and
B we have

𝑃(𝐴 ∪ 𝐵) = 𝑃(𝐴) + 𝑃(𝐵) − 𝑃(𝐴 ∩ 𝐵) ≤ 𝑃(𝐴) + 𝑃(𝐵).

Similarly, for three events A, B, and C, we can write

𝑃(𝐴 ∪ 𝐵 ∪ 𝐶) = 𝑃((𝐴 ∪ 𝐵) ∪ 𝐶)
≤ 𝑃(𝐴 ∪ 𝐵) + 𝑃(𝐶)
≤ 𝑃(𝐴) + 𝑃(𝐵) + 𝑃(𝐶).
Probability Bounds
The Union Bound and Extension:
In general,

The Union Bound

For any events 𝐴1 , 𝐴2 , … , 𝐴𝑛 , we have

𝑛 𝑛

𝑃 ራ 𝐴𝑖 ≤ ෍ 𝑃(𝐴𝑖 )
𝑖=1 𝑖=1
Probability Bounds
The Union Bound and Extension:
Example:
Your friend tells you that he had four job interviews last week. He says that based on
how the interviews went, he thinks he has a 20% chance of receiving an offer from
each of the companies he interviewed with. Nevertheless, since he interviewed with
four companies, he is 90% sure that he will receive at least one offer. Is he right?
Solution:
Let 𝐴𝑖 be the event that your friend receives an offer from the 𝑖th company, 𝑖
= 1,2,3,4. Then, by the union bound:
4 4

𝑃 ራ 𝐴𝑖 ≤ ෍ 𝑃 𝐴𝑖 = 0.2 + 0.2 + 0.2 + 0.2 = 0.8

𝑖=1 𝑖=1

Thus, the probability of receiving at least one offer is less than

or equal to 80%.
Probability Bounds
Markov Inequality
Let 𝑋 be any positive continuous random variable, we can write
∞
𝐸[𝑋] = න 𝑥𝑓𝑋 (𝑥)𝑑𝑥
−∞
∞
= ‫׬‬0 𝑥𝑓𝑋 (𝑥)𝑑𝑥 (since 𝑿 is positive-valued)
∞
≥ ‫𝑥𝑑)𝑥( 𝑋𝑓𝑥 𝑎׬‬ (for any 𝑎 > 0)
∞
≥ ‫𝑥𝑑)𝑥( 𝑋𝑓 𝑎 𝑎׬‬ (since 𝑥 > 𝑎 in the integrated region)
∞
= 𝑎 ‫𝑥𝑑)𝑥( 𝑋𝑓 𝑎׬‬
= 𝑎𝑃(𝑋 ≥ 𝑎).

Thus, we conclude
𝐸𝑋
𝑃 𝑋≥𝑎 ≤ , for any 𝑎 > 0.
𝑎

We can prove the above inequality for discrete similarly (using the generalized PDF).
Probability Bounds
Chebyshev Inequality
Let 𝑋 be any random variable. If you define 𝑌 = 𝑋 − 𝐸 𝑋 2 , then 𝑌 is a nonnegative
random variable, so we can apply Markov's inequality to 𝑌. In particular, for any positive
real number 𝑏, we have
𝐸𝑌
𝑃(𝑌 ≥ 𝑏2 ) ≤ 2
𝑏
But note that
𝐸 𝑌 = 𝐸[ 𝑋 − 𝐸𝑋 2 ] = 𝑉𝑎𝑟 𝑋 ,
𝑃(𝑌 ≥ 𝑏2 ) = 𝑃( 𝑋 − 𝐸 𝑋 2 ≥ 𝑏2 ) = 𝑃(|𝑋 − 𝐸𝑋| ≥ 𝑏).
Thus, we conclude that
𝑉𝑎𝑟 𝑋
𝑃(|𝑋 − 𝐸[𝑋]| ≥ 𝑏) ≤ .
𝑏2

Chebyshev's inequality states that the difference between 𝑋 and 𝐸𝑋 is somehow limited
by 𝑉𝑎𝑟(𝑋). This is intuitively expected as variance shows on average how far we are
from the mean
Probability Bounds
Markov and Chebyshev Inequalities
Markov's Inequality
If 𝑋 is any nonnegative random variable, then,
𝐸𝑋
𝑃 𝑋≥𝑎 ≤ , for any 𝑎 > 0.
𝑎

Chebyshev's Inequality
If X is any random variable, then for any b>0 we have
𝑉𝑎𝑟 𝑋
𝑃 𝑋−𝐸 𝑋 ≥𝑏 ≤ , for any 𝑏 > 0
𝑏2
 Probability Bounds
Markov and Chebyshev Inequalities
Example:
Let 𝑋 ∼ 𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙(𝑛, 𝑝). Using Markov's inequality, find an upper bound on 𝑃(𝑋 ≥ 𝛼𝑛),
1 3
where 𝑝 < 𝛼 < 1. Evaluate the bound for 𝑝 = 2 and 𝛼 = 4 .
Solution:
𝐸𝑋
𝑃 𝑋≥𝑎 ≤ ,
𝑎
𝐸 𝑋 = 𝑛𝑝
𝑎 = 𝛼𝑛

𝑛𝑝 𝑝
𝑃 𝑋 ≥ 𝛼𝑛 ≤ = ,
𝛼𝑛 𝛼

For 𝑝 = 1/2 and 𝛼 = 3/4, we obtain

3𝑛 2
𝑃(𝑋 ≥ )≤ .
4 3
 Probability Bounds
Markov and Chebyshev Inequalities
Example:
Let 𝑋 ∼ 𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙(𝑛, 𝑝). Using Chebyshev's inequality, find an upper bound on 𝑃(𝑋 ≥ 𝛼𝑛)
1 3
, where 𝑝 < 𝛼 < 1. Evaluate the bound for 𝑝 = 2 and 𝛼 = 4 .
Solution:
𝑉𝑎𝑟 𝑋
𝑃 𝑋−𝐸 𝑋 ≥𝑏 ≤ ,
𝑏2
𝑉𝑎𝑟 𝑋 = 𝑛𝑝𝑞
𝑉𝑎𝑟 𝑋
𝑃 𝑋 ≥ 𝛼𝑛 = 𝑃 𝑋 − 𝑛𝑝 ≥ 𝛼𝑛 − 𝑛𝑝 ≤ 𝑃 |𝑋 − 𝑛𝑝| ≥ 𝛼𝑛 − 𝑛𝑝 ≤
𝛼𝑛 − 𝑛𝑝 2
𝑛𝑝𝑞 𝑝𝑞
𝑃 𝑋 ≥ 𝛼𝑛 ≤ =
𝛼𝑛 − 𝑛𝑝 2 𝑛 𝛼 − 𝑝 2
For 𝑝 = 1/2 and 𝛼 = 3/4, we obtain
3𝑛 4
𝑃(𝑋 ≥ ) ≤ .
4 𝑛
 Limit Theorem
• Low of large number (LLN)
The LLN basically states that the average of a large number of i.i.d. random
variables converges to the expected value.

• Central Limit Theorem

The CLT states that, under some conditions, the sum of a large number of random
variables has an approximately normal distribution.
 Low of large number (LLN)
• Low of large number (LLN)
It states that if you repeat an experiment independently a large number of times and average
the result, what you obtain should be close to the expected value. There are two main versions of
the law of large numbers. They are called the weak and strong laws of the large numbers.

Definition: For i.i.d. random variables 𝑋1 , 𝑋2 , … , 𝑋𝑛 , the sample mean, denoted by 𝑋ത , is defined as
𝑋 +𝑋 +⋯.+𝑋𝑛
𝑋ത = 1 2 𝑛
𝑋ത is a random variable
Another common notation for the sample mean is 𝑀𝑛 (𝑋).

𝐸 𝑋 +𝐸 𝑋2 +⋯.+𝐸 𝑋𝑛 𝑛𝐸[𝑋]
𝐸 𝑋ത = 1 𝑛
= 𝑛
= 𝐸[𝑋]

Also, the variance of 𝑋ത is given by

𝑉𝑎𝑟 𝑋1 + 𝑋2 + ⋯ + 𝑋𝑛 𝑉𝑎𝑟(𝑋1 ) + 𝑉𝑎𝑟(𝑋2 ) + ⋯ + 𝑉𝑎𝑟(𝑋𝑛 ) 𝑛𝑉𝑎𝑟 𝑋 𝑉𝑎𝑟 𝑋
𝑉𝑎𝑟 𝑋ത = = = =
𝑛2 𝑛2 𝑛2 𝑛
 Low of large number (LLN)

• Low of large number (LLN)

Weak law of large numbers (WLLN).

Let 𝑋1 , 𝑋2 , … , 𝑋𝑛 be 𝑖. 𝑖. 𝑑. random variables with a finite expected value 𝐸𝑋𝑖 = 𝜇 < ∞.
Then, for any 𝜖 > 0,
lim 𝑃(|𝑋ത − 𝜇| ≥ 𝜖) = 0
𝑛→∞

Using Chebyshev's Inequality: (If 𝑋 is any random variable, then for any 𝑏 > 0 we have)
𝑉𝑎𝑟 𝑋
𝑃(|𝑋 − 𝐸[𝑋]| ≥ 𝑏) ≤
𝑏2

𝑉𝑎𝑟 𝑋ത 𝑉𝑎𝑟 𝑋
𝑃 𝑋ത − 𝜇 ≥ 𝜖 ≤ =
𝜖2 𝑛𝜖 2
 Central Limit Theorem
The central limit theorem (CLT) is one of the most important results in probability theory. It
states that, under certain conditions, the sum of a large number of random variables is
approximately normal.
Suppose that 𝑋1 , 𝑋2 , … , 𝑋𝑛 are i.i.d. random variables with expected values 𝐸[𝑋𝑖 ] = 𝜇 < ∞
𝑋 +𝑋 +⋯.+𝑋𝑛
and variance 𝑉𝑎𝑟(𝑋𝑖 ) = 𝜎 2 < ∞, and the sample mean 𝑋ത = 1 2 𝑛
𝜎2
has mean 𝐸[𝑋]
ത = 𝜇 and variance 𝑉𝑎𝑟(𝑋)
ത = . Then, the random variable
𝑛

𝑋ത − 𝜇 𝑋1 + 𝑋2 + ⋯ . +𝑋𝑛 − 𝑛𝜇
𝑍𝑛 = =
𝜎/ 𝑛 𝑛𝜎
converges in distribution to the standard normal random variable as 𝑛 goes to infinity, that is
lim 𝑃(𝑍𝑛 ≤ 𝑥) = Φ(𝑥), 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑥 ∈ 𝑅,
𝑛→∞
where Φ(𝑥) is the standard normal CDF.
 Central Limit Theorem𝑍 = 𝑋ത − 𝜇 = 𝑋 + 𝑋 + ⋯ . +𝑋
𝑛
1 2 𝑛 − 𝑛𝜇
𝜎/ 𝑛 𝑛𝜎
• Central Limit Theorem
Let's assume that 𝑋𝑖 ’s are 𝐵𝑒𝑟𝑛𝑜𝑢𝑙𝑙𝑖 𝑝 .
Then
𝐸 𝑋𝑖 = 𝑝,
𝑉𝑎𝑟(𝑋𝑖) = 𝑝(1 − 𝑝).

Also, 𝑌𝑛 = 𝑋1 + 𝑋2 + ⋯ + 𝑋𝑛 has
𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙(𝑛, 𝑝) distribution.

Thus,
𝑌𝑛 − 𝑛𝑝
𝑍𝑛 =
𝑛𝑝(1 − 𝑝)

where 𝑌𝑛 ∼ 𝐵𝑖𝑛𝑜𝑚𝑖𝑎𝑙(𝑛, 𝑝)
 Central Limit Theorem
• Central Limit Theorem
How to Apply The Central Limit Theorem (CLT)

❑ Write the random variable of interest, 𝑌, as the sum of n random variable 𝑋𝑖 ′s:
𝑌 = 𝑋1 + 𝑋2 +. . . +𝑋𝑛

❑ Find 𝐸[𝑌] and 𝑉𝑎𝑟[𝑌] by noting that

𝐸 𝑌 = 𝑛𝜇, 𝑉𝑎𝑟(𝑌) = 𝑛𝜎 2 ,
where 𝜇 = 𝐸[𝑋𝑖 ] and 𝜎 2 = 𝑉𝑎𝑟(𝑋𝑖 ).

𝑌−𝐸[𝑌] 𝑌−𝑛𝜇
❑ According to the CLT, conclude that = 𝑛𝜎
is approximately standard normal; thus, to
𝑉𝑎𝑟 𝑌
find 𝑃(𝑦1 ≤ 𝑌 ≤ 𝑦2 ), we can write
𝑦1 − 𝑛𝜇 𝑌 − 𝑛𝜇 𝑦2 − 𝑛𝜇 𝑦2 − 𝑛𝜇 𝑦1 − 𝑛𝜇
𝑃(𝑦1 ≤ 𝑌 ≤ 𝑦2 ) = 𝑃( ≤ ≤ ) ≈ Φ( ) − Φ( ).
𝑛𝜎 𝑛𝜎 𝑛𝜎 𝑛𝜎 𝑛𝜎
 Central Limit Theorem
Example
A bank teller serves customers standing in the queue one by one. Suppose that the service time 𝑋𝑖 for
customer 𝑖 has mean 𝐸[𝑋𝑖 ] = 2 (minutes) and 𝑉𝑎𝑟(𝑋𝑖 ) = 1. We assume that service times for different
bank customers are independent. Let 𝑌 be the total time the bank teller spends serving 50 customers.
Find 𝑃(90 < 𝑌 < 110)
Solution
𝑦1 − 𝑛𝜇 𝑌 − 𝑛𝜇 𝑦2 − 𝑛𝜇 𝑦2 − 𝑛𝜇 𝑦1 − 𝑛𝜇
𝑃(𝑦1 ≤ 𝑌 ≤ 𝑦2 ) = 𝑃( ≤ ≤ ) ≈ Φ( ) − Φ( ).
𝑛𝜎 𝑛𝜎 𝑛𝜎 𝑛𝜎 𝑛𝜎
𝑌 = 𝑋1 + 𝑋2 +. . . +𝑋𝑛 ,
Where,
𝑛 = 50,
𝐸 𝑋𝑖 = 𝜇 = 2,
𝑉𝑎𝑟 𝑋𝑖 = 𝜎 2 = 1.
90 − 100 𝑌 − 𝑛𝜇 110 − 100
𝑃 90 < 𝑌 < 110 = 𝑃 < <
50 𝑛𝜎 50
𝑌−𝑛𝜇
By the CLT, is approximately standard normal, so we can write
𝑛𝜎
𝑃 90 < 𝑌 < 110 ≈ Φ − 2 − Φ 2 = 0.8427
 Central Limit Theorem
Example
In a communication system each data packet consists of 1000 bits. Due to the noise, each bit may be
received in error with probability 0.1. It is assumed bit errors occur independently. Find the probability
that there are more than 120 errors in a certain data packet.
Solution
Let us define 𝑋𝑖 as the indicator random variable for the 𝑖th bit in the packet. That is, 𝑋𝑖 = 1 if the ith bit
is received in error, and 𝑋𝑖 = 0 otherwise. Then the 𝑋𝑖 ′s are i.i.d. and 𝑋𝑖 ∼ 𝐵𝑒𝑟𝑛𝑜𝑢𝑙𝑙𝑖(𝑝 = 0.1). If 𝑌 is
the total number of bit errors in the packet, we have
𝑌 = 𝑋1 + 𝑋2 +. . . +𝑋𝑛 ,

Since 𝑋𝑖 ∼ 𝐵𝑒𝑟𝑛𝑜𝑢𝑙𝑙𝑖(𝑝 = 0.1), we have,

𝑛 = 1000
𝐸 𝑋𝑖 = 𝜇 = 𝑝 = 0.1
𝑉𝑎𝑟 𝑋𝑖 = 𝜎 2 = 𝑝 1 − 𝑝 = 0.09

𝑌 − 𝑛𝜇 120 − 𝑛𝜇 𝑌 − 𝑛𝜇 120 − 100 20

𝑃 𝑌 > 120 = 𝑃 > =𝑃 > ≈1−Φ = 0.0175
𝑛𝜎 𝑛𝜎 𝑛𝜎 90 90