Probability theory

Sample spaces and events:

The SAMPLE SPACE of a statistical experiment is the set of all possible
outcomes. The outcomes are also known as SAMPLE POINTS.
An EVENT is a subset of the sample space.

Example 1.
Imagine I flip a coin, with two possible outcomes: heads (H) or tails (T). What is
the sample space for this experiment? What is the sample space for three flips in
a row?
Solution: For the first experiment (flip a coin once), the sample space is just
{H,T}.
For the second experiment (flip a coin three times), the sample space is
{HHH,HHT,HTH,HTT,THH,THT,TTH,TTT}. Order matters: HHT is a different outcome
than HTH.
Example 2.
For the experiment where I flip a coin three times in a row, consider the event
that I get exactly one T. Which outcomes are in this event?
Solution: The subset of the sample space that contains all outcomes with exactly
one T is {HHT,HTH,THH}.
Example 3.
Suppose I have two bowls, each containing 100 balls numbered 1 through 100. I
pick a ball at random from each bowl and look at the numbers on them. How
many elements are in the sample space for this experiment?
Solution: Using basic principles of counting, since the number of possible
outcomes for the second experiment doesn’t depend on the outcome of the first
experiment, the total number of possible outcomes is 1002 , or 10,000.

Probability definition
Probability denotes the possibility of the outcome of any random event and is
calculated by dividing the number of favorable events by the total number of
events. The probability value ranges from 0 to 1, with 0 representing uncertainty
and 1 representing certainty.
let’s assume that all outcomes in the sample space S are equally likely. If that is
the case, then the probability of an event E, which we write as P(E), is simply the
number of outcomes in E divided by number of outcomes in S:
P(E) = |E|/ |S|
Example1.
Imagine I flip a fair coin, with two possible outcomes: heads (H) or tails (T). What
is the probability that I get exactly one T if I flip the coin once? What if I flip it two
times?
Solution: First, note that I said it’s a fair coin. This is important, because it means
that on any one flip, each outcome is equally likely. If we only flip the coin once,
then the event we care about (getting T) has one possible outcome{T}, and the
sample space has two possible outcomes{H,T}, so the probability of getting T is
1/2.
If we flip the coin two times, there are two outcomes with exactly one T {HT,TH},
and four outcomes altogether {HH,HT,TH,TT}, so the probability of getting exactly
one T is 2/4.

Example 2.
Suppose I have two bowls, each containing 100 balls numbered 1 through 100. I
pick a ball at random from each bowl and look at the numbers on them. What is
the probability that the numbers add up to 200?
Solution: There is only one outcome in this event {100,100}, and sample space
contain 100x100= 10,000 outcomes altogether, so the probability of this
happening is only 1/10,000
Example 3.
Let E be the event that the numbers on the balls in the previous example add up
to exactly 51. What is the probability of E?
Solution: We already know the size of the sample space, but we also need to
determine the cardinality of E, i.e., the number of outcomes in this event.
Consider what happens if the first ball is 1. For the balls to add up to 51, it must
be the case that the second ball is 50. Similarly, if the first ball is 2, then the
second must be 49. And so on: for each possible first ball between 1 and 50, there
is exactly one second ball that will make the total equal 51. So the total number of
outcomes adding up to 51 is the same as the number of ways the first ball can be
between 1 and 50, which is to say, 50 ways. Therefore, the probability of E is
50/10,000, or 0.5%.
Example 4.
Suppose I choose a PIN containing exactly 4 digits, where each digit is chosen at
random and is equally likely to be any of the 10 digits 0-9. What is the probability
that my PIN contains four different digits?
Solution: First, consider the sample space S: all possible four-digit PINs. Using
our counting techniques (see Sets and Counting), we can compute that the
number of outcomes in this sample space is 104, or 10,000. Next, consider the
event E of interest: the set of PINs with four distinct digits. To compute the size of
this set, note that there are 10 possibilities for the first digit. Once that digit is
determined, there are only 9 possibilities for the second digit, because it must be
different from the first one. Then, 8 possibilities for the third digit (which must be
different to both of the first two), and 7 for the final digit. So, the total number of 5
PINs with four different digits is 10 x 9 x 8 x 7, or 5040.
Therefore P(E) = 5040/10,000, or very slightly more than 1/2.

Example 5.
Suppose I have a bowl with 3 green balls (g), 2 blue balls (b), and 1 red ball (r). I
draw a single ball at random from the bowl and report its color. What is the
probability I got a blue ball?
Solution: P(b) = 2/6 = 1/3 because two out of the six balls (outcomes) are blue.

MUTUALLY EXCLUSIVE (or DISJOINT) events

Mutually exclusive events, also known as disjoint events, are events that cannot
occur at the same time. For example, when you toss a coin, the result will be
either heads or tails, but not both.
Equally likely events
Equally likely events are events that have the same probability of
occurring. Some examples of equally likely events include:
• Tossing a coin
When you toss a fair coin, the probability of getting heads or tails is 1/2.
• Rolling a die
When you roll a fair six-sided die, each number (1 through 6) has an equal
probability of 1/6.

Mutually exhaustive events

Mutually exhaustive events are a set of events that together cover the entire
sample space of a given experiment. This means that one of the events in the set
must occur, regardless of the outcome.
Here’s a clear example to illustrate:
Example: Rolling a Six-Sided Die
• Sample Space: {1, 2, 3, 4, 5, 6}
• Events: Consider the events "rolling an even number" and "rolling an odd
number."
These events are mutually exhaustive because:
• The event "rolling an even number" covers {2, 4, 6}.
• The event "rolling an odd number" covers {1, 3, 5}.
• Together, these two events cover the entire sample space {1, 2, 3, 4, 5, 6}.

In summary, mutually exhaustive events ensure that the entire sample space is
covered, guaranteeing that one of the events in the set will occur in any outcome.

Probability distributions

A partition of the sample space into four mutually exclusive events. The rectangle
represents all the outcomes in the entire sample space, and each labelled region
represents the outcomes in that event.Such a set is called a PARTITION of the
sample space, and PARTITION is visualized in Figure. A simple example would be
flipping a coin, where S = {H,T} and we define n = 2 mutually exclusive events, E1
= {H} and E2 = {T}.
We then assign a probability , P(Ei), to each of the events Ei in the partition and
the set of values {P(E1)...P(En)} can be considered a PROBABILITY
DISTRIBUTION.

That is, every probability must fall between 0 and 1 (inclusive), and the sum of the
probabilities of the mutually exclusive events that cover the the sample space
must equal 1.

Combining events

Event complement and union

Suppose we have an event E that has probability P(E). What is the probability that
E does not happen? Put another way, what is the probability of the
COMPLEMENT of E, COMPLEMENT written as Ec or El is the set of outcomes that
are in the sample space S but not in E,

P(El) = 1−P(E).

To correctly compute P(A∪B) from P(A) and P(B), we need to subtract P(A∩B) that
is common in both A and B. When A and B are mutually exclusive, it is true that
P(A∪B) = P(A)+P(B), because there are no items in the intersection.

Joint probabilities(i.e. intersection A∩B) and the law of total probability

The law of total probability tells us that we can compute the probability of B by
adding up the joint probability of B with each of the Ei which is P(B∩Ei).

P(B)=P(B∩E1) +P(B∩E2) +P(B∩E3)

Conditional probabilities, independence, and Bayes’ Rule

CONDITIONAL PROBABILITY is one of the most important concepts of

probability theory.

A conditional probability expresses the probability that some event A will occur,
given that event B occurred. The conditional probability of A given B, written
P(A|B), where the | is pronounced “given”, is defined as

Exercise:

Suppose I have a group containing the students from various countries. The first
3 are male, and the last 4 female:

Let’s define A = “the student is female” and B = “the student is from the UK”.
What is P(A∪B), that is, the probability that the student is female or from the UK?
P(A∪B) = P(A) +P(B)−P(A∩B)

P(A∪B) = 4/7+4/7−3/7 = 5/7

Let us assume E1 = “student is British”, E2 = “student is Chinese”, and E3 =

“student is German”. Also let B be the event that the student is female. Apply the
law of total probability to compute P(B)

Let us assume that,

events A = “the student is male” and B = “the student is from the UK”.

What is P(A|B)?

what is P(B|A)?

The product rule and chain rule

If we rearrange the terms in the definition of conditional probability, we obtain the

PRODUCT RULE, which allows us to compute joint probabilities from conditional
probabilities:

P(A∩B) = P(A|B) P(B).

Since intersection is commutative, we could also write P(A∩B) = P(B∩A) = P(B|A)

P(A).

Example:
Suppose I have a jar of marbles, and I choose one uniformly at random. Each
marble is either red (R), green (G), or blue (B). In two-thirds of the marbles, these
colors are solid (S), while in the remainder the colors are pattern. y. If I tell you
that the probability of getting a red marble given that I have a solid-colored one is
1/2, can you compute the probability that my chosen marble is solid red?

Solution: P(R∩S) = P(R|S) P(S).

We were told that P(R|S) = 1/2, and P(S) = 2/3, so P(R∩S) = (1/2) (2/3) = 1/3.

Now suppose the probability of the marble being blue given that it’s pattern is 1/5.
What is the probability of getting a pattern blue marble?

It’s given that all marbles are either solid or pattern, and that P(S), the probability
of a solid-color marble, is 2/3. So we can compute the probability of a pattern
marble as P(Sl) = 1−P(S) = 1/3.

Then P(B∩Sl) = P(B|Sl) P(Sl) = (1/5)(1/3) = 1/15.

We can use the product rule to replace the joint probabilities in

with conditional probabilities, thereby obtaining

P(B)=P(B|E1)P(E1)+P(B|E2)P(E2)+P(B|E3)P(E3)

Independent events

Two events A and B are INDEPENDENT EVENTS iff

P(A|B) = P(A).

By substituting in the definition of conditional probability, we can equivalently

state that events A and B are independent iff P(A∩B) = P(A)P(B).

Bayes’ Rule

Bayes’ Rule is derived by replacing the P(A ∩ B) in the definition of conditional

probability with P(B|A) P(A) (using the product rule). We obtain: P(A|B) = P(B|A)
P(A)/ P(B)
Example .

Suppose I have some playing cards in my hand from a standard deck, and it turns
out that 1/4 of them are red cards (hearts or diamonds) and 3/8 are face cards
(jack, queen, or king). Also, 1/2 of the red cards are face cards. If I choose one of
the cards in my hand uniformly at random, and get a face card, what’s the
probability that it is also a red card?

Solution: Let R be the event that the card is red, and F be the event that it is a
face card. We are told that P(R) = 1/4, P(F) = 3/8, and P(F |R) = 1/2.

So, P(R|F) = P(F |R)P(R) /P(F)

= (1/2)(1/4) /(3/8)

= 1/3