Probability Theory

 Jointly with statistics, probability theory is a branch of mathematics that has been
developed to deal with uncertainty.
 Spurred initially by the mathematician’s desire to analyze gambling games and later by
the scientific analysis of mortality tables within the medical profession, the theory of
probability has been developed as a scientific tool dealing with chance.
 It provides the basis for the science of statistical inference through experimentation and
data analysis

Population:
A population consists of the totality of the observations with which we are concerned. Size of the
population is called the total number of population. For example, total number of student in this
class is said to be a population size.

Why population is necessary?

For statistical inference, it is impossible to observe the entire set of observations that make up the
population. As a result, we have to depend on a subset of a population in order to make inference
about the population. In this case, sampling is necessary. Every population follows a specific
distribution and sampling helps us to infer about the population. Distribution helps us to
overcome the test for entire observations.

Sample:
A sample is a subset of a population. Sample is necessary to infer about the population. Two
types of procedures are random sampling and biased sampling.
1. Random sampling: In this procedure, the observations are made independently and at
random. Here the probability for occurring and event is equal in each observation.

Example:

2. Biased sampling: When the observations are not made independently and not at random,
it produces inferences that are overestimate or underestimate some characteristics of the
population; this type of sampling is called biased sampling.

Example:

Sample space:
 An experiment can in general be thought of as any process or procedure for which more
than one outcome is possible.
 The goal of probability theory is to provide a mathematical structure for understanding or
explaining the chances or likelihoods of the various outcomes actually occurring.
  A first step in the development of this theory is the construction of a list of the possible
experimental outcomes. This collection of outcomes is called the sample space or state
space and is denoted by S
 The sample space S of an experiment is a set consisting of all of the possible
experimental outcomes.

Example: The sample space for tossing a coin is given by

𝑠 = {𝐻, 𝑇} Where, H = Heads, T = Tails

Each outcome in a sample space is called an element or a sample point. The outcomes can be
represented in two ways:

i. Numerical values
ii. Descriptive

Tossing a die, 𝑆 = {1,2,3,4,5,6} it is numerical value

𝑆 = {𝑒𝑣𝑒𝑛, 𝑜𝑑𝑑 } it is descriptive

Example:
An engineer in charge of the maintenance of a particular machine notices that its breakdowns can
be characterized as due to an electrical failure within the machine, a mechanical failure of some
component of the machine, or operator misuse. When the machine is running, the engineer is
uncertain what will be the cause of the next breakdown. The problem can be thought of as an
experiment with the sample space
S = {electrical, mechanical, misuse}

Example:
A company sells computer chips in boxes of 500, and each chip can be classified as either
satisfactory or defective. The number of defective chips in a particular box is uncertain, and the
sample space is
S = {0 defectives, 1 defective, 2 defectives, 3 defectives, 4 defectives,..., 499 defectives, 500
defectives}

Example:
Games of chance commonly involve the toss of a coin, the roll of a die, or the use of a pack of
cards. The toss of a single coin has a sample space
S = {head, tail}

The toss of two coins (or one coin twice) has a sample space

S = {(head, head) (head, tail) (tail, head) (tail, tail)}

Where (head, tail), say, represents the event that the first coin resulted in a head and the second
coin resulted in a tail. Notice that (head, tail) and (tail, head) are two distinct outcomes since
observing a head on the first coin and a tail on the second coin is different from observing a tail
on the first coin and a head on the second coin.

Example:
A usual six-sided die has a sample space
S = {1, 2, 3, 4, 5, 6}

It is often helpful to portray a sample space as a diagram.

Sample space for rolling two dice

Sample space for choosing one card

Probability of an event:
The probability of an event A is the sum of the weights of all sample points in A. Therefore,
0 ≤ p1 ≤ 1, 0 ≤ p2 ≤ 1, . . . , 0 ≤ pn ≤ 1

0 ≤ p(A) ≤ 1, P(∅) = 0, P(S)=1

p1 + p2 + · · · + pn = 1

Example:
Suppose that the machine breakdowns occur with probability values of P(electrical) = 0.2,
P(mechanical) = 0.5, and P(misuse) = 0.3. This is a valid probability assignment since the three
probability values 0.2, 0.5, and 0.3 are all between zero and one and they sum to one.
Events:
 Interest is often centered not so much on the individual elements of a sample space, but
rather on collections of individual outcomes. These collections of outcomes are called
events.
 An event A is a subset of the sample space S. It collects outcomes of particular interest.
The probability of an event A, P(A), is obtained by summing the probabilities of the
outcomes contained within the event A.

P(A) = 0.10 + 0.15 + 0.30 = 0.55

Complement:
The event A′, the complement of an event A, is the event consisting of everything in the sample
space S that is not contained within the event A. In this example, the probability of the
complement of A is obtained by summing the probabilities of the five outcomes not contained
within A, so that
P(A′ ) = 0.10 + 0.05 + 0.05 + 0.15 + 0.10 = 0.45

Notice that P(A) + P(A′ ) = 1, which is a general rule.

Examples of Events:
Consider the following probability values for the number of defective chips in a box of 500
chips: P(0 defectives) = 0.02, P(1 defective) = 0.11, P(2 defectives) = 0.16, P(3 defectives) =
0.21, P(4 defectives) = 0.13, P(5 defectives) = 0.08. Suppose that the probabilities of the
additional elements of the sample space (6 defectives, 7 defectives, ... , 500 defectives) are
unknown. The company is thinking of claiming that each box has no more than 5 defective chips,
and it wishes to calculate the probability that the claim is correct.
The event correct consists of the six outcomes listed above, so that
correct = {0 defectives, 1 defective, 2 defectives, 3 defectives, 4 defectives, 5 defectives} ⊂ S

The probability of the claim being correct is then

P(correct) = P(0 defectives) +···+ P(5 defectives)

= 0.02 + 0.11 + 0.16 + 0.21 + 0.13 + 0.08 = 0.71

Consequently, on average, only about 71% of the boxes will meet the company’s claim that there
are no more than 5 defective chips. The complement of the event correct is that there will be at
least 6 defective chips so that the company’s claim will be incorrect. This has a probability of 1 −
0.71 = 0.29.

Example:
The event that an even score is recorded on the roll of a die is given by
even = {2, 4, 6}
For a fair die this event would have a probability of

P(even) = P(2) + P(4) + P(6) = 1/ 6 + 1 /6 + 1/ 6 = ½

Combinations of Events:
 In general, more than one event will be of interest for a particular experiment and sample
space.
 For two events A and B, in addition to the consideration of the probability of event A
occurring and the probability of event B occurring, it is often important to consider other
probabilities
1. the probability of both events occurring simultaneously
2. the probability that neither event A nor event B occurs
3. the probability that at least one of the two events occurs
4. the probability that event A occurs, but event B does not

Intersections of Events:
The event A ∩ B is the intersection of the events A and B and consists of the outcomes that are
contained within both events A and B. The probability of this event, P(A ∩ B), is the probability
that both events A and B occur simultaneously.

Events A and B The event A ∩ B

  Event A consists of three outcomes, and its probability is given by

P(A) = 0.01 + 0.07 + 0.19 = 0.27

 Event B consists of five outcomes, and its probability is given by

P(B) = 0.07 + 0.19 + 0.04 + 0.14 + 0.12 = 0.56

 The intersection of these two events consists of the two outcomes that are contained
within both events A and B. It has a probability of P(A ∩ B) = 0.07 + 0.19 = 0.26

This is the probability that both events A and B occur simultaneously.

A ∩ A′ = ∅, where ∅ is referred to as the “empty set,” a set that does not contain anything.

P(A ∩ A′ ) = P(∅) = 0

P(A′ ∩ B) = 0.04 + 0.14 + 0.12 = 0.30 P(A ∩ B′ ) = 0.01

Mutually Exclusive Events:

Two events A and B that have no outcomes in common are said to be mutually exclusive events.
In this case A ∩ B = ∅ and P(A ∩ B) = 0.

A and B are mutually exclusive events A⊂B

Some other simple results concerning the intersections of events are as follows:

Unions of Events:
 The event A ∪ B is the union of events A and B and consists of the outcomes that are
contained within at least one of the events A and B. The probability of this event, P(A ∪
B), is the probability that at least one of the events A and B occurs.
 Notice that the outcomes in the event A ∪ B can be classified into three kinds. They are
1. in event A, but not in event B
2. in event B, but not in event A
3. in both events A and B

The event A ∪ B
P(A ∪ B) = P(A) + P(B) − P(A ∩ B)

If the events A and B are mutually exclusive so that P(A ∩ B) = 0, then

P(A ∪ B) = P(A) + P(B)

Some other simple results concerning the unions of events are as follows:
Union of Three Events:
The probability of the union of three events A, B, and C is the sum of the probability values of
the simple outcomes that are contained within at least one of the three events. It can also be
calculated from the expression
P(A ∪ B ∪ C) = [P(A) + P(B) + P(C)] – [P(A ∩ B) + P(A ∩ C) + P(B ∩ C)] + P(A ∩ B ∩ C)

Conditional Probability:
The probability of event A occurring when it is known that event B has occurred is called a
conditional probability and is denoted by P(A|B). It measures the probability that event A occurs
when it is known that event B occurs.
𝑃(𝐴 ∩ 𝐵)
𝑃 (𝐴 |B ) = , Provided that P(B)>0
𝑃(𝐵)

Example:
If a fair die is rolled the probability of scoring a 6 is P(6) = 1/6. If somebody rolls a die without
showing you but announces that the result is even, then intuitively the chance that a 6 has been
obtained is 1/3 since there are three equally likely even scores, one of which is a 6.
Mathematically, this conditional probability is calculated to be

Example:
The probability that a regularly scheduled flight departs on time is P(D) = 0.83; the probability
that it arrives on time is P(A) = 0.82; and the probability that it departs and arrives on time is
P(D ∩ A) = 0.78. Find the probability that a plane
a) Arrives on time, given that it departed on time
b) Departed on time, given that it has arrived on time.

The probability that a plane arrives on time, given that it departed on time, is

The probability that a plane departed on time, given that it has arrived on time, is
 Probabilities of Event Intersections:

General Multiplication Law

It follows from the definition of the conditional probability P(A|B) that the probability of the
intersection of two events A ∩ B can be calculated as

P(A ∩ B) = P(B) P(A|B)

P(A ∩ B) = P(A) P(B|A)

 so that the probability of events A and B both occurring can also be obtained by
multiplying the probability of event A by the probability of event B conditional on event
A.
 The probability of the intersection of a series of events A1, . . . , An can be calculated
from the expression

P(A1 ∩ · · · ∩ An) = P(A1) × P(A2|A1) × P(A3|A1 ∩ A2) × · · · × P(An|A1 ∩ · · · ∩ An-1)

Example:
Suppose that we have a fuse box containing 20 fuses, of which 5 are defective. If 2 fuses are
selected at random and removed from the box in succession without replacing the first, what is
the probability that both fuses are defective?
Solution:

We shall let A be the event that the first fuse is defective and B the event that the second fuse is
defective; then we interpret A ∩ B as the event that A occurs and then B occurs after A has
occurred. The probability of first removing a defective fuse is 1/4; then the probability of
removing a second defective fuse from the remaining 4 is 4/19. Hence,

P(A ∩ B) = (1/ 4) *(4/19) = 1/19

Independent Events:
Two events A and B are said to be independent events if
P(A|B) = P(A), P(B|A) = P(B), and P(A ∩ B) = P(A) P(B)

Any one of these three conditions implies the other two. The interpretation of two events being
independent is that knowledge about one event does not affect the probability of the other event.
Intersections of Independent Events:
The probability of the intersection of a series of independent events A1, . . . , An is simply given
by
P(A1 ∩ · · · ∩ An) = P(A1) P(A2) · · · P(An)

Example:
A small town has one fire engine and one ambulance available for emergencies. The probability
that the fire engine is available when needed is 0.98, and the probability that the ambulance is
available when called is 0.92. In the event of an injury resulting from a burning building, find the
probability that both the ambulance and the fire engine will be available, assuming they operate
independently.
Solution:

Let A and B represent the respective events that the fire engine and the ambulance are available.
Then P(A ∩ B) = P(A)P(B) = (0.98)(0.92) = 0.9016

Example:
An electrical system consists of four components as illustrated in Figure below. The system
works if components A and B work and either of the components C or D works. The reliability
(probability of working) of each component is also shown in Figure. Find the probability that
a) The entire system works
b) The component C does not work, given that the entire system works.

Assume that the four components work independently.

Solution:

In this configuration of the system, A, B, and the subsystem C and D constitute a serial circuit
system, whereas the subsystem C and D itself is a parallel circuit system.

a) Clearly the probability that the entire system works can be calculated as follows:
P[A ∩ B ∩ (C ∪ D)] = P(A)P(B)P(C ∪ D) = P(A)P(B)[1 - P(C ∩ D)]
= P(A)P(B)[1 - P(C)P(D)]
= (0.9)(0.9)[1 - (1 - 0.8)(1 - 0.8)] = 0.7776
b) To calculate the conditional probability in this case, notice that
Law of Total Probability:

If A1, . . . , An is a partition of a sample space, then the probability of an event B can be obtained
from the probabilities P(Ai) and P(B|Ai) using the formula

P(B) = P(A1) P(B|A1) + · · · + P(An) P(B|An)

Proof:

A partition A1, . . . , An and an event B

Let A1, . . . , An be a partition of a sample space S so that the events Ai are mutually exclusive
with S = A1 ∪ · · · ∪ An

Suppose that the probabilities of these n events, P(A1), . . . , P(An), are known. In addition,
consider an event B as shown and suppose that the conditional probabilities P(B|A1), . . . ,
P(B|An) are also known.

An initial question of interest is how to use the probabilities P(Ai) and P(B|Ai) to calculate P(B),
the probability of the event B. In fact, this is easily achieved by noting that

B = (A1 ∩ B) ∪ · · · ∪ (An ∩ B)

Where the events Ai ∩ B are mutually exclusive, so that

P(B) = P(A1 ∩ B) + · · · + P(An ∩ B)

= P(A1) P(B|A1) + · · · + P(An) P(B|An)

This result is known as the law of total probability.

  If it is known that one and only one of a series of events Ai can occur, then the probability
of another event B can be obtained as the weighted average of the conditional
probabilities P(B|Ai), with weights equal to the probabilities P(Ai).

Example:
In a certain assembly plant, three machines, B1, B2, and B3, make 30%, 45%, and 25%,
respectively, of the products. It is known from past experience that 2%, 3%, and 2% of the
products made by each machine, respectively, are defective. Now, suppose that a finished
product is randomly selected. What is the probability that it is defective?

Solution:

Consider the following events:

A: the product is defective

B1: the product is made by machine B1
B2: the product is made by machine B2
B3: the product is made by machine B3

Applying the rule of elimination, we can write

P(A) = P(B1)P(A|B1) + P(B2)P(A|B2) + P(B3)P(A|B3)

The three branches give the probabilities

P(B1)P(A|B1) = (0.3)(0.02) = 0.006

P(B2)P(A|B2) = (0.45)(0.03) = 0.0135

P(B3)P(A|B3) = (0.25)(0.02) = 0.005

P(A) = 0.006 + 0.0135 + 0.005 = 0.0245.

Bayes’ Theorem or Rule:

If A1, . . . , An is a partition of a sample space such that P(Ai) ≠0, then the posterior
probabilities of the events Ai conditional on an event B can be obtained from the probabilities
P(Ai) and P(B|Ai) using the formula

Proof: The probabilities P(A1), . . . , P(An) can be thought of as being the prior probabilities of
the events A1, . . . , An. However, the observation of the event B provides some additional
information that allows the revision of these prior probabilities into a set of posterior
probabilities

P(A1|B), . . . , P(An|B)

Which are the probabilities of the events A1, . . . , An conditional on the event B.

From the law of total probability, these posterior probabilities are calculated to be

P(B) = P(A1 ∩ B) + · · · + P(An ∩ B)

= P(A1) P(B|A1) + · · · + P(An) P(B|An)

Example:
For previous problem, if a product was chosen randomly and found to be defective, what is the
probability that it was made by machine B3?
Using Bayes’ rule to write

Example:
A manufacturing firm employs three analytical plans for the design and development of a
particular product. For cost reasons, all three are used at varying times. In fact, plans 1, 2, and 3
are used for 30%, 20%, and 50% of the products, respectively. The defect rate is different for the
three procedures as follows:
P(D|P1) = 0.01, P(D|P2) = 0.03, P(D|P3) = 0.02,

Where P(D|Pj) is the probability of a defective product, given plan j. If a random product was
observed and found to be defective, which plan was most likely used and thus responsible?

Solution:

From the statement of the problem

P(P1) = 0.30, P(P2) = 0.20, and P(P3) = 0.50

We must find P(Pj|D) for j = 1, 2, 3. Bayes’ rule shows

Probability Trees:
A company sells a certain type of car that it assembles in one of four possible locations. Plant I
supplies 20% of the cars; plant II, 24%; plant III, 25%; and plant IV, 31%. A customer buying a
car does not know where the car has been assembled, and so the probabilities of a purchased car
being from each of the four plants can be thought of as being 0.20, 0.24, 0.25, and 0.31. Each
new car sold carries a one-year bumper-to-bumper warranty. The company has collected data
that show:
P(claim|plant I) = 0.05, P(claim|plant II) = 0.11, P(claim|plant III) = 0.03, P(claim|plant IV) =
0.08

Find s the probability that a claim on the warranty of the car.

Probability tree for car warranties

P(claim) = P(plant I, claim) + P(plant II, claim) + P(plant III, claim) + P(plant IV, claim)

= (0.20 × 0.05) + (0.24 × 0.11) + (0.25 × 0.03) + (0.31 × 0.08)

= 0.0687
  In other words, about 6.87% of the cars purchased will have a claim on their warranty.
Notice that this overall claim rate is a weighted average of the four individual plant claim
rates, with weights corresponding to the supply proportions of the four plants.

Counting Techniques:
Remember that if a sample space S consists of N equally likely outcomes, of which n are
contained within the event A, then the probability of the event A is
𝑛
𝑝 (𝐴) =
𝑁
Multiplication Rule:
 The fundamental principle of counting, often referred to as the multiplication rule.
 If an experiment consists of k components for which the number of possible outcomes are
n1... nk, then the total number of experimental outcomes (the size of the sample space) is
equal to

n1 × n2 ×···× nk

 If an operation can be performed in n1 ways, and if for each of these a second operation
can be performed in n2 ways, and for each of the first two a third operation can be
performed in n3 ways, and so forth, then the sequence of k operations can be performed
in

n1 n2 ··· nk ways.

Example:

How many sample points are there in the sample space when a pair of dice is thrown once?

Solution:

The first die can land face-up in any one of n1 = 6 ways. For each of these 6 ways, the second die
can also land face-up in n2 = 6 ways. Therefore, the pair of dice can land in n1 n2 = (6) × (6) = 36
possible ways.

Example:

A developer of a new subdivision offers prospective home buyers a choice of Tudor, rustic,
colonial, and traditional exterior styling in ranch, two-story, and split-level floor plans. In how
many different ways can a buyer order one of these homes?
Solution: Since n1 = 4 and n2 = 3, a buyer must choose from n1n2 = (4)(3) = 12 possible homes.

Example:

If a 22-member club needs to elect a chair and a treasurer, how many different ways can these
two to be elected?

Solution: For the chair position, there are 22 total possibilities. For each of those 22 possibilities,
there are 21 possibilities to elect the treasurer. Using the multiplication rule, we obtain n1 × n2 =
22 × 21 = 462 different ways.

Example:

Sam is going to assemble a computer by himself. He has the choice of chips from two brands, a
hard drive from four, memory from three, and an accessory bundle from five local stores. How
many different ways can Sam order the parts?

Solution: Since n1 = 2, n2 = 4, n3 = 3, and n4 = 5, there are

N1 × n2 × n3 × n4 = 2 × 4 × 3 × 5 = 120

Permutations:
A permutation of k objects from n objects (n ≥ k) is an ordered sequence of k objects selected
without replacement from the group of n objects. The number of possible permutations of k
objects from n objects is

Notice that if k = n, the number of permutations is

This is just the number of ways of ordering n objects.

Example:

A food company has four different recipes for a potential new product and wishes to compare
them through consumer taste tests. In these tests, a participant is given the four types of food to
taste in a random order and is asked to rank various aspects of their taste. This ranking procedure
simply provides an ordering of the four products, and the number of possible ways in which it
can be performed is

𝑃44 = 4! = 4 × 3 × 2 × 1 = 24

In a different taste test, each participant samples eight products and is asked to pick the best, the
second best, and the third best. The number of possible answers to the test is then

𝑃38 = 8 × 7 × 6 = 336

Combinations:
 Sometimes when k objects are chosen from a group of n objects, the ordering of the
drawing of the k objects is not of importance.
 In other words, interest is focused on which k objects are chosen, but not on the order in
which they are chosen.
 Such a collection of objects is called a combination of k objects from n objects.
 A combination of k objects from n objects (n ≥ k) is an unordered collection of k objects
selected without replacement from the group of n objects. The number of possible
combinations of k objects from n objects is

Example:

Suppose that in the taste test, each participant samples eight products and is asked to select the
three best products, but not in any particular order. The number of possible answers to the test is
then