Lecture 1

ECO 523
Probability
Some history1
▶ The subject of statistics and probability seem to be of ancient origin in
India and find a mention even in the great Indian epic, the
Mahabharata,.” Eg: the following quote of Ian Hacking from the History
and Philosophy Science Seminar: “King Bhangasuri wanting to flaunt his
skill in numbers of leaves, and the number of fruits, on two great
branches of a spreading tree. There are, he avers, 2095 fruits. Nala
counts all night and is duly amazed by morning. Bhangasuri says ”I, of
dice possess the science, and in numbers thus am skilled.”
▶ The treatise Arthasastra by Kautilya during the Mauryan period had a
detailed description of the system of data collection relating to
agricultural , population and economic censuses in villages and towns
during the period.
▶ The tradition of collecting data in detail continued during the period of
Mughal emperor Akbar around 1590 A.D. The Ain-i-Akbari written by
Abul Fazal during 1596-1597 A.D. has the best compilation of that
period containing a wealth of information. This had details of several
government departments including the system of legalized measurements,
land classification and crop yields and other information. Abul Fazal was
“ regarded as a statistician ” (Jarret (1894)).
1
This discussion on the history of statistics is taken from Ghosh et. al
(1999) and Rao (2006).
Some history (contd.)

▶ In British India collection of data became more systematic.

Censuses became more regular. The first systematic census
was carried out in 1872 and decadal censuses started from
1881 and have been continuing since then.
▶ Post independence: Mahalanobis, Indian Statistical Institute
(ISI), CSO, NSSO
▶ Tagore even wrote a poem for one of issues of Sankhya, the
Indian Journal of Statistics.
Probability

▶ Mathematics is the logic of certainty; probability is the logic

of uncertainty. Probability is extremely useful in a wide variety
of fields, since it provides tools for understanding and
explaining variation, separating signal from noise, and
modeling complex phenomena.
1. Statistics: Probability is the foundation and language for
statistics, enabling many powerful methods for using data to
learn about the world.
2. Physics
3. Biology: Genetics is deeply intertwined with probability, both
in the inheritance of genes and in modeling random mutations.
4. Computer science
5. Finance
6. Medicine: Randomized clinical trials are used in which
patients are randomly assigned to receive treatment or placebo.
Basics of set theory

▶ Experiment: any action or process whose outcome is subject

to uncertainty.
▶ Sample Space: collection of all possible outcomes (or
elements) of the experiment (set S).
▶ Event: collection of elements (subset A, B, etc.) contained in
the sample space (S).
▶ s ∈ S: The outcome s belongs to the sample space S. The
contrary is defined by the symbol ∈.
/
▶ ∅ = {}: Denotes the empty set (the set of no elements). It
also defines the set of elements of an impossible event; e.g.:
the event ‘generating a negative number’ when rolling a die.
Basics of set theory(contd.)

▶ Union: The union of event A and event B, denoted A ∪ B, is the

collection (or set) of elements that belong to either A or B or both.
[A ∪ B = {x : x ∈ A or x ∈ B}]
. Remember A ∪ B is “at least one of A and B occur” Properties:
A ∪ A = A ; A ∪ S = S ; A ∪ ∅ = A.
▶ Intersection: The intersection of event A and event B, denoted
A ∩ B, is the collection (or set) of elements that belong to both A
and B. [A ∩ B = {x : x ∈ A and x ∈ B}]
Properties:A ∩ A = A; A ∩ S = A; A ∩ ∅ = ∅.
▶ Complement: The complement of event A, denoted Ac (or
equivalently A′ ), is the set of all elements that are not in A.
[Ac = {x : x ∈
/ A}]
Properties: ((Ac )c = A; ∅c = S; S c = ∅; Ac ∪ A = S; Ac ∩ A = ∅
Figure 1
Basics of set theory(contd.)
▶ A ⊆ B: The event A is contained in event B, A ⊆ B, if every element A
also belongs to B.
Properties: If A ⊆ B and B ⊆ A =⇒ A = B;
If A ⊆ B and B ⊆ C =⇒ A ⊆ C ;
∅ ⊆ A, A ⊆ A
▶ Disjoint: Event A and event B are disjoint, or mutually exclusive, if A and
B have no outcome in common. [A ∩ B = ∅ ⇐⇒ A and B are disjoint
events]
▶ Exhaustive: Event A and event B are exhaustive if their union is S.
[A ∪ B = S ⇐⇒ A and B are exhaustive events]
▶ De Morgan’s Law

(A ∪ B)C = AC ∩ B C and(A ∩ B)C = AC ∪ B C

“It is not the case that at least one of A and B occur” is the same as
saying that “A does not occur and B does not occur”, and saying that “it
is not the case that both occur” is the same as saying that “at least one
does not occur”. Analogous results hold for unions and intersections of
more than two events.
Sample Space

Example 1.
Coin flips: A coin is flipped 10 times.Writing Heads as H and
Tails as T, the sample space is the set of all possible strings of
length 10 of H’s and T’s. We can (and will) encode H as 1 and T
as 0, so that an outcome is a sequence (s1 , ..., s10 ) with sj ∈ {0, 1},
and the sample space is the set of all such sequences.
Sample Space (contd.)
1. Let A1 be the event that the first flip is Heads. As a set,
A1 = {(1, s2 , ..., s10 ) : sj ∈ {0, 1} for 2 ≤ j ≤ 10}.
This is a subset of the sample space, so it is indeed an event.
Similarly, let Aj be the event that the j th flip is Heads for j = 2, 3, .
. . , 10.
2. Let B be the event that at least one flip was Heads.
10
[
B= Aj
j=1

3. Let C be the event that all the flips were Heads.

10
\
C= Aj
j=1

4. Let D be the event that there were at least two consecutive Heads.
[9
D = (Aj ∩ Aj+1 )
j=1
Naive definition of probability
▶ Naive definition of probability: Let A be an event for an
experiment with a finite sample space S. The naive probability
of A is
|A| number of outcomes favorable to A
Pnaive (A) = =
|S| total number of outcomes in S
▶ For example, in figure 1 it says
5 4
Pnaive (A) = , Pnaive (B) = ,
9 9
8 1
Pnaive (A ∪ B) = , Pnaive (A ∩ B) =
9 9
▶ For the complements of the events just considered,
4 5
Pnaive (AC ) = , Pnaive (B C ) = ,
9 9
1 8
Pnaive ((A ∪ B)C ) = , Pnaive ((A ∩ B)C ) =
9 9
▶ The naive definition is very restrictive in that it requires S to
be finite, with equal mass for each pebble.
Multiplication Rule

Theorem 1.
Multiplication rule: Consider a compound experiment consisting
of two sub-experiments, Experiment A and Experiment B. Suppose
that Experiment A has a possible outcomes, and for each of those
outcomes Experiment B has b possible outcomes. Then the
compound experiment has ab possible outcomes.
▶ Imagine a tree diagram. Let the tree branch a ways according
to the possibilities for Experiment A, and for each of those
branches create b further branches for Experiment B.
▶ It is often easier to think about the experiments as being in
chronological order, but there is no requirement in the
multiplication rule that Experiment A has to be performed
before Experiment B.
Multiplication Rule (contd.)
▶ Suppose you are buying an ice cream cone. You can choose whether
to have a cake cone or a waffle cone, and whether to have
chocolate, vanilla, or strawberry as your flavor.

Figure 2
Example 2.
Subsets: A set with n elements has 2n subsets, including the empty set ;
and the set itself. This follows from the multiplication rule since for each
element, we can choose whether to include it or exclude it. For example,
the set {1, 2, 3} has the 8 subsets ϕ, {1}, {2}, {3}, {1, 2}, {1, 3}, {2,
3}, {1, 2, 3}.
Multiplication Rule (contd.)

Theorem 2.
Sampling with replacement:Consider n objects and making k choices from
them, one at a time with replacement (i.e., choosing a certain object does not
preclude it from being chosen again). Then there are nk possible outcomes
▶ For example, imagine a jar with n balls, labeled from 1 to n. Each
sampled ball is a sub-experiment with n possible outcomes, and there are
k sub-experiments. Thus there are nk ways to choose a sample of size k.
Theorem 3.
▶ Sampling without replacement: Consider n objects and making k choices
from them, one at a time without replacement (i.e., choosing a certain object
precludes it from being chosen again).Then there are n(n-1)...(n-k+1) possible
outcomes, for k ≤ n (and 0 possibilities for k > n).
▶ The result also follows from multiplication rule: each sampled ball is
again a sub-experiment, and the number of possible outcomes decreases
by 1 each time.
Multiplication Rule (contd.)

Example 3.
Permutations and factorials:A permutation of 1, 2, . . . ,n is
an arrangement of them in some order, e.g., 3, 5, 1, 2, 4 is a
permutation of 1, 2, 3, 4, 5. By Theorem 3 with k = n, there are
n! permutations of 1, 2, . . . ,n. For example there are n! factorial
ways in which n people can line up for ice cream. (recall that
n! = n(n − 1)....1 for any positive integer n and 0! = 1.)
▶ Birthday problem: There are k people in a room. Assume
each person’s birthday is equally likely to be any of the 365
days of the year (we exclude February 29), and that people’s
birthdays are independent (we assume there are no twins in
the room). What is the probability that two or more people in
the group have the same birthday?
▶ Solved in class
Adjusting for overcounting
In many counting problems, it is not easy to directly count each
possibility once and only once. If, however, we are able to count
each possibility exactly c times for some c, then we can adjust by
dividing by c. For example, if we have exactly double-counted each
possibility, we can divide by 2 to get the correct count. We call
this adjusting for overcounting.
▶ Committees and teams: Consider a group of four people.
1. How many ways are there to choose a two-person committee?
2. How many ways are there to break the people into two teams
of two?
▶ Solved in class
Adjusting for overcounting (contd.)
A binomial coefficient counts the number of subsets of a certain size for
a set, such as the number of ways to choose a committee of size k from
a set of n people. Sets and subsets are by definition unordered, e.g.,
{3, 1, 4} = {4, 1, 3}, so we are counting the number of ways to choose k
objects out of n, without replacement and without distinguishing between
the different orders in which they could be chosen.
▶ Binomial coefficient: For any nonnegative integers k and n, the
binomial coefficient n Ck , read as “n choose k”, is the number of
subsets of size k for a set of size n.
Theorem 4.
Binomial coefficient formula: For k ≤ n, we have

n n(n − 1)...(n − k + 1) n!
Ck = =
k! (n − k)!k!

For k > n, we have n Ck = 0

▶ Proof done in class
Adjusting for overcounting (contd.)
Permutations of a word: How many ways are there to permute
the letters in the word LALALAAA? STATISTICS?
Solved in class
Adjusting for overcounting (contd.)
Theorem 5.
▶ Binomial theorem: The binomial theorem states that
n
X
(x + y )n = n
Ck x k y n−k
k=0

▶ To prove the binomial theorem, expand out the product

(x + y )(x + y ).....(x + y )
This has n factors.
▶ Just as (a+b)(c+d) = ac+ad+bc+bd is the sum of terms
where we pick the a or the b from the first factor (but not
both) and the c or the d from the second factor (but not
both), the terms of (x + y )n are obtained by picking either
the x or the y (but not both) from each factor. There are n Ck
ways to choose exactly k of the x’s, and each such choice
yields the term x k y n−k .
Non-naive definition of probability

We have now seen several methods for counting outcomes in a

sample space, allowing us to calculate probabilities if the naive
definition applies. But the naive definition can only take us so far,
since it requires equally likely outcomes and can’t handle an
infinite sample space.
Non-naive definition of probability (contd.)

▶ General definition of probability: A probability space

consists of a sample space S and a probability function P
which takes an event A ⊆ S as input and returns P(A), a real
number between 0 and 1, as output. The function P must
satisfy the following axioms:
1. P(ϕ) = 0, P(S) = 1
2. If A1 , A2 , . . . are disjoint events, then
∞
[ ∞
X
P( Aj ) = P(Aj )
j=1 j=1

(Saying that these events are disjoint means that they are
mutually exclusive: Ai ∩ Aj = ϕ for i ̸= j.)
▶ Any function P (mapping events to numbers in the interval [0,
1]) that satisfies the two axioms is considered a valid
probability function.
Non-naive definition of probability (contd.)

▶ Properties of probability: Probability has the following

properties, for any events A and B.
1. P(AC ) = 1 - P(A)
2. If A ⊆ B, then P(A) ≤ P(B)
3. P(A ∪ B) = P(A) + P(B) - P (A ∩ B)
Proofs for the first two properties done in class. Refer to the
book for the third one.
Non-naive definition of probability (contd.)

Theorem 6.
Inclusion-exclusion: For any events A1 , ..., An ,
n
[ X X X
P( Ai ) = P(Ai ) − P(Ai ∩ Aj ) + P(Ai ∩ Aj ∩ Ak ) − .....
i=1 i i<j i<j<k

+(−1)n+1 P(A1 ∩ ...... ∩ An )

References

The material for this lecture is mainly from Chapter 1 of the

Blitzstein and Hwang book.