A Short Introduction to

Probability

Prof. Dirk P. Kroese

School of Mathematics and Physics
The University of Queensland

c 2018 D.P. Kroese. These notes can be used for educational purposes, pro-
vided they are kept in their original form, including this title page.
0

Copyright c 2018 D.P. Kroese

Contents

1 Random Experiments and Probability Models 5

1.1 Random Experiments . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Sample Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.3 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4 Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
1.5 Counting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.6 Conditional probability and independence . . . . . . . . . . . . . 22
1.6.1 Product Rule . . . . . . . . . . . . . . . . . . . . . . . . . 24
1.6.2 Law of Total Probability and Bayes’ Rule . . . . . . . . . 26
1.6.3 Independence . . . . . . . . . . . . . . . . . . . . . . . . . 27

2 Random Variables and Probability Distributions 29

2.1 Random Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.2 Probability Distribution . . . . . . . . . . . . . . . . . . . . . . . 31
2.2.1 Discrete Distributions . . . . . . . . . . . . . . . . . . . . 33
2.2.2 Continuous Distributions . . . . . . . . . . . . . . . . . . 33
2.3 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.4 Transforms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
2.5 Some Important Discrete Distributions . . . . . . . . . . . . . . . 40
2.5.1 Bernoulli Distribution . . . . . . . . . . . . . . . . . . . . 40
2.5.2 Binomial Distribution . . . . . . . . . . . . . . . . . . . . 41
2.5.3 Geometric distribution . . . . . . . . . . . . . . . . . . . . 43
2.5.4 Poisson Distribution . . . . . . . . . . . . . . . . . . . . . 44
2.5.5 Hypergeometric Distribution . . . . . . . . . . . . . . . . 46
2.6 Some Important Continuous Distributions . . . . . . . . . . . . . 47
2.6.1 Uniform Distribution . . . . . . . . . . . . . . . . . . . . . 47
2.6.2 Exponential Distribution . . . . . . . . . . . . . . . . . . 48
2.6.3 Normal, or Gaussian, Distribution . . . . . . . . . . . . . 49
2.6.4 Gamma- and χ2 -distribution . . . . . . . . . . . . . . . . 52

3 Generating Random Variables on a Computer 55

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.2 Random Number Generation . . . . . . . . . . . . . . . . . . . . 55
3.3 The Inverse-Transform Method . . . . . . . . . . . . . . . . . . . 57
3.4 Generating From Commonly Used Distributions . . . . . . . . . . 59

Copyright c 2018 D.P. Kroese

2 Contents

4 Joint Distributions 65
4.1 Joint Distribution and Independence . . . . . . . . . . . . . . . . 65
4.1.1 Discrete Joint Distributions . . . . . . . . . . . . . . . . . 66
4.1.2 Continuous Joint Distributions . . . . . . . . . . . . . . . 69
4.2 Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
4.3 Conditional Distribution . . . . . . . . . . . . . . . . . . . . . . . 79

5 Functions of Random Variables and Limit Theorems 81

5.1 Jointly Normal Random Variables . . . . . . . . . . . . . . . . . 86
5.2 Limit Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

A Exercises and Solutions 93

A.1 Problem Set 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
A.2 Answer Set 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
A.3 Problem Set 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
A.4 Answer Set 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
A.5 Problem Set 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
A.6 Answer Set 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

B Sample Exams 109

B.1 Exam 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
B.2 Exam 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

C Summary of Formulas 113

D Statistical Tables 119

Copyright c 2018 D.P. Kroese

Preface

These notes form a short, but comprehensive, second-year introduction to prob-

ability. You might find these notes useful as a source of reference and an aid to
self-learning for courses such as STAT2202, STAT2003, and STAT2004. A more
polished version, with additional material, but without appendix A (Exercises
and Solutions) and appendix B (Sample Exams), forms Part I of the book

D.P. Kroese and J.C.C. Chan (2014). Statistical Modeling and Computation,
Springer, New York.

This can be downloaded (free) from https://people.smp.uq.edu.au/DirkKroese/

statbook_part1.pdf

To completely master the material, it is important that you

1. visit the lectures, where many extra examples will be provided;

2. do the tutorial exercises and the exercises in the appendix, which are
there to help you with the “technical” side of things; you will learn here
to apply the concepts learned at the lectures,

3. carry out random experiments on the computer. This will give you a
better intuition about how randomness works.

All of these will be essential if you wish to understand probability beyond “filling
in the formulas”.

Notation and Conventions

Throughout these notes I try to use a uniform notation in which, as a rule, the
number of symbols is kept to a minimum. For example, I prefer qij to q(i, j),
Xt to X(t), and EX to E[X].

The symbol “:=” denotes “is defined as”. We will also use the abbreviations
r.v. for random variable and i.i.d. (or iid) for independent and identically and
distributed.

Copyright c 2018 D.P. Kroese

4 Preface

I will use the sans serif font to denote probability distributions. For example
Bin denotes the binomial distribution, and Exp the exponential distribution.

Numbering

All references to Examples, Theorems, etc. are of the same form. For example,
Theorem 1.2 refers to the second theorem of Chapter 1. References to formula’s
appear between brackets. For example, (3.4) refers to formula 4 of Chapter 3.

Literature

• Hsu, H. (1997). Probability, Random Variables & Random Processes.

Shaum’s Outline Series, McGraw-Hill, New York.

• Ross, S. M. (2005). A First Course in Probability, 7th ed., Prentice-Hall,

Englewood Cliffs, NJ.

• Kroese, D.. P, Chan, J.C.C. (2014). Statistical Modeling and Computa-

tion, Springer, New York.

• Feller, W. (1970). An Introduction to Probability Theory and Its Applica-

tions, Volume I., 2nd ed., Wiley & Sons, New York.

Copyright c 2018 D.P. Kroese

Chapter 1

Random Experiments and

Probability Models

1.1 Random Experiments

The basic notion in probability is that of a random experiment: an experi-

ment whose outcome cannot be determined in advance, but is nevertheless still
subject to analysis.

Examples of random experiments are:

1. tossing a die,

2. measuring the amount of rainfall in Brisbane in January,

3. counting the number of calls arriving at a telephone exchange during a

fixed time period,

4. selecting a random sample of fifty people and observing the number of

left-handers,

5. choosing at random ten people and measuring their height.

Example 1.1 (Coin Tossing) The most fundamental stochastic experiment

is the experiment where a coin is tossed a number of times, say n times. Indeed,
much of probability theory can be based on this simple experiment, as we shall
see in subsequent chapters. To better understand how this experiment behaves,
we can carry it out on a digital computer, for example in Matlab. The following
simple Matlab program, simulates a sequence of 100 tosses with a fair coin(that
is, heads and tails are equally likely), and plots the results in a bar chart.

x = (rand(1,100) < 1/2)

bar(x)

Copyright c 2018 D.P. Kroese

6 Random Experiments and Probability Models

Here x is a vector with 1s and 0s, indicating Heads and Tails, say. Typical
outcomes for three such experiments are given in Figure 1.1.

10 20 30 40 50 60 70 80 90 100

10 20 30 40 50 60 70 80 90 100

10 20 30 40 50 60 70 80 90 100

Figure 1.1: Three experiments where a fair coin is tossed 100 times. The dark
bars indicate when “Heads” (=1) appears.

We can also plot the average number of “Heads” against the number of tosses.
In the same Matlab program, this is done in two extra lines of code:

y = cumsum(x)./[1:100]
plot(y)

The result of three such experiments is depicted in Figure 1.2. Notice that the
average number of Heads seems to converge to 1/2, but there is a lot of random
fluctuation.

Copyright c 2018 D.P. Kroese

1.1 Random Experiments 7

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
0 10 20 30 40 50 60 70 80 90 100

Figure 1.2: The average number of heads in n tosses, where n = 1, . . . , 100.

Example 1.2 (Control Chart) Control charts, see Figure 1.3, are frequently
used in manufacturing as a method for quality control. Each hour the average
output of the process is measured — for example, the average weight of 10
bags of sugar — to assess if the process is still “in control”, for example, if the
machine still puts on average the correct amount of sugar in the bags. When
the process > Upper Control Limit or < Lower Control Limit and an alarm is
raised that the process is out of control, e.g., the machine needs to be adjusted,
because it either puts too much or not enough sugar in the bags. The question
is how to set the control limits, since the random process naturally fluctuates
around its “centre” or “target” line.
0
1
0
1
0
µ + c1
0
1
111111111111111111111111111111111
000000000000000000000000000000000
0
1 UCL
1
0
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
µ 000000000000000000000000000000000
111111111111111111111111111111111
0
1
0
1 Center line
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
µ - c 000000000000000000000000000000000
111111111111111111111111111111111
0
1
0
1 LCL
0
1
0
1
0
1
0
1
1 2 3 4 5 6 7 8 9 10 11 12

Figure 1.3: Control Chart

Copyright c 2018 D.P. Kroese

8 Random Experiments and Probability Models

Example 1.3 (Machine Lifetime) Suppose 1000 identical components are

monitored for failure, up to 50,000 hours. The outcome of such a random
experiment is typically summarised via the cumulative lifetime table and plot, as
given in Table 1.1 and Figure 1.3, respectively. Here F̂ (t) denotes the proportion
of components that have failed at time t. One question is how F̂ (t) can be
modelled via a continuous function F , representing the lifetime distribution of
a typical component.

t (h) failed Fd(t) t (h) failed Fd(t)

0 0 0.000 3000 140 0.140
750 22 0.020 5000 200 0.200
800 30 0.030 6000 290 0.290
900 36 0.036 8000 350 0.350
1400 42 0.042 11000 540 0.540
1500 58 0.058 15000 570 0.570
2000 74 0.074 19000 770 0.770
2300 105 0.105 37000 920 0.920

Table 1.1: The cumulative lifetime table

0.8
0.6
F(t) (est.)

0.4
0.2
0.0

0 10000 20000 30000

Figure 1.4: The cumulative lifetime table

Copyright c 2018 D.P. Kroese

1.1 Random Experiments 9

Example 1.4 A 4-engine aeroplane is able to fly on just one engine on each
wing. All engines are unreliable.

Figure 1.5: A aeroplane with 4 unreliable engines

Number the engines: 1,2 (left wing) and 3,4 (right wing). Observe which engine
works properly during a specified period of time. There are 24 = 16 possible
outcomes of the experiment. Which outcomes lead to “system failure”? More-
over, if the probability of failure within some time period is known for each of
the engines, what is the probability of failure for the entire system? Again this
can be viewed as a random experiment.

Below are two more pictures of randomness. The first is a computer-generated

“plant”, which looks remarkably like a real plant. The second is real data
depicting the number of bytes that are transmitted over some communications
link. An interesting feature is that the data can be shown to exhibit “fractal”
behaviour, that is, if the data is aggregated into smaller or larger time intervals,
a similar picture will appear.

14000

12000

10000
number of bytes

8000

6000

4000

2000

0
125 130 135 140 145 150 155
Interval

Figure 1.6: Plant growth

Figure 1.7: Telecommunications data

We wish to describe these experiments via appropriate mathematical models.

These models consist of three building blocks: a sample space, a set of events
and a probability. We will now describe each of these objects.

Copyright c 2018 D.P. Kroese

10 Random Experiments and Probability Models

1.2 Sample Space

Although we cannot predict the outcome of a random experiment with certainty

we usually can specify a set of possible outcomes. This gives the first ingredient
in our model for a random experiment.

Definition 1.1 The sample space Ω of a random experiment is the set of all
possible outcomes of the experiment.

Examples of random experiments with their sample spaces are:

1. Cast two dice consecutively,

Ω = {(1, 1), (1, 2), . . . , (1, 6), (2, 1), . . . , (6, 6)}.

2. The lifetime of a machine (in days),

Ω = R+ = { positive real numbers } .

3. The number of arriving calls at an exchange during a specified time in-

terval,
Ω = {0, 1, · · · } = Z+ .

4. The heights of 10 selected people.

Ω = {(x1 , . . . , x10 ), xi ≥ 0, i = 1, . . . , 10} = R10

+ .

Here (x1 , . . . , x10 ) represents the outcome that the length of the first se-
lected person is x1 , the length of the second person is x2 , et cetera.

Notice that for modelling purposes it is often easier to take the sample space
larger than necessary. For example the actual lifetime of a machine would
certainly not span the entire positive real axis. And the heights of the 10
selected people would not exceed 3 metres.

1.3 Events

Often we are not interested in a single outcome but in whether or not one of a
group of outcomes occurs. Such subsets of the sample space are called events.
Events will be denoted by capital letters A, B, C, . . . . We say that event A
occurs if the outcome of the experiment is one of the elements in A.

Copyright c 2018 D.P. Kroese

1.3 Events 11

Examples of events are:

1. The event that the sum of two dice is 10 or more,

A = {(4, 6), (5, 5), (5, 6), (6, 4), (6, 5), (6, 6)}.

2. The event that a machine lives less than 1000 days,

A = [0, 1000) .

3. The event that out of fifty selected people, five are left-handed,

A = {5} .

Example 1.5 (Coin Tossing) Suppose that a coin is tossed 3 times, and that
we “record” every head and tail (not only the number of heads or tails). The
sample space can then be written as

Ω = {HHH, HHT, HT H, HT T, T HH, T HT, T T H, T T T } ,

where, for example, HTH means that the first toss is heads, the second tails,
and the third heads. An alternative sample space is the set {0, 1}3 of binary
vectors of length 3, e.g., HTH corresponds to (1,0,1), and THH to (0,1,1).

The event A that the third toss is heads is

A = {HHH, HT H, T HH, T T H} .

Since events are sets, we can apply the usual set operations to them:

1. the set A ∪ B (A union B) is the event that A or B or both occur,

2. the set A ∩ B (A intersection B) is the event that A and B both occur,

3. the event Ac (A complement) is the event that A does not occur,

4. if A ⊂ B (A is a subset of B) then event A is said to imply event B.

Two events A and B which have no outcomes in common, that is, A ∩ B = ∅,

are called disjoint events.

Example 1.6 Suppose we cast two dice consecutively. The sample space is
Ω = {(1, 1), (1, 2), . . . , (1, 6), (2, 1), . . . , (6, 6)}. Let A = {(6, 1), . . . , (6, 6)} be
the event that the first die is 6, and let B = {(1, 6), . . . , (1, 6)} be the event
that the second dice is 6. Then A∩B = {(6, 1), . . . , (6, 6)}∩{(1, 6), . . . , (6, 6)} =
{(6, 6)} is the event that both die are 6.

Copyright c 2018 D.P. Kroese

12 Random Experiments and Probability Models

It is often useful to depict events in a Venn diagram, such as in Figure 1.8

PSfrag repla ements

B S
A
C

A\B A\B A \B
Figure 1.8: A Venn diagram

In this Venn diagram we see

(i) A ∩ C = ∅ and therefore events A and C are disjoint.

(ii) (A ∩ B c ) ∩ (Ac ∩ B) = ∅ and hence events A ∩ B c and Ac ∩ B are disjoint.

Example 1.7 (System Reliability) In Figure 1.9 three systems are depicted,
each consisting of 3 unreliable components. The series system works if and only
if (abbreviated as iff) all components work; the parallel system works iff at least
one of the components works; and the 2-out-of-3 system works iff at least 2 out
of 3 components work.
1

1 1 2

1 2 3 2 1 3

Series 3
2 3

Parallel 2-out-of-3
Figure 1.9: Three unreliable systems

Let Ai be the event that the ith component is functioning, i = 1, 2, 3; and let
Da , Db , Dc be the events that respectively the series, parallel and 2-out-of-3
system is functioning. Then,

Da = A1 ∩ A2 ∩ A3 ,

and
D b = A1 ∪ A2 ∪ A3 .

Copyright c 2018 D.P. Kroese

1.4 Probability 13

Also,

Dc = (A1 ∩ A2 ∩ A3 ) ∪ (Ac1 ∩ A2 ∩ A3 ) ∪ (A1 ∩ Ac2 ∩ A3 ) ∪ (A1 ∩ A2 ∩ Ac3 )

= (A1 ∩ A2 ) ∪ (A1 ∩ A3 ) ∪ (A2 ∩ A3 ) .

Two useful results in the theory of sets are the following, due to De Morgan:
If {Ai } is a collection of events (sets) then
!c
[ \
Ai = Aci (1.1)
i i

and !c
\ [
Ai = Aci . (1.2)
i i
This is easily proved via Venn diagrams. Note that if we interpret Ai as the
event that a component works, then the left-hand side of (1.1) is the event that
the corresponding parallel system is not working. The right hand is the event
that at all components are not working. Clearly these two events are the same.

1.4 Probability

The third ingredient in the model for a random experiment is the specification
of the probability of the events. It tells us how likely it is that a particular event
will occur.

Definition 1.2 A probability P is a rule (function) which assigns a positive

number to each event, and which satisfies the following axioms:
Axiom 1: P(A) ≥ 0.
Axiom 2: P(Ω) = 1.
Axiom 3: For any sequence A1 , A2 , . . . of disjoint events we have
[ X
P( Ai ) = P(Ai ) . (1.3)
i i

Axiom 2 just states that the probability of the “certain” event Ω is 1. Property
(1.3) is the crucial property of a probability, and is sometimes referred to as the
sum rule. It just states that if an event can happen in a number of different
ways that cannot happen at the same time, then the probability of this event is
simply the sum of the probabilities of the composing events.

Note that a probability rule P has exactly the same properties as the common
“area measure”. For example, the total area of the union of the triangles in
Figure 1.10 is equal to the sum of the areas of the individual triangles. This

Copyright c 2018 D.P. Kroese

14 Random Experiments and Probability Models

Figure 1.10: The probability measure has the same properties as the “area”
measure: the total area of the triangles is the sum of the areas of the idividual
triangles.

is how you should interpret property (1.3). But instead of measuring areas, P
measures probabilities.

As a direct consequence of the axioms we have the following properties for P.

Theorem 1.1 Let A and B be events. Then,

1. P(∅) = 0.

2. A ⊂ B =⇒ P(A) ≤ P(B).

3. P(A) ≤ 1.

4. P(Ac ) = 1 − P(A).

5. P(A ∪ B) = P(A) + P(B) − P(A ∩ B).

Proof.

1. Ω = Ω ∩ ∅ ∩ ∅ ∩ · · · , therefore, by the sum rule, P(Ω) = P(Ω) + P(∅) +

P(∅) + · · · , and therefore, by the second axiom, 1 = 1 + P(∅) + P(∅) + · · · ,
from which it follows that P(∅) = 0.

2. If A ⊂ B, then B = A ∪ (B ∩ Ac ), where A and B ∩ Ac are disjoint. Hence,

by the sum rule, P(B) = P(A) + P(B ∩ Ac ), which is (by the first axiom)
greater than or equal to P(A).

3. This follows directly from property 2 and axiom 2, since A ⊂ Ω.

4. Ω = A ∪ Ac , where A and Ac are disjoint. Hence, by the sum rule and

axiom 2: 1 = P(Ω) = P(A) + P(Ac ), and thus P(Ac ) = 1 − P(A).

5. Write A ∪ B as the disjoint union of A and B ∩ Ac . Then, P(A ∪ B) =

P(A) + P(B ∩ Ac ). Also, B = (A ∩ B) ∪ (B ∩ Ac ), so that P(B) =
P(A ∩ B) + P(B ∩ Ac ). Combining these two equations gives P(A ∪ B) =
P(A) + P(B) − P(A ∩ B).

Copyright c 2018 D.P. Kroese

1.4 Probability 15

We have now completed our model for a random experiment. It is up to the

modeller to specify the sample space Ω and probability measure P which most
closely describes the actual experiment. This is not always as straightforward
as it looks, and sometimes it is useful to model only certain observations in the
experiment. This is where random variables come into play, and we will discuss
these in the next chapter.

Example 1.8 Consider the experiment where we throw a fair die. How should
we define Ω and P?

Obviously, Ω = {1, 2, . . . , 6}; and some common sense shows that we should
define P by
|A|
P(A) = , A ⊂ Ω,
6
where |A| denotes the number of elements in set A. For example, the probability
of getting an even number is P({2, 4, 6}) = 3/6 = 1/2.

In many applications the sample space is countable, i.e. Ω = {a1 , a2 , . . . , an } or

Ω = {a1 , a2 , . . .}. Such a sample space is called discrete.

The easiest way to specify a probability P on a discrete sample space is to

specify first the probability pi of each elementary event {ai } and then to
define X
P(A) = pi , for all A ⊂ Ω.
i:ai ∈A

This idea is graphically represented in Figure 1.11. Each element ai in the

sample is assigned a probability weight pi represented by a black dot. To find
the probability of the set A we have to sum up the weights of all the elements
in A.

1
0
0
1
Ω 000
111
000
111
11
00 111
000
1
0
0
1 00 111
11 000
000
111 11
00
11
00 111
000
000
111
000
111
1
0 1
0
0
1
1
0
1
0 0
1
11
00 11
00
00
11 00
11
00
11 00
11
111
000 11
00 11
00
11
00 000
111 00
11
00
11 000
111 00
11
1
0 00
11 000
111
000
111
00
11
00
11 1
0
0
1
000
111 A
11
00 1
0
00
11 0
1
00
11

Figure 1.11: A discrete sample space

Copyright c 2018 D.P. Kroese

16 Random Experiments and Probability Models

Again, it is up to the modeller to properly specify these probabilities. Fortu-

nately, in many applications all elementary events are equally likely, and thus
the probability of each elementary event is equal to 1 divided by the total num-
ber of elements in Ω. E.g., in Example 1.8 each elementary event has probability
1/6.

Because the “equally likely” principle is so important, we formulate it as a

theorem.

Theorem 1.2 (Equilikely Principle) If Ω has a finite number of outcomes,

and all are equally likely, then the probability of each event A is defined as

|A|
P(A) = .
|Ω|

Thus for such sample spaces the calculation of probabilities reduces to counting
the number of outcomes (in A and Ω).

When the sample space is not countable, for example Ω = R+ , it is said to be

continuous.

Example 1.9 We draw at random a point in the interval [0, 1]. Each point is
equally likely to be drawn. How do we specify the model for this experiment?

The sample space is obviously Ω = [0, 1], which is a continuous sample space.
We cannot define P via the elementary events {x}, x ∈ [0, 1] because each of
these events must have probability 0 (!). However we can define P as follows:
For each 0 ≤ a ≤ b ≤ 1, let

P([a, b]) = b − a .

This completely specifies P. In particular, we can find the probability that the
point falls into any (sufficiently nice) set A as the length of that set.

1.5 Counting

Counting is not always easy. Let us first look at some examples:

1. A multiple choice form has 20 questions; each question has 3 choices. In

how many possible ways can the exam be completed?

2. Consider a horse race with 8 horses. How many ways are there to gamble
on the placings (1st, 2nd, 3rd).

3. Jessica has a collection of 20 CDs, she wants to take 3 of them to work.

How many possibilities does she have?

Copyright c 2018 D.P. Kroese

1.5 Counting 17

4. How many different throws are possible with 3 dice?

To be able to comfortably solve a multitude of counting problems requires a

lot of experience and practice, and even then, some counting problems remain
exceedingly hard. Fortunately, many counting problems can be cast into the
simple framework of drawing balls from an urn, see Figure 1.12.

Take k balls 1
5 2 9
Replace balls (yes/no) 3 6 4
Note order (yes/no) 8 10 7

Urn (n balls)

Figure 1.12: An urn with n balls

Consider an urn with n different balls, numbered 1, . . . , n from which k balls are
drawn. This can be done in a number of different ways. First, the balls can be
drawn one-by-one, or one could draw all the k balls at the same time. In the first
case the order in which the balls are drawn can be noted, in the second case
that is not possible. In the latter case we can (and will) still assume the balls are
drawn one-by-one, but that the order is not noted. Second, once a ball is drawn,
it can either be put back into the urn (after the number is recorded), or left
out. This is called, respectively, drawing with and without replacement. All
in all there are 4 possible experiments: (ordered, with replacement), (ordered,
without replacement), (unordered, without replacement) and (ordered, with
replacement). The art is to recognise a seemingly unrelated counting problem
as one of these four urn problems. For the 4 examples above we have the
following

1. Example 1 above can be viewed as drawing 20 balls from an urn containing

3 balls, noting the order, and with replacement.

2. Example 2 is equivalent to drawing 3 balls from an urn containing 8 balls,

noting the order, and without replacement.

3. In Example 3 we take 3 balls from an urn containing 20 balls, not noting

the order, and without replacement

4. Finally, Example 4 is a case of drawing 3 balls from an urn containing 6

balls, not noting the order, and with replacement.

Before we proceed it is important to introduce a notation that reflects whether

the outcomes/arrangements are ordered or not. In particular, we denote ordered
arrangements by vectors, e.g., (1, 2, 3) 6= (3, 2, 1), and unordered arrangements

Copyright c 2018 D.P. Kroese

18 Random Experiments and Probability Models

by sets, e.g., {1, 2, 3} = {3, 2, 1}. We now consider for each of the four cases
how to count the number of arrangements. For simplicity we consider for each
case how the counting works for n = 4 and k = 3, and then state the general
situation.

Drawing with Replacement, Ordered

Here, after we draw each ball, note the number on the ball, and put the ball
back. Let n = 4, k = 3. Some possible outcomes are (1, 1, 1), (4, 1, 2), (2, 3, 2),
(4, 2, 1), . . . To count how many such arrangements there are, we can reason as
follows: we have three positions (·, ·, ·) to fill in. Each position can have the
numbers 1,2,3 or 4, so the total number of possibilities is 4 × 4 × 4 = 43 = 64.
This is illustrated via the following tree diagram:

Third position
Second position
First position
(1,1,1)
1
2
3
4
1

3
(3,2,1)

For general n and k we can reason analogously to find:

The number of ordered arrangements of k numbers chosen from

{1, . . . , n}, with replacement (repetition) is nk .

Drawing Without Replacement, Ordered

Here we draw again k numbers (balls) from the set {1, 2, . . . , n}, and note the
order, but now do not replace them. Let n = 4 and k = 3. Again there
are 3 positions to fill (·, ·, ·), but now the numbers cannot be the same, e.g.,
(1,4,2),(3,2,1), etc. Such an ordered arrangements called a permutation of

Copyright c 2018 D.P. Kroese

1.5 Counting 19

size k from set {1, . . . , n}. (A permutation of {1, . . . , n} of size n is simply

called a permutation of {1, . . . , n} (leaving out “of size n”). For the 1st position
we have 4 possibilities. Once the first position has been chosen, we have only
3 possibilities left for the second position. And after the first two positions
have been chosen there are 2 positions left. So the number of arrangements is
4×3×2 = 24 as illustrated in Figure 1.5, which is the same tree as in Figure 1.5,
but with all “duplicate” branches removed.

Third position
Second position
First position

2
3
4
1
1

2 3
1
4 (2,3,1)
4 (2,3,4)

3 1
2 (3,2,1)

4
4

1
2
3

For general n and k we have:

The number of permutations of size k from {1, . . . , n} is n Pk =

n(n − 1) · · · (n − k + 1).

In particular, when k = n, we have that the number of ordered arrangements

of n items is n! = n(n − 1)(n − 2) · · · 1, where n! is called n-factorial. Note
that
n n!
Pk = .
(n − k)!

Drawing Without Replacement, Unordered

This time we draw k numbers from {1, . . . , n} but do not replace them (no
replication), and do not note the order (so we could draw them in one grab).
Taking again n = 4 and k = 3, a possible outcome is {1, 2, 4}, {1, 2, 3}, etc.
If we noted the order, there would be n Pk outcomes, amongst which would be
(1,2,4),(1,4,2),(2,1,4),(2,4,1),(4,1,2) and (4,2,1). Notice that these 6 permuta-
tions correspond to the single unordered arrangement {1, 2, 4}. Such unordered

Copyright c 2018 D.P. Kroese

20 Random Experiments and Probability Models

arrangements without replications are called combinations of size k from the

set {1, . . . , n}.

To determine the number of combinations of size k simply need to divide n Pk

be the number of permutations of k items, which is k!. Thus, in our example
(n = 4, k = 3) there are 24/6 = 4 possible combinations of size 3. In general
we have:

The number of combinations of size k from the set {1, . . . n} is

  n
n n Pk n!
Ck = = = .
k k! (n − k)! k!

Note the two different notations for this number. We will use the second one.

Drawing With Replacement, Unordered

Taking n = 4, k = 3, possible outcomes are {3, 3, 4}, {1, 2, 4}, {2, 2, 2}, etc.
The trick to solve this counting problem is to represent the outcomes in a
different way, via an ordered vector (x1 , . . . , xn ) representing how many times
an element in {1, . . . , 4} occurs. For example, {3, 3, 4} corresponds to (0, 0, 2, 1)
and {1, 2, 4} corresponds to (1, 1, 0, 1). Thus, we can count how many distinct
vectors (x1 , . . . , xn ) there are such that the sum of the components is 3, and
each xi can take value 0,1,2 or 3. Another way of looking at this is to consider
placing k = 3 balls into n = 4 urns, numbered 1,. . . ,4. Then (0, 0, 2, 1) means
that the third urn has 2 balls and the fourth urn has 1 ball. One way to
distribute the balls over the urns is to distribute n − 1 = 3 “separators” and
k = 3 balls over n − 1 + k = 6 positions, as indicated in Figure 1.13.

1 2 3 4 5 6

Figure 1.13: distributing k balls over n urns

The number of ways this can be done is the equal to the number of ways
k
positions for the balls can be chosen out of n − 1 + k positions, that is, n+k−1
k .
We thus have:

The number of different sets {x1 , . . . , xk } with xi ∈ {1, . . . , n}, i =

1, . . . , k is  
n+k−1
.
k

Copyright c 2018 D.P. Kroese

1.5 Counting 21

Returning to our original four problems, we can now solve them easily:

1. The total number of ways the exam can be completed is 320 = 3, 486, 784, 401.
2. The number of placings is 8 P3 = 336.
20


3. The number of possible combinations of CDs is = 1140.
3

4. The number of different throws with three dice is 83 = 56.

More examples

Here are some more examples. Not all problems can be directly related to the
4 problems above. Some require additional reasoning. However, the counting
principles remain the same.

1. In how many ways can the numbers 1,. . . ,5 be arranged, such as 13524,
25134, etc?
Answer: 5! = 120.
2. How many different arrangements are there of the numbers 1,2,. . . ,7, such
that the first 3 numbers are 1,2,3 (in any order) and the last 4 numbers
are 4,5,6,7 (in any order)?
Answer: 3! × 4!.
3. How many different arrangements are there of the word “arrange”, such
as “aarrnge”, “arrngea”, etc?
Answer: Convert this into a ball drawing problem with 7 balls, numbered
1,. . . ,7. Balls 1 and 2 correspond to ’a’, balls 3 and 4 to ’r’, ball 5 to ’n’,
ball 6 to ’g’ and ball 7 to ’e’. The total number of permutations of the
numbers is 7!. However, since, for example, (1,2,3,4,5,6,7) is identical to
(2,1,3,4,5,6,7) (when substituting the letters back), we must divide 7! by
2! × 2! to account for the 4 ways the two ’a’s and ’r’s can be arranged. So
the answer is 7!/4 = 1260.
4. An urn has 1000 balls, labelled 000, 001, . . . , 999. How many balls are
there that have all number in ascending order (for example 047 and 489,
but not 033 or 321)?
Answer: There are 10 × 9 × 8 = 720 balls with different numbers. Each
triple of numbers can be arranged in 3! = 6 ways, and only one of these
is in ascending order. So the total number of balls in ascending order is
720/6 = 120.
5. In a group of 20 people each person has a different birthday. How many
different arrangements of these birthdays are there (assuming each year
has 365 days)?
365 P
Answer: 20 .

Copyright c 2018 D.P. Kroese

22 Random Experiments and Probability Models

Once we’ve learned how to count, we can apply the equilikely principle to
calculate probabilities:

1. What is the probability that out of a group of 40 people all have different
birthdays?
Answer: Choosing the birthdays is like choosing 40 balls with replace-
ment from an urn containing the balls 1,. . . ,365. Thus, our sample
space Ω consists of vectors of length 40, whose components are cho-
sen from {1, . . . , 365}. There are |Ω| = 36540 such vectors possible,
and all are equally likely. Let A be the event that all 40 people have
different birthdays. Then, |A| = 365 P40 = 365!/325! It follows that
P(A) = |A|/|Ω| ≈ 0.109, so not very big!
2. What is the probability that in 10 tosses with a fair coin we get exactly
5 Heads and 5 Tails?
Answer: Here Ω consists of vectors of length 10 consisting of 1s (Heads)
and 0s (Tails), so there are 210 of them, and all are equally likely. Let A
be the event of exactly 5 heads. We must count how many binary vectors
there are with exactly 5 1s. This is equivalent to determining in how
many ways
the positions of the 5 1s can be chosen out of 10 positions,
10 10
that is, 5 . Consequently, P(A) = 5 /2 = 252/1024 ≈ 0.25.

10

3. We draw at random 13 cards from a full deck of cards. What is the

probability that we draw 4 Hearts and 3 Diamonds?
Answer: Give the cards a number from 1 to 52. Suppose 1–13 is Hearts,
14–26 is Diamonds, etc. Ω consists of unordered
sets of size 13, without
repetition, e.g., {1, 2, . . . , 13}. There are |Ω| = 52
13 of these sets, and they
are all equally likely. Let A be the event of 4 Hearts and 3 Diamonds.
To form A we have to choose 4 Hearts out of 13, and 3 Diamonds out
of
13, followed
by 6 cards out of 26 Spade and Clubs. Thus, |A| =
13 13 26
6 . So that P(A) = |A|/|Ω| ≈ 0.074.


4 × 3 ×

1.6 Conditional probability and independence

How do probabilities change when we know A

B
some event B ⊂ Ω has occurred? Suppose B
has occurred. Thus, we know that the out-
come lies in B. Then A will occur if and only
if A ∩ B occurs, and the relative chance of A
occurring is therefore
P(A ∩ B)/P(B).
This leads to the definition of the condi-
tional probability of A given B: Ω

Copyright c 2018 D.P. Kroese

1.6 Conditional probability and independence 23

P(A ∩ B)
P(A | B) = (1.4)
P(B)

Example 1.10 We throw two dice. Given that the sum of the eyes is 10, what
is the probability that one 6 is cast?

Let B be the event that the sum is 10,

B = {(4, 6), (5, 5), (6, 4)}.

Let A be the event that one 6 is cast,

A = {(1, 6), . . . , (5, 6), (6, 1), . . . , (6, 5)}.

Then, A ∩ B = {(4, 6), (6, 4)}. And, since all elementary events are equally
likely, we have
2/36 2
P(A | B) = = .
3/36 3

Example 1.11 (Monte Hall Problem) This is a nice application of condi-

tional probability. Consider a quiz in which the final contestant is to choose a
prize which is hidden behind one three curtains (A, B or C). Suppose without
loss of generality that the contestant chooses curtain A. Now the quiz master
(Monte Hall) always opens one of the other curtains: if the prize is behind B,
Monte opens C, if the prize is behind C, Monte opens B, and if the prize is
behind A, Monte opens B or C with equal probability, e.g., by tossing a coin
(of course the contestant does not see Monte tossing the coin!).

A B C

Suppose, again without loss of generality that Monte opens curtain B. The
contestant is now offered the opportunity to switch to curtain C. Should the
contestant stay with his/her original choice (A) or switch to the other unopened
curtain (C)?

Notice that the sample space consists here of 4 possible outcomes: Ac: The
prize is behind A, and Monte opens C; Ab: The prize is behind A, and Monte
opens B; Bc: The prize is behind B, and Monte opens C; and Cb: The prize

Copyright c 2018 D.P. Kroese

24 Random Experiments and Probability Models

is behind C, and Monte opens B. Let A, B, C be the events that the prize
is behind A, B and C, respectively. Note that A = {Ac, Ab}, B = {Bc} and
C = {Cb}, see Figure 1.14.

Ac Ab
1/6 1/6

Cb Bc
1/3 1/3

Figure 1.14: The sample space for the Monte Hall problem.

Now, obviously P(A) = P(B) = P(C), and since Ac and Ab are equally likely,
we have P({Ab}) = P({Ac}) = 1/6. Monte opening curtain B means that we
have information that event {Ab, Cb} has occurred. The probability that the
prize is under A given this event, is therefore
P({Ac, Ab} ∩ {Ab, Cb}) P({Ab}) 1/6 1
P(A | B is opened) = = = = .
P({Ab, Cb}) P({Ab, Cb}) 1/6 + 1/3 3
This is what we expected: the fact that Monte opens a curtain does not
give us any extra information that the prize is behind A. So one could think
that it doesn’t matter to switch or not. But wait a minute! What about
P(B | B is opened)? Obviously this is 0 — opening curtain B means that we
know that event B cannot occur. It follows then that P(C | B is opened) must
be 2/3, since a conditional probability behaves like any other probability and
must satisfy axiom 2 (sum up to 1). Indeed,
P({Cb} ∩ {Ab, Cb}) P({Cb}) 1/3 2
P(C | B is opened) = = = = .
P({Ab, Cb}) P({Ab, Cb}) 1/6 + 1/3 3
Hence, given the information that B is opened, it is twice as likely that the
prize is under C than under A. Thus, the contestant should switch!

1.6.1 Product Rule

By the definition of conditional probability we have

P(A ∩ B) = P(A) P(B | A). (1.5)
We can generalise this to n intersections A1 ∩A2 ∩ · · · ∩An , which we abbreviate
as A1 A2 · · · An . This gives the product rule of probability (also called chain
rule).

Theorem 1.3 (Product rule) Let A1 , . . . , An be a sequence of events with

P(A1 . . . An−1 ) > 0. Then,

P(A1 · · · An ) = P(A1 ) P(A2 | A1 ) P(A3 | A1 A2 ) · · · P(An | A1 · · · An−1 ). (1.6)

Copyright c 2018 D.P. Kroese

1.6 Conditional probability and independence 25

Proof. We only show the proof for 3 events, since the n > 3 event case follows
similarly. By applying (1.4) to P(B | A) and P(C | A ∩ B), the left-hand side of
(1.6) is we have,

P(A ∩ B) P(A ∩ B ∩ C)
P(A) P(B | A) P(C | A ∩ B) = P(A) = P(A ∩ B ∩ C) ,
P(A) P(A ∩ B)

which is equal to the left-hand size of (1.6).

Example 1.12 We draw consecutively 3 balls from a bowl with 5 white and 5
black balls, without putting them back. What is the probability that all balls
will be black?

Solution: Let Ai be the event that the ith ball is black. We wish to find the
probability of A1 A2 A3 , which by the product rule (1.6) is
5 43
P(A1 ) P(A2 | A1 ) P(A3 | A1 A2 ) = = 0.083.
10 9 8
Note that this problem can also be easily solved by counting arguments, as in
the previous section.

Example 1.13 (Birthday Problem) In Section 1.5 we derived by counting

arguments that the probability that all people in a group of 40 have different
birthdays is
365 × 364 × · · · × 326
≈ 0.109. (1.7)
365 × 365 × · · · × 365
We can derive this also via the product rule. Namely, let Ai be the event that
the first i people have different birthdays, i = 1, 2, . . .. Note that A1 ⊃ A2 ⊃
A3 ⊃ · · · . Therefore An = A1 ∩ A2 ∩ · · · ∩ An , and thus by the product rule

P(A40 ) = P(A1 )P(A2 | A1 )P(A3 | A2 ) · · · P(A40 | A39 ) .

Now P(Ak | Ak−1 = (365 − k + 1)/365 because given that the first k − 1 people
have different birthdays, there are no duplicate birthdays if and only if the
birthday of the k-th is chosen from the 365 − (k − 1) remaining birthdays.
Thus, we obtain (1.7). More generally, the probability that n randomly selected
people have different birthdays is
365 364 363 365 − n + 1
P(An ) = × × × ··· × , n≥1.
365 365 365 365
A graph of P(An ) against n is given in Figure 1.15. Note that the probability
P(An ) rapidly decreases to zero. Indeed, for n = 23 the probability of having
no duplicate birthdays is already less than 1/2.

Copyright c 2018 D.P. Kroese

26 Random Experiments and Probability Models

0.8

0.6

0.4

0.2

10 20 30 40 50 60

Figure 1.15: The probability of having no duplicate birthday in a group of n

people, against n.

1.6.2 Law of Total Probability and Bayes’ Rule

Suppose B1 , B2 , . . . , Bn is a partition of Ω. That is, B1 , B2 , . . . , Bn are disjoint

and their union is Ω, see Figure 1.16

B1 B2 B3 B4 B5 B6

Figure 1.16: A partition of the sample space

Pn
Then, by the sum rule, P(A) = i=1 P(A ∩ Bi ) and hence, by the definition of
conditional probability we have
Pn
P(A) = i=1 P(A|Bi ) P(Bi )

This is called the law of total probability.

Combining the Law of Total Probability with the definition of conditional prob-
ability gives Bayes’ Rule:

P(A|Bj ) P(Bj )
P(Bj |A) = Pn
i=1 P(A|Bi )P(Bi )

Example 1.14 A company has three factories (1, 2 and 3) that produce the
same chip, each producing 15%, 35% and 50% of the total production. The

Copyright c 2018 D.P. Kroese

1.6 Conditional probability and independence 27

probability of a defective chip at 1, 2, 3 is 0.01, 0.05, 0.02, respectively. Suppose

someone shows us a defective chip. What is the probability that this chip comes
from factory 1?

Let Bi denote the event that the chip is produced by factory i. The {Bi } form
a partition of Ω. Let A denote the event that the chip is faulty. By Bayes’ rule,
0.15 × 0.01
P(B1 | A) = = 0.052 .
0.15 × 0.01 + 0.35 × 0.05 + 0.5 × 0.02

1.6.3 Independence

Independence is a very important concept in probability and statistics. Loosely

speaking it models the lack of information between events. We say A and
B are independent if the knowledge that A has occurred does not change the
probability that B occurs. That is
A, B independent ⇔ P(A|B) = P(A)

Since P(A|B) = P(A ∩ B)/P(B) an alternative definition of independence is

A, B independent ⇔ P(A ∩ B) = P(A)P(B)

This definition covers the case B = ∅ (empty set). We can extend the definition
to arbitrarily many events:

Definition 1.3 The events A1 , A2 , . . . , are said to be (mutually) indepen-

dent if for any n and any choice of distinct indices i1 , . . . , ik ,

P(Ai1 ∩ Ai2 ∩ · · · ∩ Aik ) = P(Ai1 )P(Ai2 ) · · · P(Aik ) .

Remark 1.1 In most cases independence of events is a model assumption.

That is, we assume that there exists a P such that certain events are indepen-
dent.

Example 1.15 (A Coin Toss Experiment and the Binomial Law) We flip
a coin n times. We can write the sample space as the set of binary n-tuples:

Ω = {(0, . . . , 0), . . . , (1, . . . , 1)} .

Here 0 represent Tails and 1 represents Heads. For example, the outcome
(0, 1, 0, 1, . . .) means that the first time Tails is thrown, the second time Heads,
the third times Tails, the fourth time Heads, etc.

How should we define P? Let Ai denote the event of Heads during the ith throw,
i = 1, . . . , n. Then, P should be such that the events A1 , . . . , An are independent.
And, moreover, P(Ai ) should be the same for all i. We don’t know whether the
coin is fair or not, but we can call this probability p (0 ≤ p ≤ 1).

Copyright c 2018 D.P. Kroese

28 Random Experiments and Probability Models

These two rules completely specify P. For example, the probability that the
first k throws are Heads and the last n − k are Tails is

P({(1, 1, . . . , 1, 0, 0, . . . , 0)}) = P(A1 ) · · · P(Ak ) · · · P(Ack+1 ) · · · P(Acn )

= pk (1 − p)n−k .

Also, let Bk be the event that there are k Heads in total. The probability of
this event is the sum the probabilities of elementary events {(x1 , . . . , xn )} such
that x1 + · ·
· + xn = k. Each of these events has probability pk (1 − p)n−k , and
there are nk of these. Thus,
 
n k
P(Bk ) = p (1 − p)n−k , k = 0, 1, . . . , n .
k

We have thus discovered the binomial distribution.

Example 1.16 (Geometric Law) There is another important law associated

with the coin flip experiment. Suppose we flip the coin until Heads appears for
the first time. Let Ck be the event that Heads appears for the first time at the
k-th toss, k = 1, 2, . . .. Then, using the same events {Ai } as in the previous
example, we can write

Ck = Ac1 ∩ Ac2 ∩ · · · ∩ Ack−1 ∩ Ak ,

so that with the product law and the mutual independence of Ac1 , . . . , Ak we
have the geometric law:

P(Ck ) = P(Ac1 ) · · · P(Ack−1 ) P(Ak )

= (1 − p) · · · (1 − p)p = (1 − p)k−1 p .
| {z }
k−1 times

Copyright c 2018 D.P. Kroese

Chapter 2

Random Variables and

Probability Distributions

Specifying a model for a random experiment via a complete description of Ω

and P may not always be convenient or necessary. In practice we are only inter-
ested in various observations (i.e., numerical measurements) of the experiment.
We include these into our modelling process via the introduction of random
variables.

2.1 Random Variables

Formally a random variable is a function from the sample space Ω to R. Here

is a concrete example.

Example 2.1 (Sum of two dice) Suppose we toss two fair dice and note
their sum. If we throw the dice one-by-one and observe each throw, the sample
space is Ω = {(1, 1), . . . , (6, 6)}. The function X, defined by X(i, j) = i + j, is
a random variable, which maps the outcome (i, j) to the sum i + j, as depicted
in Figure 2.1. Note that all the outcomes in the “encircled” set are mapped to
8. This is the set of all outcomes whose sum is 8. A natural notation for this
set is to write {X = 8}. Since this set has 5 outcomes, and all outcomes in Ω
are equally likely, we have
5
P({X = 8}) = .
36
This notation is very suggestive and convenient. From a non-mathematical
viewpoint we can interpret X as a “random” variable. That is a variable that
can take on several values, with certain probabilities. In particular it is not
difficult to check that
6 − |7 − x|
P({X = x}) = , x = 2, . . . , 12.
36

Copyright c 2018 D.P. Kroese

30 Random Variables and Probability Distributions

X
Ω
6
5
4
3 2 3 4 5 6 7 8 9 10 11 12

2
1
1 2 3 4 5 6

Figure 2.1: A random variable representing the sum of two dice

Although random variables are, mathematically speaking, functions, it is often

convenient to view random variables as observations of a random experiment
that has not yet been carried out. In other words, a random variable is consid-
ered as a measurement that becomes available once we carry out the random
experiment, e.g., tomorrow. However, all the thinking about the experiment and
measurements can be done today. For example, we can specify today exactly
the probabilities pertaining to the random variables.

We usually denote random variables with capital letters from the last part of the
alphabet, e.g. X, X1 , X2 , . . . , Y, Z. Random variables allow us to use natural
and intuitive notations for certain events, such as {X = 10}, {X > 1000},
{max(X, Y ) ≤ Z}, etc.

Example 2.2 We flip a coin n times. In Example 1.15 we can find a probability
model for this random experiment. But suppose we are not interested in the
complete outcome, e.g., (0,1,0,1,1,0. . . ), but only in the total number of heads
(1s). Let X be the total number of heads. X is a “random variable” in the
true sense of the word: X could lie anywhere between 0 and n. What we are
interested in, however, is the probability that X takes certain values. That is,
we are interested in the probability distribution of X. Example 1.15 now
suggests that
 
n k
P(X = k) = p (1 − p)n−k , k = 0, 1, . . . , n . (2.1)
k

This contains all the information about X that we could possibly wish to know.
Example 2.1 suggests how we can justify this mathematically Define X as the
function that assigns to each outcome ω = (x1 , . . . , xn ) the number x1 +· · ·+xn .
Then clearly X is a random variable in mathematical terms (that is, a function).
Moreover, the event that there are exactly k Heads in n throws can be written
as
{ω ∈ Ω : X(ω) = k}.
If we abbreviate this to {X = k}, and further abbreviate P({X = k}) to
P(X = k), then we obtain exactly (2.1).

Copyright c 2018 D.P. Kroese

2.2 Probability Distribution 31

We give some more examples of random variables without specifying the sample
space.

1. The number of defective transistors out of 100 inspected ones,

2. the number of bugs in a computer program,

3. the amount of rain in Brisbane in June,

4. the amount of time needed for an operation.

The set of all possible values a random variable X can take is called the range
of X. We further distinguish between discrete and continuous random variables:

Discrete random variables can only take isolated values.

For example: a count can only take non-negative integer values.

Continuous random variables can take values in an interval.

For example: rainfall measurements, lifetimes of components, lengths,
. . . are (at least in principle) continuous.

2.2 Probability Distribution

Let X be a random variable. We would like to specify the probabilities of events

such as {X = x} and {a ≤ X ≤ b}.

If we can specify all probabilities involving X, we say that we have

specified the probability distribution of X.

One way to specify the probability distribution is to give the probabilities of all
events of the form {X ≤ x}, x ∈ R. This leads to the following definition.

Definition 2.1 The cumulative distribution function (cdf) of a random

variable X is the function F : R → [0, 1] defined by

F (x) := P(X ≤ x), x ∈ R.

Note that above we should have written P({X ≤ x}) instead of P(X ≤ x).
From now on we will use this type of abbreviation throughout the course. In
Figure 2.2 the graph of a cdf is depicted.

The following properties for F are a direct consequence of the three Axiom’s
for P.

Copyright c 2018 D.P. Kroese

32 Random Variables and Probability Distributions

F(x)

0
x

Figure 2.2: A cumulative distribution function

1. F is right-continuous: limh↓0 F (x + h) = F (x),

2. limx→∞ F (x) = 1; limx→−∞ F (x) = 0.

3. F is increasing: x ≤ y ⇒ F (x) ≤ F (y),

4. 0 ≤ F (x) ≤ 1.

Proof. We will prove (1) and (2) in STAT3004. For (3), suppose x ≤ y and
define A = {X ≤ x} and B = {X ≤ y}. Then, obviously, A ⊂ B (for example
if {X ≤ 3} then this implies {X ≤ 4}). Thus, by (2) on page 14, P(A) ≤ P(B),
which proves (3). Property (4) follows directly from the fact that 0 ≤ P(A) ≤ 1
for any event A — and hence in particular for A = {X ≤ x}.

Any function F with the above properties can be used to specify the distribution
of a random variable X. Suppose that X has cdf F . Then the probability that
X takes a value in the interval (a, b] (excluding a, including b) is given by

P(a < X ≤ b) = F (b) − F (a).

Namely, P(X ≤ b) = P({X ≤ a}∪{a < X ≤ b}), where the events {X ≤ a} and
{a < X ≤ b} are disjoint. Thus, by the sum rule: F (b) = F (a) + P(a < X ≤ b),
which leads to the result above. Note however that
P(a ≤ X ≤ b) = F (b) − F (a) + P(X = a)
= F (b) − F (a) + F (a) − lim F (a − h)
h↓0

= F (b) − lim F (a − h).

h↓0

In practice we will specify the distribution of a random variable in a different

way, whereby we make the distinction between discrete and continuous random
variables.

Copyright c 2018 D.P. Kroese

2.2 Probability Distribution 33

2.2.1 Discrete Distributions

Definition 2.2 We say that X has a discrete distribution if X is a discrete

random variable. In particular, for some finite or P countable set of values
x1 , x2 , . . . we have P(X = xi ) > 0, i = 1, 2, . . . and i P(X = xi ) = 1. We
define the probability mass function (pmf) f of X by f (x) = P(X = x).
We sometimes write fX instead of f to stress that the pmf refers to the random
variable X.

The easiest way to specify the distribution of a discrete random variable is to

specify its pmf. Indeed, by the sum rule, if we know f (x) for all x, then we can
calculate all possible probabilities involving X. Namely,
X
P(X ∈ B) = f (x) (2.2)
x∈B

for any subset B of the range of X.

Example 2.3 Toss a die and let X be its face value. X is discrete with range
{1, 2, 3, 4, 5, 6}. If the die is fair the probability mass function is given by

P
x 1 2 3 4 5 6
1 1 1 1 1 1
f (x) 1
6 6 6 6 6 6

Example 2.4 Toss two dice and let X be the largest face value showing. The
pmf of X can be found to satisfy

P
x 1 2 3 4 5 6
1 3 5 7 9 11
f (x) 1
36 36 36 36 36 36
P6
The probability that the maximum is at least 3 is P(X ≥ 3) = x=3 f (x) =
32/36 = 8/9.

2.2.2 Continuous Distributions

Definition 2.3 A random variable X is said to have a continuous distri-

bution if X is a continuous random variable for which there exists a positive
function f with total integral 1, such that for all a, b
Z b
P(a < X ≤ b) = F (b) − F (a) = f (u) du. (2.3)
a

Copyright c 2018 D.P. Kroese

34 Random Variables and Probability Distributions

The function f is called the probability density function (pdf) of X.

f(x)

a x b

Figure 2.3: Probability density function (pdf)

Note that the corresponding cdf F is simply a primitive (also called anti-
derivative) of the pdf f . In particular,
Z x
F (x) = P(X ≤ x) = f (u) du.
−∞

Moreover, if a pdf f exists, then f is the derivative of the cdf F :

d
f (x) = F (x) = F 0 (x) .
dx
We can interpret f (x) as the “density” that X = x. More precisely,
Z x+h
P(x ≤ X ≤ x + h) = f (u) du ≈ h f (x) .
x

However, it is important to realise that f (x) is not a probability — is a prob-

ability density. In particular, if X is a continuous random variable, then
P(X = x) = 0, for all x. Note that this also justifies using P(x ≤ X ≤ x + h)
above instead of P(x < X ≤ x + h). Although we will use the same notation f
for probability mass function (in the discrete case) and probability density func-
tion (in the continuous case), it is crucial to understand the difference between
the two cases.

Example 2.5 Draw a random number from the interval of real numbers [0, 2].
Each number is equally possible. Let X represent the number. What is the
probability density function f and the cdf F of X?

Solution: Take an x ∈ [0, 2]. Drawing a number X “uniformly” in [0,2] means

that P(X ≤ x) = x/2, for all such x. In particular, the cdf of X satisfies:

 0 x < 0,
F (x) = x/2 0 ≤ x ≤ 2,
1 x > 2.


By differentiating F we find

1/2 0 ≤ x ≤ 2,
f (x) =
0 otherwise

Copyright c 2018 D.P. Kroese

2.3 Expectation 35

Note that this density is constant on the interval [0, 2] (and zero elsewhere),
reflecting that each point in [0,2] is equally likely. Note also that we have
modelled this random experiment using a continuous random variable and its
pdf (and cdf). Compare this with the more “direct” model of Example 1.9.

Describing an experiment via a random variable and its pdf, pmf or cdf seems
much easier than describing the experiment by giving the probability space. In
fact, we have not used a probability space in the above examples.

2.3 Expectation

Although all the probability information of a random variable is contained in

its cdf (or pmf for discrete random variables and pdf for continuous random
variables), it is often useful to consider various numerical characteristics of that
random variable. One such number is the expectation of a random variable; it
is a sort of “weighted average” of the values that X can take. Here is a more
precise definition.

Definition 2.4 Let X be a discrete random variable with pmf f . The expec-
tation (or expected value) of X, denoted by EX, is defined by
X X
EX = x P(X = x) = x f (x) .
x x

The expectation of X is sometimes written as µX .

Example 2.6 Find EX if X is the outcome of a toss of a fair die.

Since P(X = 1) = . . . = P(X = 6) = 1/6, we have

1 1 1 7
EX = 1 ( ) + 2 ( ) + . . . + 6 ( ) = .
6 6 6 2
Note: EX is not necessarily a possible outcome of the random experiment as
in the previous example.

One way to interpret the expectation is as a type of “expected profit”. Specifi-

cally, suppose we play a game where you throw two dice, and I pay you out, in
dollars, the sum of the dice, X say. However, to enter the game you must pay
me d dollars. You can play the game as many times as you like. What would
be a “fair” amount for d? The answer is
d = EX = 2 P(X = 2) + 3 P(X = 3) + · · · + 12 P(X = 12)
1 2 1
=2 +3 + · · · + 12 =7.
36 35 36

Copyright c 2018 D.P. Kroese

36 Random Variables and Probability Distributions

Namely, in the long run the fractions of times the sum is equal to 2, 3, 4,
1 2 3
. . . are 36 , 35 , 36 , . . ., so the average pay-out per game is the weighted sum of
2,3,4,. . . with the weights being the probabilities/fractions. Thus the game is
“fair” if the average profit (pay-out - d) is zero.

Another interpretation of expectation is as a centre of mass. Imagine that point

masses with weights p1 , p2 , . . . , pn are placed at positions x1 , x2 , . . . , xn on the
real line, see Figure 2.4.

p1 p2 pn

x1 x2 xn

EX

Figure 2.4: The expectation as a centre of mass

Then there centre of mass, the place where we can “balance” the weights, is

centre of mass = x1 p1 + · · · + xn pn ,

which is exactly the expectation of the discrete variable X taking values x1 , . . . , xn

with probabilities p1 , . . . , pn . An obvious consequence of this interpretation is
that for a symmetric probability mass function the expectation is equal to
the symmetry point (provided the expectation exists). In particular, suppose
f (c + y) = f (c − y) for all y, then
X X
EX = c f (c) + xf (x) + xf (x)
x>c x<c
X X
= c f (c) + (c + y)f (c + y) + (c − y)f (c − y)
y>0 y>0
X X
= c f (c) + c f (c + y) + c f (c − y)
y>0 y>0
X
=c f (x) = c
x

For continuous random variables we can define the expectation in a similar way:

Definition 2.5 Let X be a continuous random variable with pdf f . The ex-
pectation (or expected value) of X, denoted by EX, is defined by
Z
EX = x f (x) dx .
x

If X is a random variable, then a function of X, such as X 2 or sin(X) is also

a random variable. The following theorem is not so difficult to prove, and is

Copyright c 2018 D.P. Kroese

2.3 Expectation 37

entirely “obvious”: the expected value of a function of X is the weighted average

of the values that this function can take.

Theorem 2.1 If X is discrete with pmf f , then for any real-valued function g
X
E g(X) = g(x) f (x) .
x

Similarly, if X is continuous with pdf f , then

Z ∞
E g(X) = g(x) f (x) dx .
−∞

Proof. We prove it for the discrete case only. Let Y = g(X), where X is a
discrete random variable with pmf fX , and g is a function. Y is again a random
variable. The pmf of Y , fY satisfies
X X
fY (y) = P(Y = y) = P(g(X) = y) = P(X = x) = fX (x) .
x:g(x)=y x:g(x)=y

Thus, the expectation of Y is

X X X X X X
EY = yfY (y) = y fX (x) = yfX (x) = g(x)fX (x)
y y x:g(x)=y y x:g(x)=y x

Example 2.7 Find EX 2 if X is the outcome of the toss of a fair die. We have
1 1 1 1 91
EX 2 = 12 + 2 2 + 3 2 + . . . + 62 = .
6 6 6 6 6

An important consequence of Theorem 2.1 is that the expectation is “linear”.

More precisely, for any real numbers a and b, and functions g and h

1. E(a X + b) = a EX + b .

2. E(g(X) + h(X)) = Eg(X) + Eh(X) .

Proof. Suppose X has pmf f . Then 1. follows (in the discrete case) from
X X X
E(aX + b) = (ax + b)f (x) = a x f (x) + b f (x) = a EX + b .
x x x

Similarly, 2. follows from

X X X
E(g(X) + h(X)) = (g(x) + h(x))f (x) = g(x)f (x) + h(x)f (x)
x x x
= Eg(X) + Eh(X) .

Copyright c 2018 D.P. Kroese

38 Random Variables and Probability Distributions

The continuous case is proved analogously, by replacing the sum with an inte-
gral.

Another useful number about (the distribution of) X is the variance of X.

2 , measures the spread or dispersion of
This number, sometimes written as σX
the distribution of X.

Definition 2.6 The variance of a random variable X, denoted by Var(X) is

defined by
Var(X) = E(X − EX)2 .

The square root of the variance is called the standard deviation. The number
EX r is called the rth moment of X.

The following important properties for variance hold for discrete or continu-
ous random variables and follow easily from the definitions of expectation and
variance.

1. Var(X) = EX 2 − (EX)2

2. Var(aX + b) = a2 Var(X)

Proof. Write EX = µ, so that Var(X) = E(X − µ)2 = E(X 2 − 2µX + µ2 ).

By the linearity of the expectation, the last expectation is equal to the sum
E(X 2 ) − 2µEX + µ2 = EX 2 − µ2 , which proves 1. To prove 2, note first that
the expectation of aX + b is equal to aµ + b. Thus,
Var(aX + b) = E(aX + b − (aµ + b))2 = E(a2 (X − µ)2 ) = a2 Var(X) .

2.4 Transforms

Many calculations and manipulations involving probability distributions are fa-

cilitated by the use of transforms. We discuss here a number of such transforms.

Definition 2.7 Let X be a non-negative and integer-valued random variable.

The probability generating function (PGF) of X is the function G : [0, 1] →
[0, 1] defined by
X∞
G(z) := E z =
X
z x P(X = x) .
x=0

Example 2.8 Let X have pmf f given by

λx
f (x) = e−λ , x = 0, 1, 2, . . .
x!

Copyright c 2018 D.P. Kroese

2.4 Transforms 39

We will shortly introduce this as the Poisson distribution, but for now this is
not important. The PGF of X is given by
∞
X λx
G(z) = z x e−λ
x!
x=0
∞
−λ
X (zλ)x
= e
x!
x=0
−λ zλ
= e e = e−λ(1−z) .

Knowing only the PGF of X, we can easily obtain the pmf:

1 dx
P(X = x) = G(z) , x = 1, 2, . . . .
x! dz x z=0

Proof. By definition

G(z) = z 0 P(X = 0) + z 1 P(X = 1) + z 2 P(X = 2) + · · ·

Substituting x = 0 gives G(0) = P(X = 0). If we differentiate G(z) once, then

G0 (z) = P(X = 1) + 2z P(X = 2) + 3z 2 P(X = 3) + · · ·

Thus, G0 (0) = P(X = 1). Differentiating again, we see that G00 (0) = 2 P(X =
2), and in general the n-th derivative of G at zero is G(n) (0) = x! P(X = x),
which completes the proof.

Thus we have the uniqueness property: two pmf’s are the same if and only if
their PGFs are the same.

Another useful property of the PGF is that we can obtain the moments of X
by differentiating G and evaluating it at z = 1.

Differentiating G(z) w.r.t. z gives

d Ez X
G0 (z) = = EXz X−1 .
dz
d EXz X−1
G00 (z) = = EX(X − 1)z X−2 .
dz
G000 (z) = EX(X − 1)(X − 2)z X−3 .

Et cetera. If you’re not convinced, write out the expectation as a sum, and use
the fact that the derivative of the sum is equal to the sum of the derivatives
(although we need a little care when dealing with infinite sums).

In particular,
EX = G0 (1),
and
Var(X) = G00 (1) + G0 (1) − (G0 (1))2 .

Copyright c 2018 D.P. Kroese

40 Random Variables and Probability Distributions

Definition 2.8 The moment generating function (MGF) of a random vari-

able X is the function, M : I → [0, ∞), given by
M (s) = E esX .
Here I is an open interval containing 0 for which the above integrals are well
defined for all s ∈ I.

In particular, for a discrete random variable with pmf f ,

X
M (s) = esx f (x),
x
and for a continuous random variable with pdf f ,
Z
M (s) = esx f (x) dx .
x
We sometimes write MX to stress the role of X.

As for the PGF, the moment generation function has the uniqueness property:
Two MGFs are the same if and only if their corresponding distribution functions
are the same.

Similar to the PGF, the moments of X follow from the derivatives of M :

If EX n exists, then M is n times differentiable, and

EX n = M (n) (0).
Hence the name moment generating function: the moments of X are simply
found by differentiating. As a consequence, the variance of X is found as
Var(X) = M 00 (0) − (M 0 (0))2 .

Remark 2.1 The transforms discussed here are particularly useful when deal-
ing with sums of independent random variables. We will return to them in
Chapters 4 and 5.

2.5 Some Important Discrete Distributions

In this section we give a number of important discrete distributions and list some
of their properties. Note that the pmf of each of these distributions depends on
one or more parameters; so in fact we are dealing with families of distributions.

2.5.1 Bernoulli Distribution

We say that X has a Bernoulli distribution with success probability p if X

can only assume the values 0 and 1, with probabilities
P(X = 1) = p = 1 − P(X = 0) .

Copyright c 2018 D.P. Kroese

2.5 Some Important Discrete Distributions 41

We write X ∼ Ber(p). Despite its simplicity, this is one of the most important
distributions in probability! It models for example a single coin toss experiment.
The cdf is given in Figure 2.5.
1

1−p

0 1

Figure 2.5: The cdf of the Bernoulli distribution

Here are some properties:

1. The expectation is EX = 0P(X = 0)+1P(X = 1) = 0×(1−p)+1×p = p.

2. The variance is Var(X) = EX 2 −(EX)2 = EX −(EX)2 = p−p2 = p(1−p).

(Note that X 2 = X).

3. The PGF is given by G(z) = z 0 (1 − p) + z 1 p = 1 − p + zp.

2.5.2 Binomial Distribution

Consider a sequence of n coin tosses. If X is the random variable which counts

the total number of heads and the probability of “head” is p then we say X
has a binomial distribution with parameters n and p and write X ∼ Bin(n, p).
The probability mass function X is given by
 
n x
f (x) = P(X = x) = p (1 − p)n−x , x = 0, 1, . . . , n. (2.4)
x
This follows from Examples 1.15 and 2.2. An example of the graph of the pmf
is given in Figure 2.6

Here are some important properties of the Bernoulli distribution. Some of these
properties can be proved more easily after we have discussed multiple random
variables.

1. The expectation is EX = np. This is a quite intuitive result. The ex-

pected number of successes (heads) in n coin tosses is np, if p denotes the
probability of success in any one toss. To prove this, one could simply
evaluate the sum
n  
X n x
x p (1 − p)n−x ,
x
x=0
but this is not elegant. We will see in chapter 4 that X can be viewed
as a sum X = X1 + · · · + Xn of n independent Ber(p) random variables,

Copyright c 2018 D.P. Kroese

42 Random Variables and Probability Distributions

n=10, p=0.7

0.25

0.2

0.15

0.1

0.05

0 1 2 3 4 5 6 7 8 9 10

Figure 2.6: The pmf of the Bin(10, 0.7)-distribution

where Xi indicates whether the i-th toss is a success or not, i = 1, . . . , n.

Also we will prove that the expectation of such a sum is the sum of the
expectation, therefore,

EX = E(X1 + · · · + Xn ) = EX1 + · · · + EXn = p + · · · p = np .

| {z }
n times

2. The variance of X is Var(X) = np(1 − p). This is proved in a similar way

to the expectation:

Var(X) = Var(X1 + · · · + Xn ) = Var(X1 ) + · · · + Var(Xn )

= p(1 − p) + · · · + p(1 − p) = np(1 − p) .
| {z }
n times

3. The probability generating function of X is G(z) = (1 − p + zp)n . Again,

we can easily prove this after we consider multiple random variables in
Chapter 4. Namely,

G(z) = Ez X = Ez X1 +···+Xn = Ez X1 · · · Ez Xn
= (1 − p + zp) × · · · × (1 − p + zp) = (1 − p + zp)n .

However, we can also easily prove it using Newton’s binomial formula:

n  
n
X n k n−k
(a + b) = a b .
k
k=0

Specifically,
n   n  
X
k n k n−k
X n
G(z) = z p (1−p) = (z p)k (1−p)n−k = (1−p+zp)n .
k k
k=0 k=0

Note that once we have obtained the PGF, we can obtain the expectation
and variance as G0 (1) = np and G00 (1) + G0 (1) − (G0 (1))2 = (n − 1)np2 +
np − n2 p2 = np(1 − p).

Copyright c 2018 D.P. Kroese

2.5 Some Important Discrete Distributions 43

2.5.3 Geometric distribution

Again we look at a sequence of coin tosses but count a different thing. Let X
be the number of tosses needed before the first head occurs. Then

P(X = x) = (1 − p)x−1 p, x = 1, 2, 3, . . . (2.5)

since the only string that has the required form is

ttt
| {z. . . }t h
x−1

and this has probability (1 − p)x−1 p. See also Example 1.16 on page 28. Such a
random variable X is said to have a geometric distribution with parameter p.
We write X ∼ G(p). An example of the graph of the pdf is given in Figure 2.7
p=0.3
0.3

0.25

0.2

0.15

0.1

0.05

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Figure 2.7: The pmf of the G(0.3)-distribution

We give some more properties, including the expectation, variance and PGF of
the geometric distribution. It is easiest to start with the PGF:

1. The PGF is given by

∞ ∞
X
x x−1
X zp
G(z) = z p(1 − p) = zp (z(1 − p))k = ,
1 − z (1 − p)
x=1 k=0

1
using the well-known result for geometric sums: 1 + a + a2 + · · · = 1−a ,
for |a| < 1.

2. The expectation is therefore

1
EX = G0 (1) = ,
p

which is an intuitive result. We expect to wait 1/p throws before a success

appears, if successes are generated with probability p.

Copyright c 2018 D.P. Kroese

44 Random Variables and Probability Distributions

3. By differentiating the PGF twice we find the variance:

2(1 − p) 1 1 1−p
Var(X) = G00 (1) + G0 (1) − (G00 (1))2 = 2
+ − 2 = .
p p p p2

4. The probability of requiring more than k tosses before a success is

P(X > k) = (1 − p)k .

This is obvious from the fact that {X > k} corresponds to the event of k
consecutive failures.

A final property of the geometric distribution which deserves extra attention is

the memoryless property. Think again of the coin toss experiment. Suppose
we have tossed the coin k times without a success (Heads). What is the proba-
bility that we need more than x additional tosses before getting a success. The
answer is, obviously, the same as the probability that we require more than x
tosses if we start from scratch, that is, P(X > x) = (1 − p)x , irrespective of k.
The fact that we have already had k failures does not make the event of getting
a success in the next trial(s) any more likely. In other words, the coin does
not have a memory of what happened, hence the word memoryless property.
Mathematically, it means that for any x, k = 1, 2, . . .,

P(X > k + x | X > k) = P(X > x)

Proof. By the definition of conditional probability

P({X > k + x} ∩ {X > k})

P(X > k + x | X > k) = .
P(X > k)

Now, the event {X > k + x} is a subset of {X > k}, hence their intersection
is {X > k + x}. Moreover, the probabilities of the events {X > k + x} and
{X > k} are (1 − p)k+x and (1 − p)k , respectively, so that

(1 − p)k+x
P(X > k + x | X > k) = = (1 − p)x = P(X > x),
(1 − p)k

as required.

2.5.4 Poisson Distribution

A random variable X for which

λx −λ
P(X = x) = e , x = 0, 1, 2, . . . , (2.6)
x!
(for fixed λ > 0) is said to have a Poisson distribution. We write X ∼ Poi(λ).
The Poisson distribution is used in many probability models and may be viewed
as the “limit” of the Bin(n, µ/n) for large n in the following sense: Consider a

Copyright c 2018 D.P. Kroese

2.5 Some Important Discrete Distributions 45

coin tossing experiment where we toss a coin n times with success probability
λ/n. Let X be the number of successes. Then, as we have seen X ∼ Bin(n, λ/n).
In other words,
   k 
λ n−k

n λ
P(X = k) = 1−
k n n
λk n × n − 1 × · · · × n − k + 1 λ n λ −k
   
= 1− 1−
k! n × n × ··· × n n n

As n → ∞, the second and fourth factors go to 1, and the third factor goes to
e−λ (this is one of the defining properties of the exponential function). Hence,
we have
λk −λ
lim P(X = k) = e ,
n→∞ k!
which shows that the Poisson distribution is a limiting case of the binomial one.
An example of the graph of its pmf is given in Figure 2.8
lambda = 10
0.12

0.1

0.08

0.06

0.04

0.02

0 1 2 3 4 5 6 7 8 9 1011121314151617181920

Figure 2.8: The pdf of the Poi(10)-distribution

We finish with some properties.

1. The PGF was derived in Example 2.8:

G(z) = e−λ(1−z) .

2. It follows that the expectation is EX = G0 (1) = λ. The intuitive explana-

tion is that the mean number of successes of the corresponding coin flip
experiment is np = n(λ/n) = λ.

3. The above argument suggests that the variance should be n(λ/n)(1 −

λ/n) → λ. This is indeed the case, as

Var(X) = G00 (1) + G0 (1) − (G0 (1))2 = λ2 + λ − λ2 = λ .

Thus for the Poisson distribution the variance and expectation are the
same.

Copyright c 2018 D.P. Kroese

46 Random Variables and Probability Distributions

2.5.5 Hypergeometric Distribution

We say that a random variable X has a Hypergeometric distribution with

parameters N , n and r if
r N −r



k n−k
P(X = k) = N

,
n
for max{0, r + n − N } ≤ k ≤ min{n, r}.

We write X ∼ Hyp(n, r, N ). The hypergeometric distribution is used in the

following situation.

Consider an urn with N balls, r of which are red. We draw at random n balls
from the urn without replacement. The number of red balls amongst the n
chosen balls has a Hyp(n, r, N ) distribution. Namely, if we number the red balls
1, . . . , r and the remaining balls r
+1, . . . , N , then the total number of outcomes
of the random experiment is N n , and each of these outcomes is equally likely.
−r
The number of outcomes in the event “k balls are red” is k × N r



n−k because
the k balls have to be drawn from the r red balls, and the remaining n − k balls
have to be drawn from the N − k non-red balls. In table form we have:

Red Not Red Total

Selected k n−k n

Not Selected r−k N −n−r+k N −n

Total r N −r N

Example 2.9 Five cards are selected from a full deck of 52 cards. Let X be
the number of Aces. Then X ∼ Hyp(n = 5, r = 4, N = 52).

P
k 0 1 2 3 4

P(X = k) 0.659 0.299 0.040 0.002 0.000 1

The expectation and variance of the hypergeometric distribution are

r
EX = n
N
and
r  r N −n
Var(X) = n 1− .
N N N −1
Note that this closely resembles the expectation and variance of the binomial
case, with p = r/N . The proofs will be given in Chapter 4 (see Examples 4.7
and 4.10, on pages 73 and 77).

Copyright c 2018 D.P. Kroese

2.6 Some Important Continuous Distributions 47

2.6 Some Important Continuous Distributions

In this section we give a number of important continuous distributions and

list some of their properties. Note that the pdf of each of these distributions
depends on one or more parameters; so, as in the discrete case discussed before,
we are dealing with families of distributions.

2.6.1 Uniform Distribution

We say that a random variable X has a uniform distribution on the interval

[a, b], if it has density function f , given by
1
f (x) = , a≤x≤b.
b−a
We write X ∼ U[a, b]. X can model a randomly chosen point from the inter-
val [a, b], where each choice is equally likely. A graph of the pdf is given in
Figure 2.9.
1
b−a

a b x→

Figure 2.9: The pdf of the uniform distribution on [a, b]

We have
b  2
b − a2

x 1 a+b
Z
EX = dx = = .
a b−a b−a 2 2
This can be seen more directly by observing that the pdf is symmetric around
c = (a+b)/2, and that the expectation is therefore equal to the symmetry point
c. For the variance we have
a+b 2
Z b 2  
x
Var(X) = EX − (EX) =
2 2
dx −
a b−a 2
(a − b)2
= ... = .
12
A more elegant way to derive this is to use the fact that X can be thought of
as the sum X = a + (b − a)U , where U ∼ U[0, 1]. Namely, for x ∈ [a, b]
 
x−a x−a
P(X ≤ x) = =P U ≤ = P(a + (b − a)U ≤ x) .
b−a b−a
Thus, we have Var(X) = Var(a + (b − a)U ) = (b − a)2 Var(U ). And
Z 1  2
1 1 1 1
Var(U ) = EU − (EU ) =
2 2 2
u du − = − = .
0 2 3 4 12

Copyright c 2018 D.P. Kroese

48 Random Variables and Probability Distributions

2.6.2 Exponential Distribution

A random variable X with probability density function f , given by

f (x) = λ e−λ x , x≥0 (2.7)

is said to have an exponential distribution with parameter λ. We write X ∼

Exp(λ). The exponential distribution can be viewed as a continuous version of
the geometric distribution. Graphs of the pdf for various values of λ are given
in Figure 2.10.

1.5 λ=2
f (x)

0.5
λ=1
λ = 0.5

0
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
x

Figure 2.10: The pdf of the Exp(λ)-distribution for various λ.

Here are some properties of the exponential function:

1. The moment generating function is

" #∞
∞ ∞
−e−(λ−s)x
Z Z
sx −λx −(λ−s)x
M (s) = e λe dx = λ e dx = λ
0 0 λ−s
x=0
λ
= ,
λ−s
for s < λ.

2. From the moment generating function we find by differentiation:

λ 1
EX = M 0 (0) = = .
(λ − s)2 s=0 λ

Alternatively, you can use partial integration to evaluate

Z ∞ i∞ Z ∞  −λx ∞
−λx
h
−λx −λx −e 1
EX = x λe dx = −xe + e dx = 0+ = .
0
|{z} | {z } 0 0 λ 0 λ
↓ ↓
1 −e−λx

3. Similarly, the second moment is EX 2 = M 00 (0) = 2λ

(λ−s)3 s=0
= 2/λ2 , so
that the variance becomes
2 1 1
Var(X) = EX 2 − (EX)2 = 2
− 2 = 2 .
λ λ λ

Copyright c 2018 D.P. Kroese

2.6 Some Important Continuous Distributions 49

4. The cdf of X is given by

Z x h ix
F (x) = P(X ≤ x) = λe−λu du = −e−λu = 1 − e−λx , x ≥ 0.
0 0

5. As consequence the tail probability P(X > x) is exponentially decaying:

P(X > x) = e−λx , x≥0.

The most important property of the exponential distribution is the following

Theorem 2.2 (Memoryless Property) Let X have an exponential distri-

bution with parameter λ. Then for any s, t > 0
P(X > s + t | X > s) = P(X > t), for all s, t ≥ 0. (2.8)

Proof. By (1.4)
P(X > s + t, X > s) P(X > s + t)
P(X > s + t | X > s) = =
P(X > s) P(X > s)
e−λ(t+s)
= = e−λt = P(X > t),
e−λs
where in the second equation we have used the fact that the event {X > s + t}
is contained in the event {X > s} hence the intersection of these two sets is
{X > s + t}.

For example, when X denotes the lifetime of a machine, then given the fact
that the machine is alive at time s, the remaining lifetime of the machine, i.e.
X − s, has the same exponential distribution as a completely new machine. In
other words, the machine has no memory of its age and does not “deteriorate”
(although it will break down eventually).

It is not too difficult to prove that the exponential distribution is the only
continuous (positive) distribution with the memoryless property.

2.6.3 Normal, or Gaussian, Distribution

The normal (or Gaussian) distribution is the most important distribution in the
study of statistics. We say that a random variable has a normal distribution
with parameters µ and σ 2 if its density function f is given by
1 1 x−µ 2
f (x) = √ e− 2 ( σ ) , x ∈ R. (2.9)
σ 2π
We write X ∼ N(µ, σ 2 ). The parameters µ and σ 2 turn out to be the expectation
and variance of the distribution, respectively. If µ = 0 and σ = 1 then
1 2
f (x) = √ e−x /2 ,
2π

Copyright c 2018 D.P. Kroese

50 Random Variables and Probability Distributions

and the distribution is known as a standard normal distribution. The cdf of

this latter distribution is often denoted by Φ, and is tabulated in Appendix B.
In Figure 2.11 the probability densities for three different normal distributions
have been depicted.

0.8

0.7

0.6 N(0,0.25)

0.5
f (x)

0.4
N(0,1) N(2,1)
0.3

0.2

0.1

0
−3 −2 −1 0 1 2 3 4 5
x

Figure 2.11: Probability density functions for various normal distributions

We next consider some important properties of the normal distribution.

1. If X ∼ N(µ, σ 2 ), then
X −µ
∼ N(0, 1) . (2.10)
σ
Thus by subtracting the mean and dividing by the standard deviation
we obtain a standard normal distribution. This procedure is called stan-
dardisation.
Proof. Let X ∼ N(µ, σ 2 ), and Z = (X − µ)/σ. Then,

P(Z ≤ z) = P((X − µ)/σ ≤ z) = P(X ≤ µ + σz)

Z µ+σz
1 1 x−µ 2
= √ e− 2 ( σ ) dx
σ 2π
Z−∞
z
1 −y2 /2
= √ e dy [c.o.v. y = x−µ
σ ]
−∞ 2π
= Φ(z) .

Thus Z has a standard normal distribution.

Standardisation enables us to express the cdf of any normal distribution
in terms of the cdf of the standard normal distribution. This is the reason
why only the table for the standard normal distribution is included in the
appendix.

Copyright c 2018 D.P. Kroese

2.6 Some Important Continuous Distributions 51

2. A trivial rewriting of the standardisation formula gives the following im-

portant result: If X ∼ N(µ, σ 2 ), then

X = µ + σZ, with Z ∼ N(0, 1) .

In other words, any Gaussian (normal) random variable can be viewed as

a so-called affine (linear + constant) transformation of a standard normal
random variable.

3. EX = µ. This is because the pdf is symmetric around µ.

4. Var(X) = σ 2 . This is a bit more involved. First, write X = µ + σZ, with

Z standard normal. Then, Var(X) = Var(µ + σZ) = σ 2 Var(Z). Hence,
it suffices to show that the variance of Z is 1. Consider Var(Z) = EZ 2
(note that the expectation is 0). We have
Z ∞ Z ∞
2 1 −z 2 /2 z 2
EZ =
2
z √ e dz = z × √ e−z /2 dz
−∞ 2π −∞ 2π
By writing the last integrand in this way we can apply partial integration
to the two factors to yield

−1 −z 2 /2 ∞
  Z ∞
1 2
EZ = z √ e
2
+ √ e−z /2 dz = 1 ,
2π −∞ −∞ 2π

since the last integrand is the pdf of the standard normal distribution.

5. The moment generating function of X ∼ N(µ, σ 2 ) is given by

2 σ 2 /2
EesX = esµ+s , s ∈ R. (2.11)

Proof. First consider the moment generation function of Z ∼ N(0, 1).

We have
Z ∞ Z ∞
1 2 2 1 2
EesZ = esz √ e−z /2 dz = es /2 √ e−(z−s) /2 dz
−∞ 2π −∞ 2π
2 /2
= es ,

where the second integrand is the pdf of the N(s, 1) distribution, which
therefore integrates to 1. Now, for general X ∼ N(µ, σ 2 ) write X =
µ + σZ. Then,
2 s2 /2 2 s2 /2
EesX = Ees(µ+σZ) = esµ EesσZ = esµ eσ = esµ+σ .

More on the Gaussian distribution later, especially the multidimensional cases!

Copyright c 2018 D.P. Kroese

52 Random Variables and Probability Distributions

2.6.4 Gamma- and χ2 -distribution

The gamma distribution arises frequently in statistics. Its density function is

given by
λα xα−1 e−λx
f (x) = , x ≥ 0, (2.12)
Γ(α)
where Γ is the Gamma-function defined as
Z ∞
Γ(α) = uα−1 e−u du, α > 0.
0
Parameter α is called the shape parameter, and λ is called the scale parameter.
We write X ∼ Gam(α, λ).

Of particular importance is following special case: A random variable X is said

to have a chi-square distribution with n (∈ {1, 2, . . .}) degrees of freedom
if X ∼ Gam(n/2, 1/2). We write X ∼ χ2n . A graph of the pdf of the χ2n -
distribution, for various n is given in Figure 2.12.

0.5
n=1

0.4
n=2
0.3 n=3
n=4
0.2

0.1

2 4 6 8 10

Figure 2.12: Pdfs for the χ2n -distribution, for various degrees of freedom n

We mention a few properties of the Γ-function.

1. Γ(a + 1) = a Γ(a), for a ∈ R+ .

2. Γ(n) = (n − 1)! for n = 1, 2, . . ..
√
3. Γ(1/2) = π.

The moment generating function of X ∼ Gam(α, λ) is given by

Z ∞ −λx α α−1
e λ x
M (s) = E esX
= esx dx
0 Γ(α)
α Z ∞ −(λ−s)x
(λ − s)α xα−1

λ e
= dx
λ−s 0 Γ(α)
 α
λ
= . (2.13)
λ−s

Copyright c 2018 D.P. Kroese

2.6 Some Important Continuous Distributions 53

As a consequence, we have
 α+1
α λ α
EX = M 0 (0) = = ,
λ λ−s s=0 λ

and, similarly,
α
Var(X) = .
λ2

Copyright c 2018 D.P. Kroese

54 Random Variables and Probability Distributions

Copyright c 2018 D.P. Kroese

Chapter 3

Generating Random Variables

on a Computer

3.1 Introduction

This chapter deals with the execution of random experiments via the computer,
also called stochastic simulation. In a typical stochastic simulation, random-
ness is introduced into simulation models via independent uniformly distributed
random variables, also called random numbers. These random numbers are
then used as building blocks to simulate more general stochastic systems.

3.2 Random Number Generation

In the early days of simulation, randomness was generated by manual tech-

niques, such as coin flipping, dice rolling, card shuffling, and roulette spinning.
Later on, physical devices, such as noise diodes and Geiger counters, were at-
tached to computers for the same purpose. The prevailing belief held that only
mechanical or electronic devices could produce “truly” random sequences. Al-
though mechanical devices are still widely used in gambling and lotteries, these
methods were abandoned by the computer-simulation community for several
reasons: (a) Mechanical methods were too slow for general use, (b) the gen-
erated sequences cannot be reproduced and, (c) it has been found that the
generated numbers exhibit both bias and dependence. Although certain mod-
ern physical generation methods are fast and would pass most statistical tests
for randomness (for example, those based on the universal background radia-
tion or the noise of a PC chip), their main drawback remains to be their lack
of repeatability. Most of today’s random number generators are not based on
physical devices, but on simple algorithms that can be easily implemented on
a computer, are fast, require little storage space, and can readily reproduce a
given sequence of random numbers. Importantly, a good random number gener-

Copyright c 2018 D.P. Kroese

56 Generating Random Variables on a Computer

ator captures all the important statistical properties of true random sequences,
even though the sequence is generated by a deterministic algorithm. For this
reason, these generators are sometimes called pseudorandom.

The most common methods for generating pseudorandom sequences use the so-
called linear congruential generators. These generate a deterministic sequence
of numbers by means of the recursive formula

Xi+1 = aXi + c (mod m) , (3.1)

where the initial value, X0 , is called the seed, and the a, c and m (all positive
integers) are called the multiplier, the increment and the modulus, respectively.
Note that applying the modulo-m operator in (3.1) means that aXi +c is divided
by m, and the remainder is taken as the value of Xi+1 . Thus, each Xi can only
assume a value from the set {0, 1, . . . , m − 1}, and the quantities

Xi
Ui = , (3.2)
m
called pseudorandom numbers, constitute approximations to the true sequence
of uniform random variables. Note that the sequence {Xi } will repeat itself in
at most m steps, and will therefore be periodic with period not exceeding m.
For example, let a = c = X0 = 3 and m = 5; then the sequence obtained from
the recursive formula Xi+1 = 3Xi + 3 (mod 5) is Xi = 3, 2, 4, 0, 3, which has
period 4, while m = 5. In the special case where c = 0, (3.1) simply reduces to

Xi+1 = a Xi (mod m) . (3.3)

Such a generator is called a multiplicative congruential generator. It is readily

seen that an arbitrary choice of X0 , a, c and m will not lead to a pseudorandom
sequence with good statistical properties. In fact, number theory has been used
to show that only a few combinations of these produce satisfactory results. In
computer implementations, m is selected as a large prime number that can be
accommodated by the computer word size. For example, in a binary 32-bit
word computer, statistically acceptable generators can be obtained by choosing
m = 231 − 1 and a = 75 , provided the first bit is a sign bit. A 64-bit or 128-bit
word computer computer will naturally yield better statistical results.

Most computer languages already contain a built-in pseudorandom number gen-

erator. The user is typically requested only to input the initial seed, X0 , and
upon invocation, the random number generator produces a sequence of indepen-
dent uniform (0, 1) random variables. We, therefore assume in this chapter the
availability of a “black box”, capable of producing a stream of pseudorandom
numbers. In Matlab, for example, this is provided by the rand function.

Example 3.1 (Generating uniform random variables in Matlab) This ex-

ample illustrates the use of the rand function in Matlab, to generate samples
from the U(0, 1) distribution. For clarity we have omitted the “ans = ” output
in the Matlab session below.

Copyright c 2018 D.P. Kroese

3.3 The Inverse-Transform Method 57

>> rand % generate a uniform random number

0.0196
>> rand % generate another uniform random number
0.823
>> rand(1,4) % generate a uniform random vector
0.5252 0.2026 0.6721 0.8381
>> rng(1234) % set the seed to 1234
>> rand % generate a uniform random number
0.6104
>> rng(1234) % reset the seed to 1234
>> rand
0.6104 % the previous outcome is repeated

3.3 The Inverse-Transform Method

In this section we discuss a general method for generating one-dimensional

random variables from a prescribed distribution, namely the inverse-transform
method.

Let X be a random variable with cdf F . Since F is a nondecreasing function,

the inverse function F −1 may be defined as

F −1 (y) = inf{x : F (x) ≥ y} , 0 ≤ y ≤ 1 . (3.4)

(Readers not acquainted with the notion inf should read min.) It is easy to
show that if U ∼ U(0, 1), then

X = F −1 (U ) (3.5)

has cdf F. Namely, since F is invertible and P(U ≤ u) = u, we have

P(X ≤ x) = P(F −1 (U ) ≤ x) = P(U ≤ F (x)) = F (x) . (3.6)

Thus, to generate a random variable X with cdf F , draw U ∼ U(0, 1) and set
X = F −1 (U ). Figure 3.1 illustrates the inverse-transform method given by the
following algorithm.

Algorithm 3.1 (The Inverse-Transform Method)

1. Generate U from U(0, 1).

2. Return X = F −1 (U ).

Copyright c 2018 D.P. Kroese

58 Generating Random Variables on a Computer

F (x)

x
0 X

Figure 3.1: The inverse-transform method.

Example 3.2 Generate a random variable from the pdf


 2x, 0 ≤ x ≤ 1
f (x) = (3.7)
0, otherwise.


The cdf is 

 0, x<0


 R
x 2
F (x) =
 0 2x dx = x , 0 ≤ x ≤ 1



1, x > 1.


Applying (3.5), we have

√
X = F −1 (U ) = U, 0 ≤ u ≤ 1 .
Therefore, to generate a random variable X from the pdf (3.7), first generate a
random variable U from U(0, 1), and then take its square root.

Example 3.3 (Drawing From a Discrete Distribution) Let P X be a dis-

crete random variable with P(XP= xi ) = pi , i = 1, 2, . . . , with i pi = 1. The
cdf F of X is given by F (x) = i:xi ≤x pi , i = 1, 2, . . .; see Figure 3.2.

F (x)

1 p5

p4
U
p3 f
p2 f

p1
x
0 X

Figure 3.2: The inverse-transform method for a discrete random variable.

Copyright c 2018 D.P. Kroese
3.4 Generating From Commonly Used Distributions 59

The algorithm for generating a random variable from F can thus be written as
follows:

Algorithm 3.2 (The Inverse-Transform Method for a Discrete Distri-

bution)

1. Generate U ∼ U(0, 1).

2. Find the smallest positive integer, k, such that U ≤ F (xk ) and return
X = xk .

In matlab drawing from a probability vector (p1 , . . . , pn ) can be done in one

line:

min(find(cumsum(p)> rand));

Here p is the vector of probabilities, such as (1/3,1/3,1/3), cumsum gives the

cumulative vector, e.g., (1/3, 2/3, 1), find finds the indices i such that pi > r,
where r is some random number, and min takes the smallest of these indices.

Much of the execution time in Algorithm 3.2 is spent in making the comparisons
of Step 2. This time can be reduced by using efficient search techniques.

In general, the inverse-transform method requires that the underlying cdf, F ,

exist in a form for which the corresponding inverse function F −1 can be found
analytically or algorithmically. Applicable distributions are, for example, the
exponential, uniform, Weibull, logistic, and Cauchy distributions. Unfortu-
nately, for many other probability distributions, it is either impossible or diffi-
cult to find the inverse transform, that is, to solve
Z x
F (x) = f (t) dt = u
−∞

with respect to x. Even in the case where F −1 exists in an explicit form, the
inverse-transform method may not necessarily be the most efficient random
variable generation method.

3.4 Generating From Commonly Used Distributions

The next two subsections present algorithms for generating variables from com-
monly used continuous and discrete distributions. Of the numerous algorithms
available we have tried to select those which are reasonably efficient and rela-
tively simple to implement.

Copyright c 2018 D.P. Kroese

60 Generating Random Variables on a Computer

Exponential Distribution

We start by applying the inverse-transform method to the exponential distri-

bution. If X ∼ Exp(λ), then its cdf F is given by

F (x) = 1 − e−λx , x>0. (3.8)

Hence, solving u = F (x) in terms of x gives

1
F −1 (u) = − log(1 − u) .
λ
Noting that U ∼ U(0, 1) implies 1 − U ∼ U(0, 1), we obtain the following
algorithm.

Algorithm 3.3 (Generation of an Exponential Random Variable)

1. Generate U ∼ U(0, 1).

2. Return X = − λ1 ln U as a random variable from Exp(λ).

There are many alternative procedures for generating variables from the expo-
nential distribution. The interested reader is referred to Luc Devroye’s book
Non-Uniform Random Variate Generation, Springer-Verlag, 1986. (The entire
book can be downloaded for free.)

Normal (Gaussian) Distribution

If X ∼ N(µ, σ 2 ), its pdf is given by

(x − µ)2
 
1
f (x) = √ exp − , −∞ < x < ∞ , (3.9)
σ 2π 2σ 2

where µ is the mean (or expectation) and σ 2 the variance of the distribution.

Since inversion of the normal cdf is numerically inefficient, the inverse-transform

method is not very suitable for generating normal random variables, and some
other procedures must be devised instead. We consider only generation from
N(0, 1) (standard normal variables), since any random Z ∼ N(µ, σ 2 ) can be
represented as Z = µ+σX, where X is from N(0, 1). One of the earliest methods
for generating variables from N(0, 1) is the following, due to Box and Müller.
A justification of this method will be given in Chapter 5, see Example 5.6.

Algorithm 3.4 (Generation of a Normal Random Variable, Box and

Müller Approach)

1. Generate two independent random variables, U1 and U2 , from U(0, 1).

Copyright c 2018 D.P. Kroese

3.4 Generating From Commonly Used Distributions 61

2. Return two independent standard normal variables, X and Y , via

X = (−2 ln U1 )1/2 cos(2πU2 ) ,

(3.10)
Y = (−2 ln U1 )1/2 sin(2πU2 ) .

An alternative generation method for N(0, 1) is based on the acceptance–rejection

method. First, note that in order to generate a random variable Y from N(0, 1),
one can first generate a nonnegative random variable X from the pdf
r
2 −x2 /2
f (x) = e , x ≥ 0, (3.11)
π
and then assign to X a random sign. The validity of this procedure follows
from the symmetry of the standard normal distribution about zero.

To generate a random variable X from (3.11), we bound f (x) by C g(x), where

g(x) = e−x
p is the pdf of the Exp(1). The smallest constant C such that
p f (x) ≤
Cg(x) is 2e/π, see Figure 3.3. The efficiency of this method is thus π/2e ≈
0.76.

1.2

0.8

0.6

0.4

0.2

1 2 3 4

Figure 3.3: Bounding the positive normal density

Bernoulli distribution

If X ∼ Ber(p), its pmf is of the form

f (x) = px (1 − p)1−x , x = 0, 1, (3.12)

where p is the success probability. Applying the inverse-transform method, one

readily obtains the following generation algorithm:

Algorithm 3.5 (Generation of a Bernoulli Random Variable)

1. Generate U ∼ U(0, 1).

2. If U ≤ p, return X = 1; otherwise return X = 0.

Copyright c 2018 D.P. Kroese

62 Generating Random Variables on a Computer

In Figure 1.1 on page 6 typical outcomes are given of 100 independent Bernoulli
random variables, each with success parameter p = 0.5.

Binomial distribution

If X ∼ Bin(n, p) then its pmf is of the form

 
n x
f (x) = p (1 − p)n−x , x = 0, 1, . . . , n . (3.13)
x

Recall that a binomial random variable X can be viewed as the total number of
successes in n independent Bernoulli experiments, each with success probability
p; see Example 1.15. Denoting the result of the i-th trial by Xi = 1 (success)
or Xi = 0 (failure), we can write X = X1 + · · · + Xn with the {Xi } iid Ber(p)
random variables. The simplest generation algorithm can thus be written as
follows:

Algorithm 3.6 (Generation of a Binomial Random Variable)

1. Generate iid random variables X1 , . . . , Xn from Ber(p).

2. Return X = ni=1 Xi as a random variable from Bin(n, p).

P

It is worthwhile to note that if Y ∼ Bin(n, p), then n−Y ∼ Bin(n, 1−p). Hence,
to enhance efficiency, one may elect to generate X from Bin(n, p) according to

 Y1 ∼ Bin(n, p), if p ≤ 12


X=
Y2 ∼ Bin(n, 1 − p), if p > 21 .


Geometric Distribution

If X ∼ G(p), then its pmf is of the form

f (x) = p (1 − p)x−1 , x = 1, 2 . . . . (3.14)

The random variable X can be interpreted as the the number of trials required
until the first success occurs, in a series of independent Bernoulli trials with
success parameter p. Note that P(X > m) = (1 − p)m .

We now present an algorithm which is based on the relationship between the

exponential and geometric distributions. Let Y ∼ Exp(λ), with λ such that
q = 1 − p = e−λ . Then, X = bY c + 1 has a G(p) distribution — here b c denotes
the integer part. Namely,

P(X > x) = P(bY c > x − 1) = P(Y > x) = e−λx = (1 − p)x .

Copyright c 2018 D.P. Kroese

3.4 Generating From Commonly Used Distributions 63

Hence, to generate a random variable from G(p), we first generate a random

variable from the exponential distribution with λ = − ln(1 − p), truncate the
obtained value to the nearest integer and add 1.

Algorithm 3.7 (Generation of a Geometric Random Variable)

1. Generate Y ∼ Exp(− ln(1 − p))

2. Return X = 1 + bY c as a random variable from G(p).

Copyright c 2018 D.P. Kroese

64 Generating Random Variables on a Computer

Copyright c 2018 D.P. Kroese

Chapter 4

Joint Distributions

Often a random experiment is described via more than one random variable.
Examples are:

1. We select a random sample of n = 10 people and observe their lengths.

Let X1 , . . . , Xn be the individual lengths.

2. We flip a coin repeatedly. Let Xi = 1 if the ith flip is “heads” and 0 else.
The experiment is described by the sequence X1 , X2 , . . . of coin flips.

3. We randomly select a person from a large population and measure his/her

weight X and height Y .

How can we specify the behaviour of the random variables above? We should
not just specify the pdf or pmf of the individual random variables, but also
say something about the “interaction” (or lack thereof) between the random
variables. For example, in the third experiment above if the height Y is large,
we expect that X is large as well. On the other hand, for the first and second
experiment it is reasonable to assume that information about one of the random
variables does not give extra information about the others. What we need to
specify is the joint distribution of the random variables.

The theory for multiple random variables is quite similar to that of a single
random variable. The most important extra feature is perhaps the concept of
independence of random variables. Independent random variables play a crucial
role in stochastic modelling.

4.1 Joint Distribution and Independence

Let X1 , . . . , Xn be random variables describing some random experiment. We

can accumulate the Xi ’s into a row vector X = (X1 , . . . , Xn ) or column vector

Copyright c 2018 D.P. Kroese

66 Joint Distributions

X = (X1 , . . . , Xn )T (here T means transposition). X is called a random

vector.

Recall that the distribution of a single random variable X is completely specified

by its cumulative distribution function. Analogously, the joint distribution of
X1 , . . . , Xn is specified by the joint cumulative distribution function F ,
defined by

F (x1 , . . . , xn ) = P({X1 ≤ x1 } ∩ · · · ∩ {Xn ≤ xn }) = P(X1 ≤ x1 , . . . , Xn ≤ xn ),

If we know F then we can in principle derive any probability involving the Xi ’s.
Note the abbreviation on the right-hand side. We will henceforth use this kind
of abbreviation throughout the notes.

Similar to the 1-dimensional case we distinguish between the case where the Xi
are discrete and continuous. The corresponding joint distributions are again
called discrete and continuous, respectively.

4.1.1 Discrete Joint Distributions

To see how things work in the discrete case, let’s start with an example.

Example 4.1 In a box are three dice. Die 1 is a normal die; die 2 has no 6
face, but instead two 5 faces; die 3 has no 5 face, but instead two 6 faces. The
experiment consists of selecting a die at random, followed by a toss with that
die. Let X be the die number that is selected, and let Y be the face value of
that die. The probabilities P(X = x, Y = y) are specified below.

y
P
x 1 2 3 4 5 6
1 1 1 1 1 1 1
1
18 18 18 18 18 18 3
1 1 1 1 1 1
2 0
18 18 18 18 9 3
1 1 1 1 1 1
3 0
18 18 18 18 9 3
P 1 1 1 1 1 1
1
6 6 6 6 6 6

The function f : (x, y) 7→ P(X = x, Y = y) is called the joint pmf of X and Y .

The following definition is just a generalisation of this.

Definition 4.1 Let X1 , . . . , Xn be discrete random variables. The function

f defined by f (x1 , . . . , xn ) = P(X1 = x1 , . . . , Xn = xn ) is called the joint
probability mass function (pmf) of X1 , . . . , Xn .

Copyright c 2018 D.P. Kroese

4.1 Joint Distribution and Independence 67

We sometimes write fX1 ,...,Xn instead of f to show that this is the pmf of the
random variables X1 , . . . , Xn . Or, if X is the corresponding random vector, we
could write fX instead.

Note that, by the sum rule, if we are given the joint pmf of X1 , . . . , Xn we can
in principle calculate all possible probabilities involving these random variables.
For example, in the 2-dimensional case
X
P((X, Y ) ∈ B) = P(X = x, Y = y) ,
(x,y)∈B

for any subset B of possible values for (X, Y ). In particular, we can find the
pmf of X by summing the joint pmf over all possible values of y:
X
P(X = x) = P(X = x, Y = y) .
y

The converse is not true: from the individual distributions (so-called marginal
distribution) of X and Y we cannot in general reconstruct the joint distribution
of X and Y . We are missing the “dependency” information. E.g., in Exam-
ple 4.1 we cannot reconstruct the inside of the two-dimensional table if only
given the column and row totals.

However, there is one important exception to this, namely when we are dealing
with independent random variables. We have so far only defined what inde-
pendence is for events. The following definition says that random variables
X1 , . . . , Xn are independent if the events {X1 ∈ A1 }, . . . , {Xn ∈ An } are in-
dependent for any subsets A1 , . . . , An of R. Intuitively, this means that any
information about one of them does not affect our knowledge about the others.

Definition 4.2 The random variables X1 , . . . , Xn are called independent if

for all A1 , . . . , An , with Ai ⊂ R, i = 1, . . . , n
P(X1 ∈ A1 , . . . , Xn ∈ An ) = P(X1 ∈ A1 ) · · · P(Xn ∈ An ) .

The following theorem is a direct consequence of the definition above.

Theorem 4.1 Discrete random variables X1 , . . . , Xn , are independent if and

only if

P(X1 = x1 , . . . , Xn = xn ) = P(X1 = x1 ) · · · P(Xn = xn ), (4.1)

for all x1 , x2 , . . . , xn .

Proof. The necessary condition is obvious: if X1 , . . . , Xn are independent

random variables, then {X1 = x1 }, . . . , {Xn = xn } are (mutually) independent
events. To prove the sufficient condition, write
X X
P(X1 ∈ A1 , . . . , Xn ∈ An ) = ··· P(X1 = x1 , . . . , Xn ∈ xn ).
x1 ∈A1 xn ∈An

Copyright c 2018 D.P. Kroese

68 Joint Distributions

Then, if (4.1) holds, the multiple sum can be written as

X X
P(X1 = x1 ) · · · P(Xn ∈ xn ) = P(X1 ∈ A1 ) · · · P(Xn ∈ An ),
x1 ∈A1 xn ∈An

which implies that X1 , . . . , Xn are independent random variables.

Example 4.2 We repeat the experiment in Example 4.1 with three ordinary
fair dice. What are now the joint probabilities in the table? Since the events
{X = x} and {Y = y} are now independent, each entry in the pmf table is
1 1
3 × 6 . Clearly in the first experiment not all events {X = x} and {Y = y} are
independent (why not?).

Example 4.3 (Coin Flip Experiment) Consider the experiment where we

flip a coin n times. We can model this experiments in the following way. For
i = 1, . . . , n let Xi be the result of the ith toss: {Xi = 1} means Heads,
{Xi = 0} means Tails. Also, let
P(Xi = 1) = p = 1 − P(Xi = 0), i = 1, 2, . . . , n .
Thus, p can be interpreted as the probability of Heads, which may be known
or unknown. Finally, assume that X1 , . . . , Xn are independent.

This completely specifies our model. In particular we can find any probability
related to the Xi ’s. For example, let X = X1 + · · · + Xn be the total number of
Heads in n tosses. Obviously X is a random variable that takes values between
0Pand n. Denote by A the set of
all binary vectors x = (x1 , . . . , xn ) such that
n n
i=1 xi = k. Note that A has k elements. We now have
X
P(X = k) = P(X1 = x1 , . . . , Xn = xn )
x∈A
X X
= P(X1 = x1 ) · · · P(Xn = xn ) = pk (1 − p)n−k
x∈A x∈A
 
n k
= p (1 − p)n−k .
k
In other words, X ∼ Bin(n, p). Compare this to what we did in Example 1.15
on page 27.

Remark 4.1 If fX1 ,...,Xn denotes the joint pmf of X1 , . . . , Xn and fXi the
marginal pmf of Xi , i = 1, . . . , n, then the theorem above states that inde-
pendence of the Xi ’s is equivalent to
fX1 ,...,Xn (x1 , . . . , xn ) = fX1 (x1 ) · · · fXn (xn )
for all possible x1 , . . . , xn .

Remark 4.2 An infinite sequence X1 , X2 , . . . of random variables is called in-

dependent if for any finite choice of parameters i1 , i2 , . . . , in (none of them the
same) the random variables Xi1 , . . . , Xin are independent.

Copyright c 2018 D.P. Kroese

4.1 Joint Distribution and Independence 69

Multinomial Distribution

An important discrete joint distribution is the multinomial distribution. It can

be viewed as a generalisation of the binomial distribution. First we give the
definition, then an example how this distribution arises in applications.

Definition 4.3 We say that (X1 , X2 , . . . , Xk ) has a multinomial distribution,

with parameters n and p1 , p2 , . . . , pk , if
n!
P(X1 = x1 , . . . , Xk = xk ) = px1 px2 · · · pxk k , (4.2)
x1 ! x2 ! · · · xk ! 1 2
for all x1 , . . . , xk ∈ {0, 1, . . . , n} such that x1 + x2 + · · · + xk = n. We write
(X1 , . . . , Xk ) ∼ Mnom(n, p1 , . . . , pk ).

Example 4.4 We independently throw n balls into k urns, such that each ball
is thrown in urn i with probability pi , i = 1, . . . , k.

probab. p1 p p3 pk
2

1 2 3 k
Let Xi be the total number of balls in urn i, i = 1, . . . , k. We show that
(X1 , . . . , Xk ) ∼ Mnom(n, p1 , . . . , pk ). Let x1 , . . . , xk be integers between 0 and
n that sum up to n. The probability that the first x1 balls fall in the first urn,
the next x2 balls fall in the second urn, etcetera, is
px1 1 px2 2 · · · pxk k .
To find the probability that there are x1 balls in the first urn, x2 in the second,
etcetera, we have to multiply the probability above with the number of ways in
which we can fill the urns with x1 , x2 , . . . , xk balls, i.e. n!/(x1 ! x2 ! · · · xk !). This
gives (4.2).

Remark 4.3 Note that for the binomial distribution there are only two possible
urns. Also, note that for each i = 1, . . . , k, Xi ∼ Bin(n, pi ).

4.1.2 Continuous Joint Distributions

Joint distributions for continuous random variables are usually defined via
the joint pdf. The results are very similar to the discrete case discussed in

Copyright c 2018 D.P. Kroese

70 Joint Distributions

Section 4.1.1. Compare this section also with the 1-dimensional case in Sec-
tion 2.2.2.

Definition 4.4 We say that the continuous random variables X1 , . . . , Xn have

a joint probability density function (pdf) f if
Z b1 Z bn
P(a1 ≤ X1 ≤ b1 , . . . , an ≤ Xn ≤ bn ) = ··· f (x1 , . . . , xn ) dx1 · · · dxn
a1 an

for all a1 , . . . , bn .

We sometimes write fX1 ,...,Xn instead of f to show that this is the pdf of the
random variables X1 , . . . , Xn . Or, if X is the corresponding random vector, we
could write fX instead.

We can interpret f (x1 , . . . , xn ) as a continuous analogue of a pmf, or as the

“density” that X1 = x1 , X2 = x2 , . . . , and Xn = xn . For example in the
2-dimensional case:
P(x ≤ X ≤ x + h, y ≤ Y ≤ y + h)
Z x+h Z y+h
= f (u, v) du dv ≈ h2 f (x, y) .
x y

Note that if the joint pdf is given, then in principle we can calculate all proba-
bilities. Specifically, in the 2-dimensional case we have
Z Z
P((X, Y ) ∈ B) = f (x, y) dx dy , (4.3)
(x,y)∈B

for any subset B of possible values for R2 . Thus, the calculation of probabilities
is reduced to integration.

Similarly to the discrete case, if X1 , . . . , Xn have joint pdf f , then the (individ-
ual, or marginal) pdf of each Xi can be found by integrating f over all other
variables. For example, in the two-dimensional case
Z ∞
fX (x) = f (x, y) dy.
y=−∞

However, we usually cannot reconstruct the joint pdf from the marginal pdf’s
unless we assume that the the random variables are independent. The defi-
nition of independence is exactly the same as for discrete random variables,
see Definition 4.2. But, more importantly, we have the following analogue of
Theorem 4.1.

Theorem 4.2 Let X1 , . . . , Xn be continuous random variables with joint pdf

f and marginal pdf’s fX1 , . . . , fXn . The random variables X1 , . . . , Xn are inde-
pendent if and only if, for all x1 , . . . , xn ,

f (x1 , . . . , xn ) = fX1 (x1 ) · · · fXn (xn ). (4.4)

Copyright c 2018 D.P. Kroese

4.1 Joint Distribution and Independence 71

Example 4.5 Consider the experiment were we select randomly and indepen-
dently n points from the interval [0,1]. We can carry this experiment out using
a calculator or computer, using the random generator. On your calculator this
means pushing the RAN# or Rand button. Here is a possible outcome, or
realisation, of the experiment, for n = 12.
0.9451226800 0.2920864820 0.0019900900 0.8842189383 0.8096459523
0.3503489150 0.9660027079 0.1024852543 0.7511286891 0.9528386400
0.2923353821 0.0837952423
A model for this experiment is: Let X1 , . . . , Xn be independent random vari-
ables, each with a uniform distribution on [0,1]. The joint pdf of X1 , . . . , Xn is
very simple, namely
f (x1 , . . . , xn ) = 1, 0 ≤ x1 ≤ 1, . . . , 0 ≤ xn ≤ 1 ,
(and 0 else). In principle we can now calculate any probability involving the
Xi ’s. For example for the case n = 2 what is the probability
X1 + X22
 
P 2
> sin(X1 − X2 ) ?
X1 X2
The answer, by (4.3), is
ZZ
1 dx1 dx2 = Area (A),
A
where
x1 + x22
 
2 2
A = (x1 , x2 ) ∈ [0, 1] : > sin(x1 − x2 ) .
x1 x2
(Here [0, 1]2 is the unit square in R2 ).

Remark 4.4 The type of model used in the previous example, i.e., X1 , . . . , Xn
are independent and all have the same distribution, is the most widely used
model in statistics. We say that X1 , . . . , Xn is a random sample of size n,
from some given distribution. In Example 4.5 X1 , . . . , Xn is a random sample
from a U[0, 1]-distribution. In Example 4.3 we also had a random sample, this
time from a Ber(p)-distribution. The common distribution of a random sample
is sometimes called the sampling distribution.

Using the computer we can generate the outcomes of random samples from
many (sampling) distributions. In Figure 4.1 the outcomes of a two random
samples, both of size 1000 are depicted in a histogram. Here the x-axis is
divived into 20 intervals, and the number of points in each interval is counted.
The first sample is from the U[0, 1]-distribution, and the second sample is from
the N(1/2, 1/12)-distribution. The matlab commands are:

figure(1)
hist(rand(1,1000),20)
figure(2)
hist(1/2 + randn(1,1000)*sqrt(1/12),20)

Copyright c 2018 D.P. Kroese

72 Joint Distributions

Note that the true expectation and variance of the distributions are the same.
However, the “density” of points in the two samples is clearly different, and
follows that shape of the corresponding pdf’s.

70

60

50

40

30

20

10

0
0 0.2 0.4 0.6 0.8 1

140

120

100

80

60

40

20

0
−0.4 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4

Figure 4.1: A histogram of a random sample of size 100 from the U[0, 1]-
distribution (above) and the N(1/2, 1/12)-distribution (below).

4.2 Expectation

Similar to the 1-dimensional case, the expected value of any real-valued function
of X1 , . . . , Xn is the weighted average of all values that this function can take.

Copyright c 2018 D.P. Kroese

4.2 Expectation 73

Specifically, if Z = g(X1 , . . . , Xn ) then in the discrete case

X X
EZ = ··· g(x1 , . . . , xn ) f (x1 , . . . , xn ) ,
x1 xn

where f is the joint pmf; and in the continuous case

Z ∞ Z ∞
EZ = ··· g(x1 , . . . , xn )f (x1 , . . . , xn ) dx1 . . . dxn ,
−∞ −∞

where f is the joint pdf.

Example 4.6 Let X and Y be continuous, possibly dependent, r.v.’s with joint
pdf f . Then,
Z ∞Z ∞
E(X + Y ) = (x + y)f (x, y) dxdy
−∞ −∞
Z ∞Z ∞ Z ∞Z ∞
= xf (x, y) dxdy + yf (x, y) dxdy
−∞ −∞ −∞ −∞
Z ∞ Z ∞
= xfX (x) dx + yfY (y) dy
−∞ −∞
= EX + EY .

The previous example is easily generalised to the following result:

Theorem 4.3 Suppose X1 , X2 , . . . , Xn are discrete or continuous random

variables with means µ1 , µ2 , . . . , µn . Let

Y = a + b1 X1 + b2 X2 + · · · + bn Xn

where a, b1 , b2 , . . . , bn are constants. Then

EY = a + b1 EX1 + · · · + bn EXn
= a + b1 µ1 + · · · + bn µn

Example 4.7 (Expectation of Bin(n, p) and Hyp(n, r, N ) ) We can now prove

that the expectation of a Bin(n, p) random variable is np, without having to
resort to difficult arguments. Namely, if X ∼ Bin(n, p), then X can be written
as the sum X1 +· · ·+Xn of iid Ber(p) random variables, see Example 4.3. Thus,

EX = E(X1 + · · · + Xn ) = EX1 + · · · + EXn = p + · · · p = np , (4.5)

| {z }
n times

because the expectation of each Xi is p.

Notice that we do not use anywhere the independence of the {Xi }. Now, let
X ∼ Hyp(n, r, N ), and let p = r/N . We can think of X as the total number of

Copyright c 2018 D.P. Kroese

74 Joint Distributions

red balls when n balls are drawn from an urn with r red balls and N − r other
balls. Without loss of generality we may assume the balls are drawn one-by-one.
Let Xi = 1 if the i-th balls is red, and 0 otherwise. Then, again X1 + · · · + Xn ,
and each Xi ∼ Ber(p), but now the {Xi } are dependent. However, this does not
affect the result (4.5), so that the expectation of X is np = nr/N .

Another important result is the following.

Theorem 4.4 If X1 , . . . , Xn are independent, then

EX1 X2 · · · Xn = EX1 EX2 · · · EXn .

Proof. We prove it only for the 2-dimensional continuous case. Let f denote
the joint pdf of X and Y , and fX and fY the marginals. Then, f (x, y) =
fX (x)fY (y) for all x, y. Thus
ZZ ZZ
EXY = x y f (x, y) dx dy = x y fX (x) fY (y) dx dy
Z Z
= xfX (x)dx fY (y)dy = EXEY .

The generalization to the n-dimensional continuous/discrete case is obvious.

Theorem 4.4 is particularly handy in combination with transform techniques.

We give two examples.

Example 4.8 Let X ∼ Poi(λ), then we saw in Example 2.8 on page 38 that
its PGF is given by
G(z) = e−λ(1−z) . (4.6)
Now let Y ∼ Poi(µ) be independent of X. Then, the PGF of X + Y is given by

E[z X+Y ] = E[z X ] E[z Y ] = e−λ(1−z) e−µ(1−z) = e−(λ+µ)(1−z) .

Thus, by the uniqueness property of the MGF, X + Y ∼ Poi(λ + µ).

Example 4.9 The MGF of X ∼ Gam(α, λ) is given, see (2.13) on page 52, by
 α
λ
E[e ] =
sX
.
λ−s
As a special case, the moment generating function of the Exp(λ) distribution is
given by λ/(λ − s). Now let X1 , . . . , Xn be iid Exp(λ) random variables. The
MGF of Sn = X1 + · · · + Xn is
 n
λ
E[e ] = E[e
sSn sX1
···esXn
] = E[e ] · · · E[e
sX1 sXn
]= ,
λ−s
which shows that Sn ∼ Gam(n, λ).

Copyright c 2018 D.P. Kroese

4.2 Expectation 75

Definition 4.5 (Covariance) The covariance of two random variables X

and Y is defined as the number
Cov(X, Y ) := E(X − EX)(Y − EY ).

The covariance is a measure for the amount of linear dependency between the
variables. If small values of X (smaller than the expected value of X) go
together with small values of Y , and at the same time large values of X go
together with large values of Y , then Cov(X, Y ) will be positive. If on the other
hand small values of X go together with large values of Y , and large values of
X go together with small values of Y , then Cov(X, Y ) will be negative.

For easy reference we list some important properties of the variance and covari-
ance. The proofs follow directly from the definitions of covariance and variance
and the properties of the expectation.

1. Var(X) = EX 2 − (EX)2 .
2. Var(aX + b) = a2 Var(X).
3. Cov(X, Y ) = EXY − EXEY .
4. Cov(X, Y ) = Cov(Y, X).
5. Cov(aX + bY, Z) = a Cov(X, Z) + b Cov(Y, Z).
6. Cov(X, X) = Var(X).
7. Var(X + Y ) = Var(X) + Var(Y ) + 2 Cov(X, Y ).
8 X and Y indep. =⇒ Cov(X, Y ) = 0.

Table 4.1: Properties of variance and covariance

Proof. Properties 1. and 2. were already proved on page 38. Properties 4.

and 6. follow directly from the definitions of covariance and variance. Denote
for convenience µX = EX and µY = EY .

3. Cov(X, Y ) = E[(X − µX )(Y − µY )] = E[XY − Y µX − XµY + µX µY ] =

EXY − µY µX − µX µY + µX µY = EXY − µX µY .
5. Using property 3. we get
Cov(aX + bY, Z) = E[(aX + bY )Z] − E[aX + bY ] E[Z]
= a E[XZ] + b E[Y Z] − a E[X] E[Z] − b E[Y ] E[Z]
= a{E[XZ] − E[X]E[Z]} + b{E[Y Z] − E[Y ]E[Z]}
= a Cov(X, Z) + b Cov(Y, Z).

7. By property 6. we have
Var(X + Y ) = Cov(X + Y, X + Y ) .

Copyright c 2018 D.P. Kroese

76 Joint Distributions

By property 5. we can expand this to

Cov(X + Y, X + Y ) = Cov(X, X + Y ) + Cov(Y, X + Y )
= Cov(X, X) + Cov(X, Y ) + Cov(Y, X) + Cov(Y, Y ),
where we have also used the symmetry property 4. to expand the second
argument of the covariance. Now, by properties 4. and 6. we can simplify
the above sum to Var(X) + Var(Y ) + 2Cov(X, Y ), which had to be shown.
8. If X and Y are independent, then EXY = EXEY , so that Cov(X, Y ) =
EXY − EXEY = 0.

A scaled version of the covariance is given by the correlation coefficient.

Cov(X, Y )
ρ(X, Y ) := p p .
Var(X) Var(Y )

Theorem 4.5 The correlation coefficient always lies between −1 and 1.

Proof. Let a be an arbitrary real number, and denote the standard deviations
of X and Y by σX and σY . Obviously the variance of ±aX + Y is always
non-negative. Thus, using the properties of covariance and variance

Var(−aX + Y ) = a2 σX
2
+ σY2 − 2aCov(X, Y ) ≥ 0.

So that after rearranging and dividing by σx σY , we obtain

 
1 a σX σY
ρ(X, Y ) ≤ + .
2 σY aσX
Similarly,
Var(aX + Y ) = a2 σX
2
+ σY2 + 2aCov(X, Y ) ≥ 0,
so that  
1 a σX σY
ρ(X, Y ) ≥ − + .
2 σY aσX
By choosing a = σY /σX , we see that −1 ≤ ρ(X, Y ) ≤ 1.

In Figure 4.2 an illustration of the correlation coefficient is given. Each figure

corresponds to samples of size 40 from a different 2-dimensional distribution.
In each case EX = EY = 0 and Var(X) = Var(Y ) = 1.

As a consequence of properties 2. and 7., we have

Theorem 4.6 Suppose X1 , X2 , . . . , Xn are discrete or continuous independent

random variables with variances σ12 , σ 2 , . . . , σn2 . Let

Y = a + b1 X1 + b2 X2 + · · · + bn Xn

Copyright c 2018 D.P. Kroese

4.2 Expectation 77

3 3 3

2 2 2

1 1 1

0 0 0

-1 -1 -1

-2 -2 -2

-3 -3 -3
-2 -1 0 1 2 3 -2 -1 0 1 2 3 -2 -1 0 1 2 3

3 3 3

2 2 2

1 1 1

0 0 0

-1 -1 -1

-2 -2 -2

-3 -3 -3
-2 -1 0 1 2 3 -2 -1 0 1 2 3 -2 -1 0 1 2 3

Figure 4.2: Illustration of correlation coefficient. Above: ρ = 0, ρ = 0.4, ρ =

0.7. Below: ρ = 0.9, ρ = −0.8, ρ = −0.98.

where a, b1 , b2 , . . . , bn are constants. Then

Var(Y ) = b21 Var(X1 ) + · · · + b2n Var(Xn )

= b21 σ12 + · · · + b2n σn2

Proof. By virtue of property 6., and repetitive application of property 5., we

have (note that the constant a does not play a role in the variance):

Var(Y ) = Cov(b1 X1 + b2 X2 + · · · + bn Xn , b1 X1 + b2 X2 + · · · + bn Xn )
Xn X
= Cov(bi Xi , bi Xi ) + 2 Cov(bi Xi , bj Xj ) .
i=1 i<j

Since Cov(bi Xi , bi Xi ) = b2i Var(Xi ) and all covariance term are zero because of
the independence of Xi and XJ j (i 6= j), the result follows.

Example 4.10 (Variance of Bin(n, p) and Hyp(n, r, N )) Consider again Ex-

ample 4.7 where we derived the expectation of X ∼ Bin(n, p) and X ∼ Hyp(n, r, N )
by writing X as
X = X1 + · · · + Xn
of independent (in the binomial case) or dependent (in the hypergeometric case)
Ber(p) random variables, where p = r/N in the hypergeometric case. Using
Theorem 4.6, the variance of the binomial distribution follows directly from

Var(X) = Var(X1 ) + · · · + Var(Xn ) = nVar(X1 ) = np(1 − p).

Copyright c 2018 D.P. Kroese

78 Joint Distributions

For the hypergeometric case must include the covariance terms as well:
X
Var(X) = Var(X1 ) + · · · + Var(Xn ) + 2 Cov(Xi , Xj ) .
i<j

By symmetry all the Cov(Xi , Xj ) are the same (i 6= j). Hence,

Var(X) = nVar(X1 ) + n(n − 1)Cov(X1 , X2 ) .

Since Var(X1 ) = p(1 − p), and Cov(X1 , X2 ) = EX1 X2 − p2 , it remains to find

EX1 X2 = P(X1 = 1, X2 = 1), which is
r−1
P(X1 = 1, X2 = 1) = P(X1 = 1)P(X2 = 1 | X1 = 1) = p .
N −1
Simplifying gives,
N −n
Var(X) = np(1 − p) .
N −1

Expectation Vector and Covariance Matrix

Let X = (X1 , . . . , Xn )T be a random vector. Sometimes it is convenient to

write the expectations and covariances in vector notation.

Definition 4.6 For any random vector X we define the expectation vector
as the vector of expectations

µ = (µ1 , . . . , µn )T := (EX1 , . . . , EXn )T .

The covariance matrix Σ is defined as the matrix whose (i, j)th element is

Cov(Xi , Xj ) = E(Xi − µi )(Xj − µj ) .

If we define the expectation of a vector (matrix) to be the vector (matrix) of

expectations, then we can write:

µ = EX

and
Σ = E(X − µ)(X − µ)T .
Note that µ and Σ take the same role as µ and σ 2 in the 1-dimensional case.
We sometimes write µX and ΣX if we wish to emphasise that µ and Σ belong
to the vector X.

Remark 4.5 Note that any covariance matrix Σ is a symmetric matrix. In

fact, it is positive semi-definite, i.e., for any (column) vector u, we have

uT Σ u ≥ 0 .

Copyright c 2018 D.P. Kroese

4.3 Conditional Distribution 79

To see this, suppose Σ is the covariance matrix of some random vector X with
expectation vector µ. Write Y = X − µ. Then

uT Σ u = uT EY Y T u = E uT Y Y T u

T
2
= E Y T u Y T u = E Y T u ≥ 0.

Note that Y T u is a random variable.

4.3 Conditional Distribution

Suppose X and Y are both discrete or both continuous, with joint pmf/pdf
fX,Y , and suppose fX (x) > 0. The conditional pdf/pmf of Y given X = x is
defined as
fX,Y (x, y)
fY |X (y | x) = , for all y . (4.7)
fX (x)
We can interpret fY |X (· | x) as the pmf/pdf of Y given the information that
X takes the value x. For discrete random variables the definition is simply a
consequence or rephrasing of the conditioning formula (1.4) on page 23. Namely,
P(X = x, Y = y) fX,Y (x, y)
fY |X (y | x) = P(Y = y | X = x) = = .
P(X = x) fX (x)

Example 4.11 We draw “uniformly” a point (X, Y ) from the 10 points on the
triangle D below. Thus, each point is equally likely to be dawn. That is, the
joint pmf is The joint and marginal pmf’s are easy to determine:
1
fX,Y (x, y) = P(X = x, Y = y) = , (x, y) ∈ D,
10

D
4

0
0 1 2 3 4

The marginal pmf’s of X and Y are

5−x
fX (x) = P(X = x) = , x ∈ {1, 2, 3, 4},
10

Copyright c 2018 D.P. Kroese

80 Joint Distributions

and
y
fY (y) = P(Y = y) = , y ∈ {1, 2, 3, 4}.
10
Clearly X and Y are not independent. In fact, if we know that X = 2, then Y
can only take the values j = 2, 3 or 4. The corresponding probabilities are
P(Y = y, X = 2) 1/10 1
P(Y = y | X = 2) = = = .
P(X = 2) 3/10 3
In other words, the conditional pmf of Y given X = 2 is
fX,Y (2, y) 1
fY | X (y | 2) = = , y = 2, 3, 4 .
fX (2) 3
Thus, given X = 2, Y takes the values 2,3 and 4 with equal probability.

When X is continuous, we can not longer directly apply (1.4) to define the
conditional density. Instead, we define first the conditional cdf of Y given
X = x as the limit
FY (y | x) := lim FY (y | x < X ≤ x + h) .
h→0

Now, (1.4) can be applied to FY (y | x < X ≤ x + h) to yield

R y R x+h
fX,Y (u, v) du dv
FY (y | x < X ≤ x + h) = −∞ Rx x+h
x fX (u) du
R x+h
Now, for small h the integral x fX,Y (u, v) du is approximately equal to
hfX,Y (x, v) plus some small term which goes to zero faster than h. Similarly,
R x+h
x fX (u) du ≈ hfX (x) (plus smaller order terms). Hence, for h → 0, the
limit of FY (y | x < X ≤ x + h) is
Ry
fX,Y (x, v) dv
FY (y | x) = −∞ .
fX (x)
Note that FY (y | x) as a function of y has all the properties of a cdf. By
differentiating this cdf with respect to y we obtain the conditional pdf of Y
given X = x in the continuous case, which gives the same formula (4.7) as for
the discrete case.

Since the conditional pmf (pdf) has all the properties of a probability mass
(density) function, it makes sense to define the corresponding conditional ex-
pectation as (in the continuous case)
Z
E[Y | X = x] = y fY |X (y | x) dy .

Conditional pmf’s and pdf’s for more than two random variables are defined
analogously. For example, the conditional pmf of Xn given X1 , . . . , Xn−1 is
given by
P(X1 = x1 , . . . , Xn = xn )
fXn | X1 ,...,Xn−1 (xn | x1 , . . . , xn−1 ) = .
P(X1 = x1 , . . . , Xn−1 = xn−1 )

Copyright c 2018 D.P. Kroese

Chapter 5

Functions of Random
Variables and Limit Theorems

Suppose X1 , . . . , Xn are the measurements on a random experiment. Often we

are interested in certain functions of the measurements only, rather than all
measurements themselves. For example, if X1 , . . . , Xn are the repeated mea-
surements of the strength of a certain type of fishing line, then what we are
really interested in is not the individual values for X1 , . . . , Xn but rather quan-
tities such as the average strength (X1 + · · · + Xn )/n, the minimum strength
min(X1 , . . . , Xn ) and the maximum strength max(X1 , . . . , Xn ). Note that these
quantities are again random variables. The distribution of these random vari-
ables can in principle be derived from the joint distribution of the Xi ’s. We
give a number of examples.

Example 5.1 Let X be a continuous random variable with pdf fX , and let
Y = aX + b, where a 6= 0. We wish to determine the pdf fY of Y . We first
express the cdf of Y in terms of the cdf of X. Suppose first that a > 0. We
have for any y

FY (y) = P(Y ≤ y) = P(X ≤ (y − b)/a) = FX ((y − b)/a) .

Differentiating this with respect to y gives fY (y) = fX ((y − b)/a) /a. For a < 0
we get similarly fY (y) = fX ((y − b)/a) /(−a) . Thus in general
 
1 y−b
fY (y) = fX . (5.1)
|a| a

Example 5.2 Let X ∼ N(0, 1). We wish to determine the distribution of

Y = X 2 . We can use the same technique as in the example above, but note
first that Y can only take values in [0, ∞). For y > 0 we have
√ √
FY (y) = P(Y ≤ y) = P(− y ≤ X ≤ y)
√ √ √
= FX ( y) − FX (− y) = 2FX ( y) − 1 .

Copyright c 2018 D.P. Kroese

82 Functions of Random Variables and Limit Theorems

Differentiating this with respect to y gives

 
√ 1 1 1 √ 2 1
fY (y) = 2 fX ( y) √ = √ exp − ( y) √
2 y 2π 2 y
(1/2)1/2 y −1/2 e−y/2
= .
Γ(1/2)

This is exactly the formula for the pdf of a χ21 -distribution. Thus Y ∼ χ21 .

Example 5.3 (Minimum and Maximum) Suppose X1 , . . . , Xn are indepen-

dent and have cdf F . Let Y = min(X1 , . . . , Xn ) and Z = max(X1 , . . . , Xn ).
The cdf of Y and Z are easily obtained. First, note that the maximum of the
{Xi } is less than some number Z if and only if all Xi are less than z. Thus,

P(Z ≤ z) = P(X1 ≤ z, X2 ≤ z, . . . , Xn ≤ z) = P(X1 ≤ z)P(X2 ≤ z) · · · P(Xn ≤ z),

where the second equation follows from the independence assumption. It follows
that
FZ (z) = (F (z))n .
Similarly,

P(Y > y) = P(X1 > y, X2 > y, . . . , Xn > y) = P(X1 > y)P(X2 > y) · · · P(Xn > y),

so that
FY (y) = 1 − (1 − F (y))n .

Example 5.4 In Chapter 3 we saw an important application of functions of

random variables: the inverse-transform method for generating random vari-
ables. That is, U ∼ U(0, 1), and let F be continuous and strictly increasing cdf.
Then Y = F −1 (U ) is a random variable that has cdf F .

We can use simulation to get an idea of the distribution of a function of one or

more random variables, as explained in the following example.

Example 5.5 Let X and Y be independent and both U(0, 1) distributed.

What what does the pdf of Z = X + Y look like? Note that Z takes values in
(0,2). The following matlab line draws 10,000 times from the distribution of Z
and plots a histogram of the data (Figure 5.1

hist(rand(1,10000)+rand(1,10000),50)

Copyright c 2018 D.P. Kroese

 83

400

300

200

100

0
0 0.5 1 1.5 2

Figure 5.1: Histogram for adding two uniform random variables

This looks remarably like a triangle. Perhaps the true pdf of Z = X + Y has a
triangular shape? This is indeed easily proved. Namely, first observe that the
pdf of Z must be symmetrical around 1. Thus to find the pdf, it suffices to find
its form for z ∈ [0, 1]. Take such a z. Then, see Figure 5.2,
1
ZZ
FZ (z) = P(Z ≤ z) = P((X, Y ) ∈ A) = f (x, y) dxdy = area(A) = z 2 .
A 2
where have used the fact that the joint density f (x, y) is equal to 1 on the
square [0, 1] × [0, 1]. By differentiating the cdf FZ we get the pdf fZ
fZ (z) = z, z ∈ [0, 1],
and by symmetry
fZ (z) = 2 − z, z ∈ [1, 2],
which is indeed a triangular density. If we rescaled the histogram such that the
total area under the bars would be 1, the fit with the true distribution would
be very good.
1
x+y=z

0 z 1

Figure 5.2: The random point (X, Y ) must lie in set A

Copyright c 2018 D.P. Kroese

84 Functions of Random Variables and Limit Theorems

Linear Transformations

Let x = (x1 , . . . , xn )T be a (column) vector in Rn and A an (n × m)-matrix.

The mapping x 7→ z, with
z = Ax
is called a linear transformation. Now consider a random vector X =
(X1 , . . . , Xn )T , and let
Z = AX .
Then Z is a random vector in Rm . Again, in principle, if we know the joint
distribution of X then we can derive the joint distribution of Z. Let us first
see how the expectation vector and covariance matrix are transformed.

Theorem 5.1 If X has expectation vector µX and covariance matrix ΣX , then

the expectation vector and covariance matrix of Z = AX are respectively given
by
µZ = AµX (5.2)
and
Σ Z = A Σ X AT . (5.3)

Proof. We have µZ = EZ = EAX = AEX = AµX and

ΣZ = E(Z − µZ )(Z − µZ )T = EA(X − µX )(A(X − µX ))T

= AE(X − µX )(X − µX )T AT
= A ΣX AT

which completes the proof.

From now on assume A is an invertible (n × n) − matrix. If X has joint density

fX , what is the joint density fZ of Z?

Consider Figure 5.3. For any fixed x, let z = Ax. Hence, x = A−1 z. Consider
the n-dimensional cube C = [z1 , z1 + h] × · · · × [zn , zn + h]. Let D be the image
of C under A−1 , i.e., the parallelepiped of all points x such that Ax ∈ C. Then,

P(Z ∈ C) ≈ hn fZ (z) .

Now recall from linear algebra that any n-dimensional rectangle with “volume”
V is transformed into a n-dimensional parallelepiped with volume V |A|, where
|A| := | det(A)|. Thus,

P(Z ∈ C) = P(X ∈ D) ≈ hn |A−1 | fX (x) = hn |A|−1 fX (x)

Letting h go to 0 we conclude that

fX (A−1 z)
fZ (z) = , z ∈ Rn . (5.4)
|A|

Copyright c 2018 D.P. Kroese

 85

D
−1 C
A
x z
A

Figure 5.3: Linear transformation

General Transformations

We can apply the same technique as for the linear transformation to general
transformations x 7→ g(x), written out:
   
x1 g1 (x)
 x2   g2 (x) 
 ..  7→  ..  .
   
 .   . 
xn gn (x)

For a fixed x, let z = g(x). Suppose g is invertible, hence, x = g −1 (z). Any

infinitesimal n-dimensional rectangle at x with volume V is transformed into
a n-dimensional parallelepiped at z with volume V |Jx (g)|, where Jx (g) is the
matrix of Jacobi at x of the transformation g:
 ∂g1 ∂g1 
∂x1 ··· ∂xn
Jx (g) =  .. ..
 .
 
. ··· .
∂gn ∂gn
∂x1 ··· ∂xn

Now consider a random column vector Z = g(X). Let C be a small cube

around z with volume hn . Let D be image of C under g −1 . Then, as in the
linear case,
P(Z ∈ C) ≈ hn fZ (z) ≈ hn |Jz (g −1 )| fX (x) .
Hence, we have the transformation rule

fZ (z) = fX (g −1 (z)) |Jz (g −1 )|, z ∈ Rn . (5.5)

(Note: |Jz (g −1 )| = 1/|Jx (g)|)

Remark 5.1 In most coordinate transformations it is g −1 that is given — that

is, an expression for x as a function of z, — rather than g.

Example 5.6 (Box-Müller) Let X and Y be two independent standard nor-

mal random variables. (X, Y ) is a random point in the plane. Let (R, Θ) be

Copyright c 2018 D.P. Kroese

86 Functions of Random Variables and Limit Theorems

the corresponding polar coordinates. The joint pdf fR,Θ of R and Θ is given by
1 −r2 /2
fR,Θ (r, θ) = e r , for r ≥ 0 and θ ∈ [0, 2π).
2π
Namely, specifying x and y in terms of r and θ gives

x = r cos θ and y = r sin θ . (5.6)

The Jacobian of this coordinate transformation is

 ∂x ∂x 
∂r ∂θ cos θ −r sin θ
det ∂y ∂y = =r.
∂r ∂θ
sin θ r cos θ

The result now follows from the transformation rule (5.5), noting that the joint
1 −(x2 +y 2 )/2
pdf of X and Y is fX,Y (x, y) = 2π e . It is not difficult to verify that
2
R and Θ are independent, that Θ ∼ U[0, 2π)√and that P(R > r) = e−r /2 . This
means
√ that R has the same distribution as V , with V ∼ Exp(1/2). Namely,
2
P( V > v) = P(V > v 2 ) = e−v /2 . Both Θ and R are easy to generate, and
are transformed via (5.6) to independent standard normal random variables.

5.1 Jointly Normal Random Variables

In this section we have a closer look at normally distributed random variables

and their properties. Also, we will introduce normally distributed random vec-
tors.

It is helpful to view normally distributed random variables as simple transfor-

mations of standard normal random variables. For example, let X ∼ N(0, 1).
Then, X has density fX given by
1 x2
fX (x) = √ e− 2 .
2π
Now consider the transformation

Z = µ + σX.

Then, by (5.1) Z has density

1 (z−µ)2
fZ (z) = √ e− 2σ 2 .
2πσ 2
In other words, Z ∼ N(µ, σ 2 ). We could also write this as follows, if Z ∼
N(µ, σ 2 ), then (Z − µ)/σ ∼ N(0, 1). This standardisation procedure was
already mentioned in Section 2.6.3.

Let’s generalise this to n dimensions. Let X1 , . . . , Xn be independent and stan-

dard normal random variables. The joint pdf of X = (X1 , . . . , Xn )T is given
by
1 T
fX (x) = (2π)−n/2 e− 2 x x , x ∈ Rn . (5.7)

Copyright c 2018 D.P. Kroese

5.1 Jointly Normal Random Variables 87

Consider the transformation

Z = µ + B X, (5.8)

for some (m×n) matrix B. Note that by Theorem 5.1 Z has expectation vector
µ and covariance matrix Σ = BB T . Any random vector of the form (5.8) is
said to have a jointly normal (or multi-variate normal) distribution. We write
Z ∼ N(µ, Σ).

Suppose B is an invertible (n × n)-matrix. Then, by (5.4) the density of Y =

Z − µ is given by
1 1 −1 T −1 1 1 T −1 T −1
fY (y) = p e− 2 (B y) B y = p e− 2 y (B ) B y .
|B| (2π) n |B| (2π) n

We have |B| = |Σ| and (B −1 )T B −1 = (B T )−1 B −1 = (BB T )−1 = Σ−1 , so

p

that
1 1 T −1
fY (y) = p e− 2 y Σ y .
n
(2π) |Σ|
Because Z is obtained from Y by simply adding a constant vector µ, we have
fZ (z) = fY (z − µ), and therefore

1 1
(z−µ)T Σ−1 (z−µ)
fZ (z) = p e− 2 , z ∈ Rn . (5.9)
(2π)n |Σ|

Note that this formula is very similar to the 1-dimensional case.

Example 5.7 Consider the 2-dimensional case with µ = (µ1 , µ2 )T , and

 
σ1 p0
B= . (5.10)
σ2 ρ σ2 1 − ρ2

The covariance matrix is now

σ12
 
ρσ1 σ2
Σ= . (5.11)
ρσ1 σ2 σ22

Therefore, the density is


1 1
fZ (z) = exp −
ρ2 )
p
2πσ1 σ2 1 − ρ 2 2(1 −
(z1 − µ1 )2 (z1 − µ1 )(z2 − µ2 ) (z2 − µ2 )2
 
− 2ρ + . (5.12)
σ12 σ1 σ2 σ22

Here are some pictures of the density, for µ1 = µ2 = 0 and σ1 = σ2 = 1, and

for various ρ.

Copyright c 2018 D.P. Kroese

88 Functions of Random Variables and Limit Theorems

0.2
3 0.2 3
0.15
2
0.15 2

0.1 1
1
0.1
0
0.05 0
−1 0.05
−1
0
−2
3 0 −2
2
1
0
−1 −3 3 2
−2 1
−3 0 −3
−1 −2 −3

ρ=0 ρ = 0.5

0.4 3

0.3 2

1
0.2

0
0.1

−1
0
3 −2
2
1
0
−1
−2 −3
−3
ρ = 0.9

We say that (Z1 , Z2 )T has a bivariate normal distribution. Note that in

this example EZi = µi , i = 1, 2. Moreover, since we have chosen B such that
the covariance matrix has the form (5.11), we have Var(Zi ) = σi2 , i = 1, 2,
and ρ(Z1 , Z2 ) = ρ. We will see shortly that Z1 and Z2 both have normal
distributions.

Compare the following with property 8 of Table 4.1.

Theorem 5.2 If Z1 and Z2 have a jointly normal distribution then

Cov(Z1 , Z2 ) = 0 =⇒ Z1 and Z2 are independent.

Proof. If Cov(Z1 , Z2 ) = 0, then B in (5.10) is a diagonal matrix. Thus,

trivially Z1 = σ1 X1 and Z2 = σ2 X2 are independent.

One of the most (if not the most) important properties of the normal distri-
bution is that linear combinations of independent normal random variables are
normally distributed. Here is a more precise formulation.

Theorem 5.3 If Xi ∼ N(µi , σi2 ), independently, for i = 1, 2, . . . , n, then

n n n
!
X X X
Y =a+ bi Xi ∼ N a + bi µi , b2i σi2 . (5.13)
i=1 i=1 i=1

Copyright c 2018 D.P. Kroese

5.2 Limit Theorems 89

Proof. The easiest way to prove this is by using moment generating functions.
First, recall that the MGF of a N(µ, σ 2 )-distributed random variable X is given
by
1 2 2
MX (s) = eµs+ 2 σ s .
Let MY be the moment generating function of Y . Since X1 , . . . , Xn are inde-
pendent, we have
n
X
MY (s) = E exp{as + bi Xi s}
i=1
n
Y
= eas MXi (bi s)
i=1
n
Y 1
= eas exp{µi (bi s) + σi2 (bi s)2 }
2
i=1
n n
X 1X 2 2 2
= exp{sa + s bi µi + bi σi s },
2
i=1 i=1

which is the MGF of a normal distribution of the form (5.13).

Remark 5.2 Note that from Theorems 4.3 and 4.6 we had already established
the expectation and variance of Y in (5.13). But we have now found that the
distribution is normal.

Example 5.8 A machine produces ball bearings with a N(1, 0.01) diameter
(cm).The balls are placed on a sieve with a N(1.1, 0.04) diameter. The diameter
of the balls and the sieve are assumed to be independent of each other.
Question: What is the probability that a ball will fall through?

Answer: Let X ∼ N(1, 0.01) and Y ∼ N(1.1, 0.04). We need to calculate

P(Y > X) = P(Y − X > 0). But, Z := Y − X ∼ N(0.1, 0.05). Hence
 
Z − 0.1 −0.1
P(Z > 0) = P √ >√ = Φ(0.447) ≈ 0.67 ,
0.05 0.05
where Φ is the cdf of the N(0, 1)-distribution.

5.2 Limit Theorems

In this section we briefly discuss two of the main results in probability: the
Law of Large Numbers (LLN) and the Central Limit Theorem (CLT). Both are
about sums of independent random variables.

Let X1 , X2 , . . . be independent and identically distributed random variables.

For each n let
Sn = X1 + · · · + Xn .

Copyright c 2018 D.P. Kroese

90 Functions of Random Variables and Limit Theorems

Suppose EXi = µ and Var(Xi ) = σ 2 . We assume that both µ and σ 2 are finite.
By the rules for expectation and variance we know that
ESn = n EX1 = nµ
and
Var(Sn ) = n Var(X1 ) = nσ 2 .
Moreover, if the Xi have moment generating function M , then the MGF of Sn
is simply given by
Ees(X1 +···+Xn ) = EesX1 · · · EesXn = [M (s)]n .

The law of large numbers roughly states that Sn /n is close to µ, for large n.
Here is a more precise statement.

Theorem 5.4 ((Weak) Law of Large Numbers) If X1 , . . . , Xn are inde-

pendent and identically distributed with expectation µ, then for all  > 0
 
Sn
lim P −µ ≥ =0.
n→∞ n

Proof. First, for any z > 0, and any positive random variable Z we have
Z z Z ∞ Z ∞
EZ = tf (t) dt + tf (t) dt ≥ tf (t) dt
0 z z
Z ∞
≥ zf (t) dt = z P(Z ≥ z) ,
z

from which follows immediately the following Markov inequality: if Z ≥ 0,

then for all z > 0,
EZ
P(Z ≥ z) ≤ . (5.14)
z
Now take Z = (Sn /n − µ)2 and z = 2 . Then,
E(Sn /n − µ)2
P(Z 2 ≥ 2 ) ≤
2
The left-hand size of the above equation can also be written as P(|Sn /n − µ| ≥
), and the right-hand side is equal to the variance of Sn /n, which is σ 2 /n.
Combining gives,
σ2
P(|Sn /n − µ| ≥ ) ≤ 2 ,
n
σ 2
for any  > 0. As n → ∞ the quotient n 2 tends to zero, and therefore
P(|Sn /n − µ| ≥ ) goes to zero as well, which had to be shown.

There is also a strong law of large numbers, which implies the weak law,
but is more difficult to prove. It states the following:
 
Sn
P lim = µ = 1,
n→∞ n

Copyright c 2018 D.P. Kroese

5.2 Limit Theorems 91

as n → ∞, meaning that the set of outcomes ω such that Snn(ω) → µ, has

probability one. In other words if we were to run a computer simulation, then
all paths that we would simulate would converge to µ.

The Central Limit Theorem says something about the approximate distribution
of Sn (or Sn /n). Roughly it says this:

The sum of a large number of iid random variables

has approximately a normal distribution

Here is a more precise statement.

Theorem 5.5 (Central Limit Theorem) If X1 , . . . , Xn are independent and

identically distributed with expectation µ and variance σ 2 < ∞, then for all
x ∈ R,
 
Sn − nµ
lim P √ ≤ x = Φ(x) ,
n→∞ σ n
where Φ is the cdf of the standard normal distribution.

In other words, Sn has approximately a normal distribution with expectation

nµ and variance nσ 2 .

Proof.(Sketch) Without loss of generality assume µ = 0 and σ = 1. This

amounts to replacing Xn by (Xn − µ)/σ. A Taylor-expansion of the MGF
around s = 0 yields
1 1
M (s) = E esX1 = 1 + sEX1 + s2 EX12 + o(s2 ) = 1 + s2 + o(s2 ),
2 2
where o(·) is a function for which limx↓0 o(x)/x = 0. Because the X1 , X2 , . . .
√
are i.i.d., it follows that the MGF of Sn / n satisfies
    Y n  
Sn s s
E exp s √ = E exp √ (X1 + · · · Xn ) = E exp √ Xi
n n n
i=1
n
s2 s2
   
s
= Mn √ = 1+ + o( ) .
n 2n n
2
For n → ∞ this converges to es /2 , which is the MGF of the standard normal
√
distribution. Thus, it is plausible that the cdf of Sn / n converges to Φ. To
make this argument rigorous, one needs to show that convergence of the moment
generating function implies convergence of the cdf. Moreover, since for some
distributions the MGF does not exist in a neighbourhood of 0, one needs to
replace the MGF in the argument above with a more flexible transform, namely
the Fourier transform, also called characteristic function: r 7→ EeirX , r ∈ R.

To see the CLT in action consider Figure 5.4. The first picture shows the pdf’s
of S1 , . . . , S4 for the case where the Xi have a U[0, 1] distribution. The second

Copyright c 2018 D.P. Kroese

92 Functions of Random Variables and Limit Theorems

show the same, but now for an Exp(1) distribution. We clearly see convergence
to a bell shaped curve.

n=1
1
n=2 0.8
0.8 n=3 n=1
0.6
0.6
n=4
0.4 n=2
0.4
n=3

0.2 0.2 n=4

00 1 2 3 4 5 00 2 4 x 6 8

Figure 5.4: Illustration of the CLT for the uniform and exponential distribution

The CLT is not restricted to continuous distributions. For example, Figure 5.5
shows the cdf of S30 in the case where the Xi have a Bernoulli distribution with
success probability 1/2. Note that S30 ∼ Bin(30, 1/2), see Example 4.3.

0.8

0.6

0.4

0.2

00 5 10 15
x 20 25 30

Figure 5.5: The cdf of a Bin(20, 1/2)-distribution and its normal approximation.

In general we have:

Theorem 5.6 Let X ∼ Bin(n, p). For large n we have

P(X ≤ k) ≈ P(Y ≤ k),

where Y ∼ N(np, np(1−p)) . As a rule of thumb, the approximation is accurate

if both np and n(1 − p) are larger than 5.

Copyright c 2018 D.P. Kroese

Appendix A

Exercises and Solutions

A.1 Problem Set 1

1. We toss a fair coin three times.

(a) Find the sample space, if we observe the exact sequences of Heads
(= 1) and Tails (= 0).
(b) Find the sample space, if we observe only the total number of Heads.

2. Assign a probability to all elementary events in the sample spaces 1.(a)

and 1.(b).

3. We randomly select 3 balls from an urn with 365 balls, numbered 1,

. . . ,365, noting the order.

(a) How many possible outcomes of the experiment are there, if we put
each ball back into the urn before we draw the next?
(b) Answer the same question as above, but now if we don’t put the balls
back.
(c) Calculate the probability that in case (a) we draw 3 times the same
ball.

4. Let P(A) = 0.9 and P(B) = 0.8. Show that P(A ∩ B) ≥ 0.7.

5. What is the probability that none of 54 people in a room share the same
birthday?

6. Consider the experiment of throwing 2 fair dice.

(a) Find the probability that both dice show the same face.
(b) Find the same probability, using the extra information that the sum
of the dice is not greater than 4.

7. We draw 3 cards from a full deck of cards, noting the order. Number the
cards from 1 to 52.

Copyright c 2018 D.P. Kroese

94 Exercises and Solutions

(a) Give the sample space. Is each elementary event equally likely?
(b) What is the probability that we draw 3 Aces?
(c) What is the probability that we draw 1 Ace, 1 King and 1 Queen?
(d) What is the probability that we draw no pictures (no A,K,Q,J)?

8. We draw at random a number in the interval [0,1] such that each number
is “equally likely”. Think of the random generator on you calculator.

(a) Determine the probability that we draw a number less than 1/2.
(b) What is the probability that we draw a number between 1/3 and
3/4?
(c) Suppose we do the experiment two times (independently), giving us
two numbers in [0,1]. What is the probability that the sum of these
numbers is greater than 1/2? Explain your reasoning.

9. Select at random 3 people from a large population. What is the proba-

bility that they all have the same birthday?

10. We draw 4 cards (at the same time) from a deck of 52, not noting the
order. Calculate the probability of drawing one King, Queen, Jack and
Ace.

11. In a group of 20 people there are three brothers. The group is separated at
random into two groups of 10. What is the probability that the brothers
are in the same group?

12. How many binary vectors are there of length 20 with exactly 5 ones?

13. We draw at random 5 balls from an urn with 20 balls (numbered 1,. . . ,20),
without replacement or order. How many different possible combinations
are there?

14. In a binary transmission channel, a 1 is transmitted with probability 2/3

and a 0 with probability 1/3. The conditional probability of receiving a 1
when a 1 was sent is 0.95, the conditional probability of receiving a 0 when
a 0 was sent is 0.90. Given that a 1 is received, what is the probability
that a 1 was transmitted?

15. Consider the following system. Each component has a probability 0.1 of
failing. What is the probability that the system works?
2

16. Two fair dice are thrown and the smallest of the face values, Z say, is
noted.

Copyright c 2018 D.P. Kroese

A.2 Answer Set 1 95

z * * * ...
(a) Give the pmf of Z in table form:
P(Z = z) * * * ...
(b) Calculate the expectation of Z.

17. In a large population 40% votes for A and 60% for B. Suppose we select
at random 10 people. What is the probability that in this group exactly
4 people will vote for A?

18. We select “uniformly” a point in the unit square: {(x, y) : 0 ≤ x ≤ 1, 0 ≤

y ≤ 1}. Let Z be the largest of the coordinates. Give the cdf and pdf of
Z and draw their graphs.

19. A continuous random variable X has cdf F given by,


0,
 x<0
F (x) = x ,3 x ∈ [0, 1]

1 x>1.


(a) Determine the pdf of X.

(b) Calculate P(1/2 < X < 3/4).
(c) Calculate E[X].

A.2 Answer Set 1

1. (a) Ω = {(0, 0, 0), . . . , (1, 1, 1)}.

(b) Ω = {0, 1, 2, 3}.

2. (a) P({(x, y, z)}) = 1/8 for all x, y, z ∈ {0, 1}.

(b) P({0}) = 1/8, P({1}) = 3/8, P({2}) = 3/8, P({3}) = 1/8.

3. (a) |Ω| = 3653 .

(b) |Ω| = 365 × 364 × 363.
(c) 365/3653 = 1/3652 .

4. P(A ∩ B) = P(A) + P(B) − P(A ∪ B). Since P(A ∪ B) ≤ 1, we have

P(A ∩ B) ≥ P(A) + P(B) − 1 = 1.7 − 1 = 0.7.
365×364×···×(365−54+1)
5. 36554
≈ 0.016.

6. (a) 1/6.
(b) Let A be the event that the dice show the same face, and B the event
that the sum is not greater than 4. Then B = {(1, 1), (1, 2), (1, 3)(2, 1), (2, 2), (3, 1)},
and A ∩ B = {(1, 1), (2, 2)}. Hence, P(A | B) = 2/6 = 1/3.

7. (a) Ω = {(1, 2, 3), . . . , (52, 51, 50)}. Each elementary event is equally
likely.

Copyright c 2018 D.P. Kroese

96 Exercises and Solutions

4×3×2
(b) 52×51×50 .
12×8×4
(c) 52×51×50 .
36×35×34
(d) 52×51×50 .

8. (a) 1/2.
(b) 5/12.
(c) 7/8.

9. 1/3652 , see question 3 (c).

10. |Ω| = 52


4 (all equally likely outcomes. Note that the outcomes are
repsesented as (unordered sets), e.g., {1, 2, 3, 4}.
Let A

be
the event of
drawing one K, Q, J and Ace each. Then, |A| = 41 × 41 × 41 × 41 = 44 .

Thus, P(A) = 44 / 52


4 .

11. Suppose we choose 10 people to go in group 1 (the
rest go in group 2).

The total number of ways this can be done is 20 10 . Let A be the event
that the brothers belong to the same group. The number of ways in which
17


they can be chosen into group 1 is: 7 . The number of ways they can
be chosen into group 2 is the same, 17 17 17

20

10 = 7 . Thus, P(A) = 2 7 / 10 .

12. 20


5 , because we have to choose the 5 positions for the 1s, out of 20
positions.

13. 20


5

14. Let B be the event that a 1 was sent, and A the event that a 1 is received.
Then, P(A | B) = 0.95, and P(Ac | B c ) = 0.90. Thus, P(Ac | B) = 0.05 and
P(A | B c ) = 0.10. Moreover, P(B) = 2/3 and P(B c ) = 1/3. By Bayes:

P(A | B)P(B) 0.95 × 23

P(B | A) = =
P(A | B)P(B) + P(A | B c )P(B c ) 0.95 × 23 + 0.10 × 1
3

15. Let Ai be the event that component i works, i = 1, 2, 3, and let A be the
event that the system works. We have A = A1 ∩ (A2 ∪ A3 ). Thus, by the
independence of the A0i s:

P(A) = P(A1 ) × P(A2 ∪ A3 )

= P(A1 ) × [P(A2 ) + P(A3 ) − P(A2 ∩ A3 )]
= P(A1 ) × [P(A2 ) + P(A3 ) − P(A2 )P(A3 )]
= 0.9 × [0.9 + 0.9 − 0.81]

z 1 2 3 4 5 6
16. (a)
P(Z = z) 11/36 9/36 7/36 5/36 3/36 1/36
(b) E[Z] = 1×11/36+2×9/36+3×7/36+4×5/36+5×3/36+6×1/36.

17. Let X be the number that vote for A. Then X ∼ Bin(10, 0.4). Hence,
P(X = 4) = 10 4 (0.6)6 .


4 (0.4)

Copyright c 2018 D.P. Kroese

A.3 Problem Set 2 97

18. The region where the largest coordinate is less than z is a square with
area z 2 (make a picture). Divide this area by the area of the unit square
(1), to obtain P(Z ≤ z) = z 2 , for all z ∈ [0, 1]. Thus,

0
 z<0
F (z) = P(Z ≤ z) = z 2 0≤z≤1

1 z>1.


Differentiate to get the pdf:

(
2z 0≤z≤1
f (z) =
0 otherwise .

19. (a) (
3 x2 0≤x≤1
f (x) =
0 otherwise .
R 3/4
(b) f (x) dx = F (3/4) − F (1/2) = (3/4)3 − (1/2)3 .
1/2
R1 R1
(c) E[X] = 0 x 3x2 dx = 3 0 x3 dx = 3/4.

A.3 Problem Set 2

1. In a binary communication channel, 0s and 1s are transmitted with equal

probability. The probability that a 0 is correctly received (as a 0) is 0.95.
The probability that a 1 is correctly received (as a 1) is 0.99. Suppose we
receive a 0, what is the probability that, in fact, a 1 was sent?

2. Throw two fair dice one after the other.

(a) What is the probability that the second die is 3, given that the sum
of the dice is 6?
(b) What is the probability that the first die is 3 and the second not 3?

3. We flip a fair coin 20 times.

(a) What is the probability of exactly 10 Heads?

(b) What is the probability of 15 or more Heads?

4. We toss two fair dice until their sum is 12.

(a) What is probabillity that we have to wait exactly 10 tosses?

(b) What is the probability that we do not have to wait more than 100
tosses?

5. We independently throw 10 balls into one of 3 boxes, numbered 1,2 and

3, with probabilities 1/4, 1/2 and 1/4, respectively.

Copyright c 2018 D.P. Kroese

98 Exercises and Solutions

(a) What is the probability that box 1 has 2, box 2 has 5 and box 3 has
3 balls?
(b) What is the probability that box 1 remains empty?

6. Consider again the experiment where we throw two fair dice one after the
other. Let the random variable Z denote the sum of the dice.

(a) How is Z defined as a function?

(b) What is the pmf of Z?

7. Consider the random experiment of question 4. Let X be the number of

tosses required until the sum of the dice is 12. Give the pmf of X.

8. We draw at random and uniformly a point from the interior of a circle

with radius 4. Let R be the distance of this point to the centre of the
circle.

(a) What is the probability that R > 2?

(b) What is the pdf of R?

9. Let X ∼ Bin(4, 1/2). What is the pmf of Y = X 2 ?

10. Let X ∼ U[0, 1]. What is the pdf of Y = X 2 ?

11. Let X ∼ N(0, 1), and Y = 1 + 2X. What is the pdf of Y ?

12. Let X ∼ N(0, 1). Find P(X ≤ 1.4) from the table of the N(0, 1) distribu-
tion. Also find P(X > −1.3).

13. Let Y ∼ N(1, 4). Find P(Y ≤ 3), and P(−1 ≤ Y ≤ 2).

14. If X ∼ Exp(1/2) what is the pdf of Y = 1 + 2X? Sketch the graph.

15. We draw at random 5 numbers from 1,. . . 100, with replacement (for ex-
ample, drawing number 9 twice is possible). What is the probability that
exactly 3 numbers are even?

16. We draw at random 5 numbers from 1,. . . 100, without replacement. What
is the probability that exactly 3 numbers are even?

17. A radioactive source of material emits a radioactive particle with probabil-

ity 1/100 in each second. Let X be the number of particles emitted in one
hour. X has approximately a Poisson distribution with what parameter?
Draw (with the use of a computer) or sketch the pmf of X.

18. An electrical component has a lifetime X that is exponentially distributed

with parameter λ = 1/10 per year. What is the probability the component
is still alive after 5 years?

19. A random variable X takes the values 0, 2, 5 with probabilities 1/2, 1/3, 1/6,
respectively. What is the expectation of X?

Copyright c 2018 D.P. Kroese

A.4 Answer Set 2 99

20. A random variable X has expectation 2 and variance 1. Calculate E[X 2 ].

21. We draw at random a 10 balls from an urn with 25 red and 75 white balls.
What is the expected number of red balls amongst the 10 balls drawn?
Does it matter if we draw the balls with or without replacement?

22. Let X ∼ U[0, 1]. Calculate Var(X).

23. If X ∼ U[0, 1], what is the expectation of Y = 10 + 2X?

24. We repeatedly throw two fair dice until two sixes are thrown. What is
the expected number of throws required?

25. Suppose we divide the population of Brisbane (say 1,000,000 people) ran-
domly in groups of 3.

(a) How many groups would you expect there to be in which all persons
have the same birthday?
(b) What is the probability that there is at least one group in which all
persons have the same birthday?

26. An electrical component has an exponential lifetime with expectation 2

years.

(a) What is the probability that the component is still functioning after
2 years?
(b) What is the probability that the component is still functioning after
10 years, given it is still functioning after 7 years?

27. Let X ∼ N(0, 1). Prove that Var(X) = 1. Use this to show that Var(Y ) =
σ 2 , for Y ∼ N(µ, σ 2 ).

28. let X ∼ Exp(1). Use the Moment Generating Function to show that
E[X n ] = n!.

29. Explain how to generate random numbers from the Exp(10) distribution.
Sketch a graph of the scatterplot of 10 such numbers.

30. Explain how to generate random numbers from the U[10, 15] distribution.
Sketch a graph of the scatterplot of 10 such numbers.

31. Suppose we can generate random numbers from the N(0, 1) distribution,
e.g., via Matlabs randn function. How can we generate from the N(3, 9)
distribution?

A.4 Answer Set 2

0.01× 12
1. 0.01× 21 +0.95× 12
(Bayes’ rule).

Copyright c 2018 D.P. Kroese

100 Exercises and Solutions

1
2. (a) 5 (conditional probability, the possible outcomes are (1, 5), (2, 4), . . . , (5, 1).
In only one of these the second die is 3).
1
(b) 6 × 56 (independent events).
20

20
3. (a) 10 /2 ≈ 0.176197.
P20 k

20
(b) k=15 20 /2 ≈ 0.0206947.
35 9 1


4. (a) 36 36 ≈ 0.021557. (geometric formula)

100
(b) 1 − 35 36 ≈ 0.94022.
10! 1 2 1 5 1 3 315




5. (a) 2! 5! 3! 4 2 4 = 4096 ≈ 0.076904. (multinomial)
(b) (3/4)10 ≈ 0.0563135.

6. (a) Z((x, y)) = x + y for all x, y ∈ {1, 2, . . . , 6}.

(b) Range: SX = {2, 3, . . . , 12}. Pmf: P(X = x) = 6−|x−7|
36 , x ∈ SX .
35 x−1 1
7. P(X = x) = 36


36 , x ∈ {1, 2, . . .}. (geometric formula)
π16−π 4 3
8. (a) π 16 = 4 .
(b) First, the cdf of R is FR (r) = π r2 /(π16) = r2 /16, r ∈ (0, 4). By
differentiating we obtain the pdf f (r) = 8r , 0 < r < 4 .
(k4)
9. SY = {0, 1, 4, 9, 16}. P(Y = k 2 ) = P(X = k) = 16 , k = 0, 1, . . . , 4.

10. SY = [0, 1]. For 0 ≤ y ≤ 1 we have P(Y ≤ y) = P(X 2 ≤ y) = P(X ≤

√ √ 1
y) = y. Thus, fY (y) = 2 √ y, 0 < y ≤ 1.

11. Y ∼ N(1, 4). (affine transformation of a normal r.v. gives again a normal
r.v.)

12. P(X ≤ 1.4) = Φ(1.4) ≈ 0.9192. P(X > −1.3) = P(X < 1.3) (by symme-
try of the pdf — make a picture). P(X < 1.3) = P(X ≤ 1.3) = Φ(1.3) ≈
0.9032.

13. P(Y ≤ 3) = P(1 + 2X ≤ 3), with X standard normal. P(1 + 2X ≤

3) = P(X ≤ 1) ≈ 0.8413. P(−1 ≤ Y ≤ 2) = P(−1 ≤ X ≤ 1/2) = P(X ≤
1/2)−P(X ≤ −1) = Φ(1/2)−(1−Φ(1)) ≈ 0.6915−(1−0.8413) = 0.5328.

14. fY (y) = fX ((y − 1)/2)/2 = e−(y−1)/4 /4, y≥1.

15. Let X be the total
number of “even” numbers. Then, X ∼ Bin(5, 1/2).
And P(X = 3) = 53 /32 = 10/32 = 0.3125

16. Let X be the total number of “even” numbers. Then X ∼ Hyp(5, 50, 100).
(50)(50)
Hence P(X = 3) = 3 100 2 = 6125/19206 ≈ 0.318911.
(5)
17. λ = 3600/100 = 36.

18. e−5/10 ≈ 0.6065.

Copyright c 2018 D.P. Kroese

 A.5 Problem Set 3 101

19. 3/2.

20. 5

21. It does matter for the distribution of the number of red balls, which is Bin(10, 1/4)
if we replace and Hyp(10, 25, 100) if we don’t replace. However the expectation
is the same for both cases: 2.5.
R1
22. Var(X) = EX 2 −(EX)2 . By symmetry EX = 1/2. And EX 2 = 0 x2 dx = 1/3.
So Var(X) = 1/12.

23. 10 + 2EX = 11.

24. 36 (expectation for the G(1/36) distribution).

25. Let N be the number of groups in which each person has the same birthday.
Then N ∼ Bin(333333, 1/3652 ). Hence (a) EN ≈ 2.5, and (b) P(N > 0) = 1 −
P(N = 0) = 1 − (1 − 1/3652 )333333 ≈ 0.92. [Alternatively N has approximately
a Poi(2.5) distribution, so P(N = 0) ≈ e−2.5 , which gives the same answer 0.92.]

26. First note λ = 1/2. Let X be the lifetime.

(a) P(X > 2) = e2/2 = e−1 ≈ 0.368.

(b) P(X > 10 | X > 7) = P(X > 3) = e−1.5 ≈ 0.223 (memoryless
property).
R ∞ 2 −x2 /2 R∞ −x2 /2 R ∞ −x2 /2
27. Var(X) = EX 2 −EX = EX 2 = −∞ x e√2π dx = −∞ x xe√2π dx = −∞ e √2π dx =
1. Note that we have used partial integration in the fifth equality. For general
Y ∼ N(µ, σ 2 ), write Y = µ+σX, so that Var(Y ) = Var(µ+σX) = σ 2 Var(X) =
σ2.

28. M (s) = λ/(λ − s). Differentiate: M 0 (s) = λ/(λ − s)2 , M 00 (s) = 2λ/(λ − s)3 ,
. . . , M (n) (s) = n! λ/(λ − s)n+1 . Now apply the moment formula.
1
29. Draw U ∼ U(0, 1). Return X = − 10 log U .

30. Draw U ∼ U(0, 1). Return X = 10 + 5U .

31. Draw X ∼ N(0, 1). Return Y = 3 + 3X.

A.5 Problem Set 3

1. Consider the random experiment where we draw independently n num-

bers from the interval [0,1]; each number in [0,1] being equally likely to
be drawn. Let the independent and U[0, 1]-distributed random variables
X1 , . . . , Xn represent the numbers that are to be drawn.

(a) Let M be the smallest of the n numbers, and X̄ the average of the
n numbers. Express M and X̄ in terms of X1 , . . . , Xn .

Copyright c 2018 D.P. Kroese

102 Exercises and Solutions

(b) Determine the pdf of M .

(c) Give the expectation and variance of X̄.

2. The joint pmf of X and Y is given by the table

y
x 1 3 6 8
2 0 0.1 0.1 0
5 0.2 0 0 0
6 0 0.2 0.1 0.3

(a) Determine the (marginal) pmf of X and of Y .

(b) Are X and Y independent?
(c) Calculate E[X 2 Y ].

3. Explain how, in principle we can calculate

 2 
X1 + sin(X2 )
P >1 ,
X12 X2

if we know the joint pdf of X1 and X2 .

4. Suppose X1 , X2 , . . . , Xn are independent random variables, with cdfs F1 , F2 , . . . ,

Fn , respectively. Express the cdf of M = max(X1 , . . . , Xn ) in terms of
the {Fi }.

5. Let X1 , . . . , X6 be the weights of 6 people, selected from a large pop-

ulation. Suppose the weights have a normal distribution with a mean
of 75 kg and a standard deviation of 10 kg. What do Y1 = 6X1 and
Y2 = X1 + · · · + X6 represent, physically? Explain why Y1 and Y2 have
different distributions. Which one has the smallest variance?

6. Let X ∼ Bin(100, 1/4). Approximate, using the CLT, the probability

P(20 ≤ X ≤ 30).

7. Let X and Y be independent and Exp(1) distributed. Consider the coor-

dinate transformation

x = uv , y = u − uv (thus u = x + y and v = x/(x + y)) .

Let U = X + Y and V = X/(X + Y ).

(a) Determine the Jacobian of the above coordinate transformation.

(b) Determine the joint pdf of U and V .
(c) Show that U and V are independent.

8. A random vector (X, Y ) has joint pdf f , given by

f (x, y) = 2 e−x−2y , x > 0, y > 0 .

Copyright c 2018 D.P. Kroese

A.5 Problem Set 3 103

(a) Calculate E[XY ].

(b) Calculate the covariance of X + Y and X − Y .

9. Consider the random experiment where we make repeated measurements

of the voltage across a resistor in an electric circuit. Let X1 , . . . , Xn be the
voltage readings. We assume that the X1 , . . . , Xn are independent and
normally distributed with the same (unknown) mean µ and (unknown)
variance σ 2 . Suppose x1 , . . . , xn are the outcomes of the random variables
X1 , . . . , Xn . Let X̄ = (X1 + · · · + Xn )/n.

(a) How would you estimate the unknown parameter µ if you had the
data x1 , . . . , xn ?
(b) Show that EX̄ = µ.
(c) Show that VarX̄ goes to 0 as n → ∞.
(d) What is the distribution of X̄?
(e) Discuss the implications of (b) and (c) for your estimation of the
unknown µ.

10. Let X ∼ Bin(100, 1/2). Approximate, using the CLT, the probability
P(X ≥ 60).

11. Let X have a uniform distribution on the interval [1, 3]. Define Y = X 2 −4.
Derive the probability density function (pdf) of Y . Make sure you also
specify where this pdf is zero.

12. Let X ∼ U(1, 3). Define Y = 4X + 5. What is the distribution of Y ?

13. Let Y ∼ N(2, 5).

(a) Sketch the pdf and cdf of Y .

(b) Calculate P(Y ≤ 5). [Use the table for the cdf of the N(0, 1)-
distribution.]
(c) Let Z = 3Y − 4. Determine P(1 ≤ Z ≤ 5). [Use the table for the cdf
of the N(0, 1)-distribution.]

14. Some revision questions. Please make sure you can comfortably answer
the questions below by heart.

(a) Give the formula for the pmf of the following distributions:
i. Bin(n, p),
ii. G(p),
iii. Poi(λ).
(b) Give the formula for the pdf of the following distributions:
i. U(a, b),
ii. Exp(λ),
iii. N(0, 1).

Copyright c 2018 D.P. Kroese

104 Exercises and Solutions

(c) Give examples of random experiments where the distributions in (a)

and (b) above occur.

15. Random variables X1 , X2 , . . . are independent and have a standard normal

distribution. Let Y1 = X1 , Y2 = X1 + X2 , and, generally, Yn = X1 + · · · +
Xn , n = 1, 2, . . ..

(a) Sketch a typical outcome of X1 , X2 , . . . , Xn and Y1 , Y2 , . . . , Yn for

large n.
(b) Determine EYn and Var(Yn ).
(c) Determine Cov(Ym , Yn ), m ≤ n.

16. A lift can carry a maximum of 650 kg. Suppose that the weight of a person
is normally distributed with expectation 75 kg and standard deviation 10
kg. Let Zn be the total weight of n randomly selected persons.

(a) Determine the probability that Z8 ≥ 650.

(b) Determine n such that P(Zn ≥ 650) ≤ 0.01.

17. The thickness of a printed circuit board is required to lie between the
specification limits of 0.150 - 0.004 and 0.150 + 0.004 cm. A machine
produces circuit boards with a thickness that is normally distributed with
mean 0.151 cm and standard deviation 0.003 cm.

(a) What is the probability that the thickness X of a circuit board which
is produced by this machine falls within the specification limits?
(b) Now consider the mean thickness X̄ = (X1 +· · ·+X25 )/25 for a batch
of 25 circuit boards. What is the probability that this batch mean
will fall within the specification limits? Assume that X1 , . . . , X25
are independent random variables with the same distribution as X
above.

18. We draw n numbers independently and uniformly from the interval [0,1]
and note their sum Sn .

(a) Draw the graph of the pdf of S2 .

(b) What is approximately the distribution of S20 ?
(c) Calculate the probability that the average of the 20 numbers is
greater than 0.6.

19. Consider the following game: You flip 10 fair coins, all at once, and count
how many Heads you have. I’ll pay you out the squared number of Heads,
in dollars. However, you will need to pay me some money in advance. How
much would you prepare to give me if you could play this game as many
times as you’d like?

Copyright c 2018 D.P. Kroese

A.6 Answer Set 3 105

A.6 Answer Set 3

1. (a) M = min(X1 , . . . , Xn ) and X̄ = (X1 + · · · + Xn )/n.
(b) P(M > x) = P(X1 > x, · · · , Xn > x) = (by indep. of X1 , . . . , Xn ) P(X1 >
x) · · · P(Xn > x) = (1 − x)n , for all 0 ≤ x ≤ 1. Hence the cdf of
M is given by FM (x) = 1 − (1 − x)n , 0 ≤ x ≤ 1 . Consequently, the
pdf of M is given by fM (x) = n(1 − x)n−1 , 0 ≤ x ≤ 1 .
(c) EX̄ = E X1 +···+X = n (EX1 + · · · + EXn ) = n1 nEX1 = 21 .
n

1
n
X1 +···+Xn
= n12 (VarX1 +· · ·+VarXn ) = n12 nVar(X1 ) =


Var(X̄) = Var n |{z}
by indep.
1
12n .
2. (a) P(X = 2) = 0.2, P(X = 5) = 0.2, P(X = 6) = 0.6.
P(Y = 1) = 0.2, P(Y = 3) = 0.3, P(Y = 6) = 0.2, P(Y = 8) =
0.3 .
(b) No. For example P(X = 2, Y = 1) = 0 6= P(X = 2) · P(Y = 1)
(c) E[X 2 Y ] = 0.1(22 3) + 0.1(22 6) + 0.2(52 1) + 0.2(62 3) + 0.1(62 6) +
0.3(62 8) = 138.2 .
x21 +sin(x2 )
3. By integrating the joint pdf over the region A = {(x1 , x2 ) : x21 x2
> 1}.

4. P(M ≤ m) = P(X1 ≤ x1 , . . . , Xn ≤ xn ) = P(X1 ≤ x1 ) · · · P(Xn ≤ xn ) =

F1 (m) · · · Fn (m).
5. Y2 represents the sum of the weights of 6 different people, whereas Y1
represents 6 times the weight of 1 person. The expectation of both Y1
and Y2 is 6 × 75. However, the variance of Y1 is 62 Var(X1 ) = 3600,
whereas the variance of Y2 is 6 times smaller: 6Var(X1 ) = 600. Thus,
Y1 ∼ N(75, 3600) and Y2 ∼ N(75, 600).
6. P(20 ≤ X ≤ 30) ≈ P(20 ≤ Y ≤ 30), with Y ∼ N(100 × 41 , 100 × 1
4 × 34 ) =
N(25, 75/4). We have
20 − 25 30 − 25
P(20 ≤ Y ≤ 30) = P( p ≤Z≤ p ) = P(−1.1547 ≤ Z ≤ 1.1547),
75/4 75/4
where Z ∼ N(0, 1). The cdf of the standard normal distribution in 1.1547
is P(Z ≤ 1.1547) = Φ(1.1547) = 0.875893. Hence, P(−1.1547 ≤ Z ≤
1.1547) = Φ(1.1547) − (1 − Φ(1.1547)) = 2Φ(1.1547) − 1 = 0.752. [The
exact answer is 0.796682.]
7. (a) The Jacobian is
 ∂x ∂x   
v u
det ∂u∂y
∂v
∂y = det = | − u| = u .
∂u ∂v
1 − v −u

(b) The joint pdf of U and V follows from the transformation rule:
fU,V (u, v) = fX,Y (x, y) u = e−(x+y) u = e−u u,
for u ≥ 0 and 0 ≤ v ≤ 1.

Copyright c 2018 D.P. Kroese

106 Exercises and Solutions

(c) By integrating over u we see that V is uniformly distributed over

[0, 1], and by integrating over v we find that fU (u) = ue−u , u ≥ 0.
Thus, the joint pdf of U and V is equal to the marginal pdfs of U
and V , and hence U and V are independent.

8. Note that f (x, y) can be written as the product of f1 (x) = e−x , x ≥ 0 and
f2 (y) = 2e−2y , y ≥ 0. It follows that X and Y are independent random
variables, and that X ∼ Exp(1) and Y ∼ Exp(2).

(a) Because X and Y are independent: E[XY ] = E[X] × E[Y ] = 1 ×

1/2 = 1/2.
(b) Cov(X+Y, X−Y ) = Cov(X, X)−Cov(X, Y )+Cov(Y, X)−Cov(Y, Y ) =
Var(X) − Var(Y ) = 1 − 1/4 = 3/4.

9. (a) Take the average x̄ = (x1 + · · · + xn )/n.

(b) EX̄ = E[(X1 + · · · + Xn )/n] = n1 (E[X1 ] + · · · + E[Xn ]) = 1
n nµ = µ.
1
(c) Var(X̄) = Var[(X1 + · · · + Xn )/n] = n2
(Var[X1 ] + · · · + Var[Xn ]) =
1
n2
nσ 2 = σ 2 /n. This goes to 0 as n → ∞.
(d) N(µ, σ 2 /n).
(e) The larger n is, the more accurately µ can be approximated with x̄.

10. Similar to question 6: P(X ≥ 60) ≈ P(Y ≥ 60), with Y ∼ N(50, 25).
Moreover, P(Y ≥ 60) = P(Z ≥ (60 − 50)/5) = P(Z ≥ 2) = 1 − Φ(2) =
0.02275, where Z ∼ N(0, 1).

11. First draw the graph of the function y = x2 − 4 on the interval [1,3].
Note that the function is increasing from −3 to 5. To find the pdf, first
calculate the cdf:

FY (y) = P(Y ≤ y) = P(X 2 −4 ≤ y) = P(X ≤ y + 4) = FX ( y + 4), −3 ≤ y ≤ 5 .

p p

Now take the derivative with respect to y:

d p p 1 1
fY (y) = FX ( y + 4) = fX ( y + 4)× √ = √ , −3 ≤ y ≤ 5 .
dy 2 4+y 4 4+y

12. U(9, 17).

0.175

0.15

0.125

0.1

0.075

0.05

0.025

13. (a) -4 -2 2 4 6 8 10
√
(b) P(Y ≤ 5) = Φ((5 − 2)/ 5) ≈ 0.9101.

Copyright c 2018 D.P. Kroese

A.6 Answer Set 3 107

√
2 ×5) = N(2, 45). P(1 ≤ Z ≤ 5) = P((1−2)/ 45 ≤
(c) Z ∼ N(3×2−4,√ 3
V ≤ (5 −
√ 2)/ 45), with V√∼ N(0, 1). The latter probability is equal
to Φ(3/ 45) − (1 − Φ(1/ 45)) ≈ 0.2319.

14. See the notes.

15. (a)
2

0
Xn

−2

−4
0 20 40 60 80 100
n
10

0
Yn

−10

−20
0 20 40 60 80 100
n

(b) E[Yn ] = nE[X1 ] = 0, and Var(Yn ) = nVar(X1 ) = n.

(c) Cov(Ym , Yn ) = Cov(Ym , Ym + Xm+1 + · · · + Xn ) = Var(Ym ) + 0 = m.

16. (a) Z8 ∼ N(8 × 75, 8 × 100) =√N(600, 800). P(Z8 ≥ 650) = 1 − P(Z8 ≤
650) = 1 − Φ((650 − 600)/ 800) = 1 − Φ(1.7678) ≈ 0.0385.
√
(b) P(Zn ≥ 650) = 1 − Φ((650 − n75)/ n100). For n = 8 the probability
is 0.0385. For n = 7 it is much smaller than 0.01. So the largest
such n is n = 7.

17. (a) P(0.150−0.004 ≤ X ≤ 0.150+0.004) = P((0.150−0.004−0.151)/0.003 ≤

Z ≤ (0.150 + 0.004 − 0.151)/0.003) = P(−1.66667 ≤ Z ≤ 1) =
Φ(1) − (1 − Φ(1.66667)) ≈ 0.794, where Z ∼ N(0, 1).
(b) Note first that X̄ ∼ N(0.151, (0.003)2 /25). Thus, P(0.150 − 0.004 ≤
X̄ ≤ 0.150 + 0.004) = P((0.150 − 0.004 − 0.151)/(0.003/5) ≤ Z ≤
(0.150 + 0.004 − 0.151)/(0.003/5)) = P(−1.66667 × 5 ≤ Z ≤ 5) =
Φ(5) − (1 − Φ(8.3333)) ≈ 1.
1

0.8

0.6

0.4

0.2

18. (a) 0.5 1 1.5 2

Copyright c 2018 D.P. Kroese

108 Exercises and Solutions

(b) N(10, 20/12), because the expectation of U(0, 1) is 1/2 and the vari-
ance is 1/12.
(c) P(X̄ > 0.6) = P(X1 + · · · + X20 > 12) ≈ P(Y > 12), with Y ∼
N(10,p20/12). Now, P(Y > 12) = 1 − P(Y ≤ 12) = 1 − Φ((12 −
10)/ 20/12) = 1 − Φ(1.5492) ≈ 0.0607.

19. The payout is X 2 , with X ∼ Bin(10, 1/2). The expected payout is EX 2 =

Var(X) + (EX)2 = 2.5 + 52 = 27.5. So, if you pay less than 27.5 dollars
in advance, your expected profit per game is positive.

Copyright c 2018 D.P. Kroese

Appendix B

Sample Exams

B.1 Exam 1

1. Two fair dice are thrown and the sum of the face values, Z say, is noted.

z * * * ...
(a) Give the pmf of Z in table form: [4]
P(Z = z) * * * ...
(b) Calculate the variance of Z. [4]
(c) Consider the game, where a player throws two fair dice, and is paid
Y = (Z − 7)2 dollars, with Z the sum of face values. To enter the
game the player is required to pay 5 dollars.
What is the expected profit (or loss) of the player, if he/she plays
the game 100 times (each time paying 5 dollars to play)? [4]

2. Consider two electrical components, both with an expected lifetime of 3

years. The lifetime of component 1, X say, is assumed to have an expo-
nential distribution, and the lifetime of component 2, Y say, is modeled
via a normal distribution with a standard deviation of 1/2 years.

(a) What is the probability that component 1 is still functioning after

4.5 years, given that it still works after 4 years? [6]
(b) What is the probability that component 2 is still functioning after
4.5 years, given that it still works after 4 years? [6]

3. A sieve with diameter d is used to separate a large number of blueberries

into two classes: small and large.
large

small
d d

Suppose the diameters of the blueberries are normally distributed with

an expectation µ = 1 (cm) and a standard deviation σ = 0.1 (cm).

Copyright c 2018 D.P. Kroese

110 Sample Exams

(a) How large should the diameter of the sieve be, so that the proportion
of large blueberries is 30%? [6]
(b) Suppose that the diameter is chosen such as in (a). What is the
probability that out of 1000 blueberries, fewer than 280 end up in
the “large” class? [6]

4. We draw a random vector (X, Y ) non-uniformly from the triangle (0, 0)–
(1, 0)–(1, 1)
y

0 1 x

in the following way: First we draw X uniformly on [0, 1]. Then, given
X = x we draw Y uniformly on [0, x].

(a) Give the conditional pdf of Y given X = x. Specify where this

conditional pdf is 0. [3]
(b) Find the joint pdf of X and Y . [4]
(c) Calculate the pdf of Y and sketch its graph. [5]

5. We draw n numbers independently and uniformly from the interval [0,1]

and note their sum Sn .

(a) Draw the graph of the pdf of S2 . [3]

(b) What is approximately the distribution of S20 ? [4]
(c) Calculate the probability that the average of the 20 numbers is
greater than 0.6. [5]

B.2 Exam 2

1. Two fair dice are thrown and the smallest of the face values, Z say, is
noted.

(a) Give the probability mass function (pmf) of Z in table form:

z * * * ...
[3]
P(Z = z) * * * ...
(b) Calculate the expectation of 1/Z. [2]
(c) Consider the game, where a player throws two fair dice, and is paid
Z dollars, with Z as above. To enter the game the player is required
to pay 3 dollars.
What is the expected profit (or loss) of the player, if he/she plays
the game 100 times (each time paying 3 dollars to play)? [3]

Copyright c 2018 D.P. Kroese

B.2 Exam 2 111

2. Let U and V be independent random variables, with P(U = 1) = P(V =

1) = 1/4 and P(U = −1) = P(V = −1) = 3/4. Define X = U/V and
Y =U +V.

(a) Give the joint pmf of X and Y . [4]

(b) Calculate Cov(X, Y ). [4]

3. In a binary transmission channel, a 1 is transmitted with probability 1/4

and a 0 with probability 3/4. The conditional probability of receiving a
1 when a 1 was sent is 0.90, the conditional probability of receiving a 0
when a 0 was sent is 0.95.

(a) What is the probability that a 1 is received? [3]

(b) Given that a 1 is received, what is the probability that a 1 was
transmitted? [5]

4. Consider the probability density function (pdf) given by

4 e−4(x−1) , x ≥ 1 ,

f (x) =
0 x<1.

(a) If X is distributed according to this pdf f , what is the expectation

of X? [3]
(b) Specify how one can generate a random variable X from this pdf,
using a random number generator that outputs U ∼ U(0, 1). [5]

5. A certain type of electrical component has an exponential lifetime distri-

bution with an expected lifetime of 1/2 year. When the component fails
it is immediately replaced by a second (new) component; when the second
component fails, it is replaced by a third, etc. Suppose there are 10 such
identical components. Let T be the time that the last of the components
fails.

(a) What is the expectation and variance of T ? [3]

(b) Approximate, using the central limit theorem and the table of the
standard normal cdf, the probability that T exceeds 6 years. [3]
(c) What is the exact distribution of T ? [2]

Copyright c 2018 D.P. Kroese

112 Sample Exams

Copyright c 2018 D.P. Kroese

Appendix C

Summary of Formulas

1. Sum rule: P( i Ai ) = i P(Ai ),

S P

when A1 , A2 , . . . are disjoint.

2. P(Ac ) = 1 − P(A).

3. P(A ∪ B) = P(A) + P(B) − P(A ∩ B).

4. Cdf of X: F (x) = P(X ≤ x), x ∈ R.

5. Pmf of X: (discrete r.v.) f (x) = P(X = x).

6. Pdf of X: (continuous r.v.) f (x) = F 0 (x).

7. For a discrete r.v. X:

X
P(X ∈ B) = P(X = x).
x∈B

8. For a continuous r.v. X with pdf f :

Z
P(X ∈ B) = f (x) dx.
B

Rx
9. In particular (continuous), F (x) = −∞ f (u) du.

10. Similar results 7-8 hold for random vectors, e.g.

ZZ
P((X, Y ) ∈ B) = fX,Y (x, y) dx dy .
B

R
11. Marginal from joint pdf: fX (x) = fX,Y (x, y) dy.

12. Important discrete distributions:

Copyright c 2018 D.P. Kroese

114 Summary of Formulas

Distr. pmf x∈
Ber(p) px (1

x − p) n−x
1−x {0, 1}
n
Bin(n, p) x p (1 − p) {0, 1, . . . , n}
−λ λx
Poi(λ) e x! {0, 1, . . .}
G(p) p(1 − p)x−1 {1, 2, . . .}
−r
(xr )(Nn−x )
Hyp(n, r, N ) {0, . . . , n}
(Nn )

13. Important continuous distributions:

Distr. pdf x∈
1
U[a, b] b−a [a, b]
Exp(λ) λ e−λx R+
λα xα−1 e−λx
Gam(α, λ) Γ(α) R+
2
− 12 ( x−µ
σ )
N(µ, σ 2 ) √1
σ 2π
e R

P(A∩B)
14. Conditional probability: P(A | B) = P(B) .

15. Law of total probability:

P(A) = ni=1 P(A | Bi ) P(Bi ),
P
with B1 , B2 , . . . , Bn a partition of Ω.

16. Bayes’ Rule: P(Bj |A) = PnP(Bj ) P(A|Bj ) .

i=1 P(Bi ) P(A|Bi )

17. Product rule:

P(A1 · · · An ) = P(A1 ) P(A2 | A1 ) · · · P(An | A1 · · · An−1 ).

18. Memoryless property (Exp and G distribution):

P(X > s + t | X > s) = P(X > t), ∀s, t.

19. Independent events: P(A ∩ B) = P(A) P(B) .

20. Independent r.v.’s: (discrete)

Qn
P(X1 = x1 , . . . , Xn = xn ) = k=1 P(Xk = xk ) .

21. Independent r.v.’s: (continuous)

Qn
fX1 ,...,Xn (x1 , . . . , xn ) = k=1 fXk (xk ) .

22. Expectation (discr.): EX = x x P(X = x).

P

23. (of function) E g(X) = x g(x) P(X = x) .

P

24. Expectation (cont.): EX = x f (x) dx.

R

25. (of function) E g(X) = g(x)f (x) dx,

R

Copyright c 2018 D.P. Kroese

 115

26. Similar results 18–21 hold for random vectors.

27. Expected sum : E(aX + bY ) = a EX + b EY .

28. Expected product (only if X, Y independent):

E[X Y ] = EX EY.
EX
29. Markov inequality: P(X ≥ x) ≤ x .

30. EX and Var(X) for various distributions:

EX Var(X)
Ber(p) p p(1 − p)
Bin(n, p) np np(1 − p)
1 1−p
G(p) p p2
Poi(λ) λ λ
−n
Hyp(n, pN, N ) np np(1 − p) N
N −1
a+b (b−a)2
U(a, b) 2 12
1 1
Exp(λ) λ λ2
α α
Gam(α, λ) λ λ2
N(µ, σ 2 ) µ σ2

31. n-th moment: EX n .

32. Covariance: cov(X, Y ) = E(X − EX)(Y − EY ).

33. Properties of Var and Cov:

Var(X) = EX 2 − (EX)2 .
Var(aX + b) = a2 Var(X).
cov(X, Y ) = EXY − EXEY .
cov(X, Y ) = cov(Y, X).
cov(aX + bY, Z) = a cov(X, Z) + b cov(Y, Z).
cov(X, X) = Var(X).
Var(X+Y ) = Var(X)+Var(Y )+2 cov(X, Y ).
X and Y independent =⇒ cov(X, Y ) = 0.

34. Probability Generating Function (PGF):

∞
X
G(z) := E z N
= P(N = n)z n , |z| < 1 .
n=0

35. PGFs for various distributions:

Ber(p) 1 − p + zp
Bin(n, p) (1 − p + zp)n
zp
G(p) 1−z (1−p)
Poi(λ) e−λ(1−z)

Copyright c 2018 D.P. Kroese

116 Summary of Formulas

36. P(N = n) = 1
n! G(n) (0) . (n-th derivative, at 0)

37. EN = G0 (1)

38. Var(N ) = G00 (1) + G0 (1) − (G0 (1))2 .

39. Moment Generating Function (MGF):

Z ∞
M (s) = E e =
sX
esx f (x) dx ,
−∞

s ∈ I ⊂ R, for r.v.’s X for which all moments exist.

40. MGFs for various distributions:

ebs −eas
U(a, b)
s(b−a)
α
λ
Gam(α, λ) λ−s
2 2
N(µ, σ 2 ) esµ+σ s /2

41. Moment property: EX n = M (n) (0).

42. MX+Y (t) = MX (t) MY (t), ∀t, if X, Y independent.

43. If Xi ∼ N(µi , σi2 ), i = 1, 2, . . . , n (independent),

a + ni=1 bi Xi ∼ N a + ni=1 bi µi , ni=1 b2i σi2 .
P P P

44. Conditional pmf/pdf

fX,Y (x,y)
fY | X (y | x) := fX (x) , y ∈ R.

45. The corresponding conditional expectation (discrete case):

E[Y | X = x] = y y fY | X (y | x).
P

fX (A−1 z)
46. Linear transformation: fZ (z) = |A| .

fX (x)
47. General transformation: fZ (z) = , with x = g −1 (z), where
|Jx (g)|
|Jx (g)| is the Jacobian of g evaluated at x.

48. Pdf of the multivariate normal distribution:

1 1 T −1
fZ (z) = p e− 2 (z−µ) Σ (z−µ) .
n
(2π) |Σ|

Σ is the covariance matrix, and µ the mean vector.

49. If X is a column vector with independent N(0, 1) components, and B is a

matrix with Σ = BB T (such a B can always be found), then Z = µ+BX
has a multivariate normal distribution with mean vector µ and covariance
matrix Σ.

Copyright c 2018 D.P. Kroese

 117

50. Weak Law of Large Numbers:

 
Sn
lim P − µ >  = 0, ∀ .
n→∞ n

51. Strong Law of Large Numbers:

 
Sn
P lim =µ =1.
n→∞ n

52. Central Limit Theorem:

 
Sn − nµ
lim P √ ≤ x = Φ(x),
n→∞ σ n

where Φ is the cdf of the standard normal distribution.

53. Normal Approximation to Binomial: If X ∼ Bin(n, p), then, for

large n, P(X ≤ k) ≈ P(Y ≤ k), where Y ∼ N(np, np(1 − p)) .

Other Mathematical Formulas

1. Factorial. n! = n (n − 1) (n − 2) · · · 1. Gives the number of permutations

(orderings) of {1, . . . , n}.

2. Binomial coefficient. nk = k! (n−k)!

n!


. Gives the number combinations (no
order) of k different numbers from {1, . . . , n}.

3. Newton’s binomial theorem: (a + b)n = nk=0 nk ak bn−k .

P

1−an+1
4. Geometric sum: 1 + a + a2 + · · · + an = 1−a (a 6= 1).
1
If |a| < 1 then 1 + a + a2 + · · · = 1−a .

5. Logarithms:

(a) log(x y) = log x + log y.

(b) elog x = x.

6. Exponential:
x2 x3
(a) ex = 1 + x + 2! + 3! + ···.
x n
(b) ex = limn→∞ 1 +


n .
(c) ex+y = ex ey .

7. Differentiation:

(a) (f + g)0 = f 0 + g 0 ,
(b) (f g)0 = f 0 g + f g 0 ,
 0 0 g0
(c) fg = f g−f g2
.

Copyright c 2018 D.P. Kroese

118 Summary of Formulas

d n n−1 .
(d) dx x = n x
d x x
(e) dx e = e .
d 1
(f) dx log(x) = x .

8. Chain rule: (f (g(x)))0 = f 0 (g(x)) g 0 (x).

Rb
9. Integration: a f (x) dx = [F (x)]ba = F (b) − F (a), where F 0 = f .
Rb Rb
10. Integration by parts: a f (x) G(x) dx = [F (x) G(x)]ba − a F (x) g(x) dx .
(Here F 0 = f and G0 = f .)

11. Jacobian: Let x = (x1 , . . . , xn ) be an n-dimensional vector, and g(x) =

(g1 (x), . . . , gn (x)) be a function from Rn to Rn . The matrix of Jacobi is
the matrix of partial derivatives: (∂gi /∂xj ). The corresponding determi-
nant is called the Jacobian. In the neighbourhood of any fixed point, g
behaves like a linear transformation specified by the matrix of Jacobi at
that point.
R∞
12. Γ function: Γ(α) = 0 uα−1 e−u du, α > 0. Γ(α + 1) = α Γ(α), for
√
α ∈ R+ . Γ(n) = (n − 1)! for n = 1, 2, . . .. Γ(1/2) = π.

Copyright c 2018 D.P. Kroese

Appendix D

Statistical Tables

– Standard normal distribution

– Binomial distribution with p = 1/2

 Standard normal distribution

This table gives the cummulative distribution function (cdf) Φ of a N(0, 1)-
distributed random variable Z.
Z z
1 2
Φ(z) = P(Z ≤ z) = √ e−x /2 dx.
2π −∞

The last column gives the probability density function (pdf) ϕ of the N(0, 1)-
distribution
1 2
φ(z) = √ e−z /2 .
2π

z .00 .01 .02 .03 .04 .05 .06 .07 .08 .09 φ(z)
0.0 5000 5040 5080 5120 5160 5199 5239 5279 5319 5359 0.3989
0.1 5398 5438 5478 5517 5557 5596 5636 5675 5714 5753 0.3970
0.2 5793 5832 5871 5910 5948 5987 6026 6064 6103 6141 0.3910
0.3 6179 6217 6255 6293 6331 6368 6406 6443 6480 6517 0.3814
0.4 6554 6591 6628 6664 6700 6736 6772 6808 6844 6879 0.3683
0.5 6915 6950 6985 7019 7054 7088 7123 7157 7190 7224 0.3521
0.6 7257 7291 7324 7357 7389 7422 7454 7486 7517 7549 0.3332
0.7 7580 7611 7642 7673 7704 7734 7764 7794 7823 7852 0.3123
0.8 7881 7910 7939 7967 7995 8023 8051 8078 8106 8133 0.2897
0.9 8159 8186 8212 8238 8264 8289 8315 8340 8365 8389 0.2661
1.0 8413 8438 8461 8485 8508 8531 8554 8577 8599 8621 0.2420
1.1 8643 8665 8686 8708 8729 8749 8770 8790 8810 8830 0.2179
1.2 8849 8869 8888 8907 8925 8944 8962 8980 8997 9015 0.1942
1.3 9032 9049 9066 9082 9099 9115 9131 9147 9162 9177 0.1714
1.4 9192 9207 9222 9236 9251 9265 9279 9292 9306 9319 0.1497
1.5 9332 9345 9357 9370 9382 9394 9406 9418 9429 9441 0.1295
1.6 9452 9463 9474 9484 9495 9505 9515 9525 9535 9545 0.1109
1.7 9554 9564 9573 9582 9591 9599 9608 9616 9625 9633 0.0940
1.8 9641 9649 9656 9664 9671 9678 9686 9693 9699 9706 0.0790
1.9 9713 9719 9726 9732 9738 9744 9750 9756 9761 9767 0.0656
2.0 9772 9778 9783 9788 9793 9798 9803 9808 9812 9817 0.0540
2.1 9821 9826 9830 9834 9838 9842 9846 9850 9854 9857 0.0440
2.2 9861 9864 9868 9871 9875 9878 9881 9884 9887 9890 0.0355
2.3 9893 9896 9898 9901 9904 9906 9909 9911 9913 9916 0.0283
2.4 9918 9920 9922 9925 9927 9929 9931 9932 9934 9936 0.0224
2.5 9938 9940 9941 9943 9945 9946 9948 9949 9951 9952 0.0175
2.6 9953 9955 9956 9957 9959 9960 9961 9962 9963 9964 0.0136
2.7 9965 9966 9967 9968 9969 9970 9971 9972 9973 9974 0.0104
2.8 9974 9975 9976 9977 9977 9978 9979 9979 9980 9981 0.0079
2.9 9981 9982 9982 9983 9984 9984 9985 9985 9986 9986 0.0060
3.0 9987 9987 9987 9988 9988 9989 9989 9989 9990 9990 0.0044
3.1 9990 9991 9991 9991 9992 9992 9992 9992 9993 9993 0.0033
3.2 9993 9993 9994 9994 9994 9994 9994 9995 9995 9995 0.0024
3.3 9995 9995 9995 9996 9996 9996 9996 9996 9996 9997 0.0017
3.4 9997 9997 9997 9997 9997 9997 9997 9997 9997 9998 0.0012
3.5 9998 9998 9998 9998 9998 9998 9998 9998 9998 9998 0.0009
3.6 9998 9998 9999 9999 9999 9999 9999 9999 9999 9999 0.0006

Example: Φ(1.65) = P(Z ≤ 1.65) = 0.9505

 Binomial distribution with p = 1/2
c
1 n
Below is a list of F (c) = P(X ≤ c) = n 1
P



k 2
if X ∼ Bin n, 2
.
k=0
(Only the first 4 digits are given, and probabilities ≥ 0.99995 have been omitted.)

n c F (c) n c F (c) n c F (c) n c F (c) n c F (c)

9 0 0020 14 0 0001 17 13 9936 21 0 0000 23 18 9987
1 0195 1 0009 14 9988 1 0000 19 9998
2 0898 2 0065 15 9999 2 0001
3 2539 3 0287 3 0007 24 0 0000
4 5000 4 0898 18 0 0000 4 0036 1 0000
5 7461 5 2120 1 0001 5 0133 2 0000
6 9102 6 3953 2 0007 6 0392 3 0001
7 9805 7 6047 3 0038 7 0946 4 0008
8 9980 8 7880 4 0154 8 1917 5 0033
9 9102 5 0481 9 3318 6 0113
10 0 0010 10 9713 6 1189 10 5000 7 0320
1 0107 11 9935 7 2403 11 6682 8 0758
2 0547 12 9991 8 4073 12 8083 9 1537
3 1719 13 9999 9 5927 13 9054 10 2706
4 3770 10 7597 14 9608 11 4194
5 6230 15 0 0000 11 8811 15 9867 12 5806
6 8281 1 0005 12 9519 16 9964 13 7294
7 9453 2 0037 13 9846 17 9993 14 8463
8 9893 3 0176 14 9962 18 9999 15 9242
9 9990 4 0592 15 9993 16 9680
5 1509 16 9999 22 0 0000 17 9887
11 0 0005 6 3036 1 0000 18 9967
1 0059 7 5000 19 0 0000 2 0001 19 9992
2 0327 8 6964 1 0000 3 0004 20 9999
3 1133 9 8491 2 0004 4 0022
4 2744 10 9408 3 0022 5 0085 25 0 0000
5 5000 11 9824 4 0096 6 0262 1 0000
6 7256 12 9963 5 0318 7 0669 2 0000
7 8867 13 9995 6 0835 8 1431 3 0001
8 9673 7 1796 9 2617 4 0005
9 9941 16 0 0000 8 3238 10 4159 5 0020
10 9995 1 0003 9 5000 11 5841 6 0073
2 0021 10 6762 12 7383 7 0216
12 0 0002 3 0106 11 8204 13 8569 8 0539
1 0032 4 0384 12 9165 14 9331 9 1148
2 0193 5 1051 13 9682 15 9738 10 2122
3 0730 6 2272 14 9904 16 9915 11 3450
4 1938 7 4018 15 9978 17 9978 12 5000
5 3872 8 5982 16 9996 18 9996 13 6550
6 6128 9 7728 19 9999 14 7878
7 8062 10 8949 20 0 0000 15 8852
8 9270 11 9616 1 0000 23 0 0000 16 9461
9 9807 12 9894 2 0002 1 0000 17 9784
10 9968 13 9979 3 0013 2 0000 18 9927
11 9998 14 9997 4 0059 3 0002 19 9980
5 0207 4 0013 20 9995
13 0 0001 17 0 0000 6 0577 5 0053 21 9999
1 0017 1 0001 7 1316 6 0173
2 0112 2 0012 8 2517 7 0466
3 0461 3 0064 9 4119 8 1050
4 1334 4 0245 10 5881 9 2024
5 2905 5 0717 11 7483 10 3388
6 5000 6 1662 12 8684 11 5000
7 7095 7 3145 13 9423 12 6612
8 8666 8 5000 14 9793 13 7976
9 9539 9 6855 15 9941 14 8950
10 9888 10 8338 16 9987 15 9534
11 9983 11 9283 17 9998 16 9827
12 9999 12 9755 17 9947