Lecture 3

Inverse transform method

• How does one transform a sample of the uniform[0,1] random
variable into a sample of a given distribution ?

• Methods for transformation

– Inverse transform method

– Convolution method

– Composition method

– Acceptance-rejection method
Inverse transform method 3–2

Generating random numbers

Problem: Generate sample of a random variable X with a given

density f . (The sample is called a random variate)

What does this mean ?

Answer: Develop an algorithm such that if one used it repeatedly
(and independently) to generate a sequence of samples
X1, X2, . . . , Xn then as n becomes large, the proportion of samples
that fall in any interval [a, b] is close to P(X ∈ [a, b]), i.e.

#{Xi ∈ [a, b]}

≈ P(X ∈ [a, b])
n

Solution: 2-step process

• Generate a random variate uniformly distributed in [0, 1] .. also

called a random number

• Use an appropriate transformation to convert the random number

to a random variate of the correct distribution

why is this approach good ?

Answer: Focus on generating samples from ONE distribution only.

Inverse transform method 3–3

Pseudo-random numbers
Many different methods of generating a (uniform[0,1]) random
number ...

1. physical methods : roulette wheel, balls for the lottery, electrical

circuits etc.
2. numerical/arithmetic : sequential method ... each new number is
a deterministic function of the past numbers

For simulation ... numerical method works best

• numbers generated by the numerical method are never random !

• enough that the numbers “look” uniformly distributed and have

no correlation between them i.e. pass statistical tests

• All we want is that the central limit theorem should kick in ...

Properties that psuedo-random number generators should possess

1. it should be fast and not memory intensive

2. be able to reproduce a given stream of random numbers ... why ?
• debugging or verification of computer programs
• may want to use identical numbers to compare different
systems
3. provision for producing several different independent “streams” of
random numbers
Inverse transform method 3–4

Linear Congruential generators

• Will not focus too much on generating random numbers ... all
simulators and languages have good generators

• A good example ... Linear congruential generators

Zi
Zi = (aZi−1 + c) (mod m) i ≥ 1, Ui = .
m
modulus m, multiplier a, increment c, and seed Z0 are
nonnegative
a < m, and c, Z0 < m
Example
Zi = (5Zi−1 + 3)(mod16) Zi = (5Zi−1 + 6)(mod16)
i Zi i Zi i Zi i Zi i Zi i Zi i Zi i Zi
0 7 5 10 10 9 15 4 0 7 5 1 10 3 15 13
1 6 6 15 11 0 16 7 1 9 6 11 11 5 16 7
2 1 7 12 12 3 17 6 2 3 7 13 12 15 17 9
3 8 8 15 13 2 18 1 3 5 8 7 13 1 18 3
4 11 9 14 14 13 19 8 4 15 9 9 14 11 19 5

• Several other types of generators

more general congruences Zi = g(Zi−1, Zi−2 , . . .)(mod m).
a popular example ... Tautsworth generator
Zi = (c1Zi−1 + c2Zi−2 + . . . + cq Zi−q )(mod 2)

Assume that we have an algorithm (program) that generates

independent samples of uniform[0,1] random variable.
Inverse transform method 3–5

Generating random variates

• Inverse transform method

• Composition approach

• Convolution method

• Acceptance-Rejection technique
Inverse transform method 3–6

Continuous random variables

• Generate a continuous random variable X ∼ F as follows :

1. Generate a uniform random variable U

2. Set X = F −1 (U )

(Assumption: The inverse F −1(x) exists ...)

Using the inverse and hence ... Inverse Transform Method
• Proof: Have to show that the CDF of the samples produced by
this method is precisely F (x).

P(X ≤ x) = P(F −1 (U ) ≤ x)
= P(U ≤ F (x)) (1)
= F (x) (2)
where
– (1) follows by the fact that F is an increasing function
– (2) follows from the fact 0 ≤ F (x) ≤ 1 and the CDF of a
uniform FU (y) = y for all y ∈ [0, 1]
• Algorithm:
– Given: the CDF F (x) or the density f (x). If density given
then first integrate to get CDF. (Most frequent error:
incorrect limits on the integration)
– Set F (X) = U and solve for X in terms of U . Have to make
sure that the solution X lies in the correct range.
Inverse transform method 3–7

Example : exp(λ)

• The density f (x) = λe−λx and the cdf F (x) = 1 − e−λx.

• Set F (X) = U and solve for U

1 − e−λX = U
e−λX = 1 − U
1
X = − log(1 − U )
λ
• Algorithm :

1. Generate random number U

2. Set X = − λ1 log(1 − U )

• But if U is uniform[0,1] then 1 − U is also uniform[0,1] so one

might as well define
1
X = − log(U )
λ
Inverse transform method 3–8

Inverse transform method : Discrete

random variables

• Want to generate a discrete random variable X with pmf

P(X = xi) = pi, i = 1, . . . , m

• Consider the following algorithm

1. Generate a random number U

2. Transform U into X as follows,
j−1 j
X = xj if pi ≤ U < pi
X X

i=1 i=1

• Proof the algorithm works ...

j−1 j
P(X = xj ) = P( pi ≤ U < pi )
X X

i=1 i=1
j j−1
= pi − pi = p j
X X

i=1 i=1

QED
Inverse transform method 3–9

• Suppose x1 < x2 < . . . < xm then the cdf of X FX is







0, x < x1 ,
 P
j
FX (x) = 
 i=1 pi, xj ≤ x < xj+1 , j ≤ m − 1
1, x ≥ xm





1
PSfrag replacements
F (x5 )

F (x4 )
U
F (x3 )

F (x2 )

F (x1 ) −1
X = FX (U ) = x4

x1 x2 x3 x4 x5 x6

Thus ... X = xj if FX (xj−1) ≤ U < FX (xj )

• If one defines the generalized inverse FX−1 by

FX−1(x) = min{y | FX (y) ≥ x}

then ... X = FX−1(U )

Inverse transform method 3 – 10

Examples : simple distribution

• Want to simulate the random variable X with the pmf

p1 = 0.2, p2 = 0.15, p3 = 0.25, p4 = 0.40

• Algorithm :

1. Generate a random number U

2. If U < p1 = 0.2, set X ← x1 and stop
3. If U < p1 + p2 = 0.35, set X ← x2 and stop
4. If U < p1 + p2 + p3 = 0.60, set X ← x3 and
stop
5. Else set X ← x4 and stop

• Is this the most efficient way of generating the random variate ?

The amount of time is proportional to the number of intervals

one must search. Thus, it helps to consider the values xj in
decreasing order of pi’s.
Inverse transform method 3 – 11

• More efficient algorithm :

1. Generate a random number U

2. If U < p4 = 0.4, set X ← x4 and stop
3. If U < p4 + p3 = 0.65, set X ← x3 and stop
4. If U < p4 + p3 + p1 = 0.85, set X ← x1 and
stop
5. Else set X ← x2 and stop
Inverse transform method 3 – 12

Example : Geometric distribution

• In this case get a closed form expression for FX−1 (U ).

• The pmf of the geometric random variable X is :

P(X = i) = pq i−1, i ≥ 1, q = (1 − p)

• The cdf of the random variable

F (i) = P(X ≤ i),
= 1 − P(X > i),
= 1 − P(first i trials are failures) = 1 − q i.

• Transformation : X = j if
F (j − 1) ≤ U < F (j) ⇔ 1 − q j−1 ≤ U < 1 − q j
⇔ q j < 1 − U ≤ q j−1
Therefore
X = min{j | q j < 1 − U }
= min{j
 ¯
| j log(q) < log(1− U )}

 ¯
¯
log(1 − U ) 

= min  j j>
¯
¯
 ¯¯ log(q)  

i.e.
log(1 − U )
X=d e
log(1 − p)

where dze is the smallest integer larger than z.

Inverse transform method 3 – 13

Example : Binomial random variable

• No closed form solution ... algorithm

• the pmf is
 
n i i
pi = P(X = i) = 
  p (1 − p) , i = 0, . . . , n
i

Therefore ...

n − i  p 
  

pi =  pi−1
i+1 1−p
• Algorithm

1. Generate a random number U

2. Set i ← 0, P ← (1 − p)n, F ← P ,
3. If U < F , set X ← i and stop
µ ¶µ ¶
n−i p
4. Set P ← i+1 1−p P, F ← F +P, i ← i+1
5. Goto step 3
Inverse transform method 3 – 14

Disadvantages and advantages

Disadvantages:

1. requires a closed form expression for F (x)

2. speed ... often very slow because a number of comparisons

required

Advantages:

Inverse transform method preserves monotonicity and correlation

which helps in

1. Variance reduction methods ...

2. Generating truncated distributions ...

3. Order statistics ...

Inverse transform method 3 – 15

Convolution method

• Suppose X is a sum of independent random variables

Z1, Z2, . . . , Zm, i.e.

X = Z 1 + Z2 + . . . + Z m

where Zi ∼ Fi and are all independent.

• Algorithm :

1. Generate m random numbers U1, U2, ..., Um

2. Inverse transform method : Zi = Fi−1 (Ui)
Pm
3. Set X = i=1 Zi

• Why is this called the convolution method ?

Let Z1, Z2 independent with densities f1 and f2 resp.

Let X = Z1 + Z2 then the density fX of X is given by
Z
fX (x) = f1(u)f2(x − u)du
Operation on f1 and f2 called convolution. Denoted by f1 ∗ f2.
Method samples from the density f1 ∗ f2 ∗ . . . ∗ fm.
Inverse transform method 3 – 16

Example : Erlang(λ,m)

• Problem: Generate a sample from Erlang(λ, m) distribution.

The density of fλ,m of Erlang(λ, m) dist. is

λ(λx)m−1 −λx
fλ,m (x) = e
(m − 1)!

• Fact : If Zi ∼ exp(λ) and independent, then

X = Z1 + Z2 + . . . + Zm is Erlang(λ, m)

• All set for the convolution method ...

1. Generate m random numbers U1, U2, ..., Um

2. Set Zi = − λ1 log(Ui)
Pm
3. Set X = i=1 Zi

• Not very efficient ... need m random numbers to generate 1

sample

• Will discuss a more efficient method later.