Random Variate Generation

Christos Alexopoulos and Dave Goldsman

Georgia Institute of Technology, Atlanta, GA, USA

5/21/10

Alexopoulos and Goldsman 5/21/10 1 / 73

 Outline

1 Introduction
2 Inverse Transform Method
3 Cutpoint Method
4 Convolution Method
5 Acceptance-Rejection Method
6 Composition Method
7 Special-Case Techniques
8 Multivariate Normal Distribution
9 Generating Stochastic Processes

Alexopoulos and Goldsman 5/21/10 2 / 73

 Introduction

Goal: Use U(0, 1) numbers to generate observations (variates) from

other distributions, and even stochastic processes.

Discrete distributions, like Bernoulli, Binomial, Poisson, and

empirical
Continuous distributions like exponential, normal (many ways),
and empirical
Multivariate normal
Autoregressive moving average time series
Waiting times
etc., etc.

Alexopoulos and Goldsman 5/21/10 3 / 73

 Inverse Transform Method

Outline

1 Introduction
2 Inverse Transform Method
3 Cutpoint Method
4 Convolution Method
5 Acceptance-Rejection Method
6 Composition Method
7 Special-Case Techniques
8 Multivariate Normal Distribution
9 Generating Stochastic Processes

Alexopoulos and Goldsman 5/21/10 4 / 73

 Inverse Transform Method

Inverse Transform Method

Inverse Transform Theorem: Let X be a continuous random variable

with cdf F (x). Then
F (X) ∼ U(0, 1).
Proof: Let Y = F (X) and suppose that Y has cdf G(y). Then

G(y) = P(Y ≤ y) = P(F (X) ≤ y)

= P(X ≤ F −1 (y)) = F (F −1 (y))
= y. 2

Alexopoulos and Goldsman 5/21/10 5 / 73

 Inverse Transform Method

In the above, we can define the inverse cdf by

F −1 (u) = inf[x : F (x) ≥ u] u ∈ [0, 1].

This representation can be applied to continuous or discrete or mixed

distributions (see figure).

Let U ∼ U(0, 1). Then the random variable F −1 (U ) has the same
distribution as X.

Here is the inverse transform method for generating a RV X having cdf

F (x):
1 Sample U from U(0, 1).
2 Return X = F −1 (U ).

Alexopoulos and Goldsman 5/21/10 6 / 73

 Inverse Transform Method

Alexopoulos and Goldsman 5/21/10 7 / 73

 Inverse Transform Method

Discrete Example: Suppose


 −1 w.p. 0.6


X ≡ 2.5 w.p. 0.3

 4 w.p. 0.1


x P (X = x) F (x) U(0, 1)’s

−1 0.6 0.6 [0.0,0.6]
2.5 0.3 0.9 (0.6,0.9]
4 0.1 1.0 (0.9,1.0]

Thus, if U = 0.63, we take X = 2.5. 2

Alexopoulos and Goldsman 5/21/10 8 / 73

 Inverse Transform Method

Example: The Unif(a, b) distribution.

F (x) = (x − a)/(b − a), a ≤ x ≤ b.

Solving (X − a)/(b − a) = U for X, we get X = a + (b − a)U . 2

Example: The discrete uniform distribution on {a, a + 1, . . . , b} with

1
P (X = k) = , a, a + 1, . . . , b.
b−a+1

Clearly,
X = a + ⌊(b − a + 1)U ⌋,
where ⌊·⌋ is the floor function. 2

Alexopoulos and Goldsman 5/21/10 9 / 73

 Inverse Transform Method

Example: The exponential distribution.

F (x) = 1 − e−λx , x > 0.

Solving F (X) = U for X,

1 1
X = − ℓn(1 − U ) or X = − ℓn(U ). 2
λ λ

Example: The Weibull distribution.

β
F (x) = 1 − e−(λx) , x > 0.

Solving F (X) = U for X,

1 1
X= [−ℓn(1 − U )]1/β or X = [−ℓn(U )]1/β 2.
λ λ

Alexopoulos and Goldsman 5/21/10 10 / 73

 Inverse Transform Method

Example: The triangular (0,1,2) distribution has pdf

(
x if 0 ≤ x < 1
f (x) =
2 − x if 1 ≤ x ≤ 2.

The cdf is (
x2 /2 if 0 ≤ x < 1
F (x) =
1 − (x − 2)2 /2 if 1 ≤ x ≤ 2.

√
(a) If U < 1/2, we solve X 2 /2 = U to get X = 2U .
(b) If U ≥ 1/2, the only root of 1 − (X − = U in [1, 2] is 2)2 /2
p
X = 2 − 2(1 − U ).
√
Thus, for example, if U = 0.4, we take X = 0.8. 2

Remark: Do not replace U by 1 − U here!

Alexopoulos and Goldsman 5/21/10 11 / 73
 Inverse Transform Method

Example: The standard normal distribution. Unfortunately, the inverse

cdf Φ−1 (·) does not have an analytical form. This is often a problem
with the inverse transform method.

Easy solution: Do a table lookup. E.g., If U = 0.975, then

Z = Φ−1 (U ) = 1.96. 2

Crude portable approximation (Banks et al.): The following

approximation gives at least one decimal place of accuracy for
0.00134 ≤ U ≤ 0.98865:

U 0.135 − (1 − U )0.135
Z = Φ−1 (U ) ≈ . 2
0.1975

Alexopoulos and Goldsman 5/21/10 12 / 73

 Inverse Transform Method

Here’s a better portable solution: The following approximation has

absolute error ≤ 0.45 × 10−3 :

c0 + c1 t + c2 t2
 
Z = sign(U − 1/2) t − ,
1 + d1 t + d2 t2 + d3 t3

where sign(x) = 1, 0, −1 if x is positive, zero, or negative, respectively,

t = {−ℓn[min(U, 1 − U )]2 }1/2 ,

and
c0 = 2.515517, c1 = 0.802853, c2 = 0.010328,
d1 = 1.432788, d2 = 0.189269, d3 = 0.001308.

Alexopoulos and Goldsman 5/21/10 13 / 73

 Inverse Transform Method

Example: The geometric distribution with pmf

P (X = k) = (1 − p)k−1 p, k = 1, 2, . . . .

The cdf is F (k) = 1 − (1 − p)k , k = 1, 2, . . ..

Hence,
 
k ℓn(1 − U )
X = min[k : 1 − (1 − p) ≥ U ] = ,
ℓn(1 − p)

where ⌈·⌉ is the “ceiling” function.

For instance, if p = 0.3 and U = 0.72, we obtain

 
ℓn(0.28)
X = = 4. 2
ℓn(0.7)

Alexopoulos and Goldsman 5/21/10 14 / 73

 Cutpoint Method

Outline

1 Introduction
2 Inverse Transform Method
3 Cutpoint Method
4 Convolution Method
5 Acceptance-Rejection Method
6 Composition Method
7 Special-Case Techniques
8 Multivariate Normal Distribution
9 Generating Stochastic Processes

Alexopoulos and Goldsman 5/21/10 15 / 73

 Cutpoint Method

Cutpoint Method

Suppose we want to generate from the discrete distribution

P (X = k) = pk , k = a, a + 1, . . . , b

with large b − a. Let

qk = P (X ≤ k), k = a, a + 1, . . . , b.

For fixed m, the cutpoint method of Fishman and Moore computes and
stores the cutpoints
 
j−1
Ij = min k : qk > , j = 1, . . . , m.
m

These cutpoints help us scan through the list of possible k-values

much more quickly than regular inverse transform.
Alexopoulos and Goldsman 5/21/10 16 / 73
 Cutpoint Method

Here is the algorithm that computes the cutpoints . . .

Algorithm CMSET
j ← 0, k ← a − 1, and A ← 0
While j < m:
While A ≤ j:
k ←k+1
A ← mqk
j ←j+1
Ij ← k

Alexopoulos and Goldsman 5/21/10 17 / 73

 Cutpoint Method

Once the cutpoints are computed, we can use the cutpoint method.

Algorithm CM
Generate U from U(0, 1)
L ← ⌊mU ⌋ + 1
X ← IL
While U > qX : X ← X + 1

In short, this algorithm selects an integer L = ⌊mU ⌋ + 1 and starts the

search at the value IL . Its correctness results from the fact that

P (IL ≤ X ≤ IL+1 ) = 1 (Im+1 = b).

Alexopoulos and Goldsman 5/21/10 18 / 73

 Cutpoint Method

Let E(Cm ) be the expected number of comparisons until Algorithm CM

terminates. Given L, the maximum number of required comparisons is
IL+1 − IL + 1. Hence,

I2 − I1 + 1 Im+1 − Im + 1
E(Cm ) ≤ + ··· +
m m
b − I1 + m
= .
m

Note that if m ≥ b, then E(Cm ) ≤ (2m − 1)/m ≤ 2.

Alexopoulos and Goldsman 5/21/10 19 / 73

 Cutpoint Method

Example: Consider the distribution

k 1 2 3 4 5 6 7 8
pk .01 .04 .07 .15 .28 .19 .21 .05
qk .01 .05 .12 .27 .55 .74 .95 1

For m = 8, we have the following cutpoints:

I1 = min[i : qi > 0] = 1
I2 = min[i : qi > 1/8] = 4
I3 = min[i : qi > 2/8] = 4
I4 = min[i : qi > 3/8] = 5 = I5
I6 = min[i : qi > 5/8] = 6
I7 = min[i : qi > 6/8] = 7 = I8
I9 = 8

For U = 0.219, we have L = ⌊8(0.219)⌋ + 1 = 2, and

X = min[i : I2 ≤ i ≤ I3 , qi ≥ 0.219] = 4. 2
Alexopoulos and Goldsman 5/21/10 20 / 73
 Convolution Method

Outline

1 Introduction
2 Inverse Transform Method
3 Cutpoint Method
4 Convolution Method
5 Acceptance-Rejection Method
6 Composition Method
7 Special-Case Techniques
8 Multivariate Normal Distribution
9 Generating Stochastic Processes

Alexopoulos and Goldsman 5/21/10 21 / 73

 Convolution Method

Convolution Method

Convolution refers to adding things up.

Example: Binomial(n, p).

Pn
Suppose X1 , . . . , Xn are iid Bern(p). Then Y = i=1 Xi ∼ Bin(n, p).

So how do you get Bernoulli RV’s?

Suppose U1 , . . . , Un are iid U(0,1). If Ui ≤ p, set Xi = 1; otherwise, set

Xi = 0. Repeat for i = 1, . . . , n. 2

Alexopoulos and Goldsman 5/21/10 22 / 73

 Convolution Method

Example: Erlangn (λ).

Pn
Suppose X1 , . . . , Xn are iid Exp(λ). Then Y = i=1 Xi ∼ Erlangn (λ).

Notice that by inverse transform,

n
X
Y = Xi
i=1
n  
X −1
= ℓn(Ui )
λ
i=1
n
Y 
−1
= ℓn Ui
λ
i=1

(This only takes one natural log evaluation, so it’s pretty efficient.) 2

Alexopoulos and Goldsman 5/21/10 23 / 73

 Convolution Method

Example: A crude “desert island” Nor(0,1) generator.

Pn
Suppose that U1 , . . . , Un are iid U(0,1), and let Y = i=1 Ui .

Note that E[Y ] = n/2 and Var(Y ) = n/12.

By the Central Limit Theorem, for large n.

Y ≈ Nor(n/2, n/12).

In particular, let’s choose n = 12, and assume that it’s “large.” Then
12
X
Y −6 = Ui − 6 ≈ Nor(0, 1).
i=1

It sort of works OK, but I wouldn’t use it in real life. 2

By the way, if Z ∼ Nor(0, 1) and you want X ∼ Nor(µ, σ 2 ), just take

X ← µ + σZ.
Alexopoulos and Goldsman 5/21/10 24 / 73
 Convolution Method

Other convolution-related tidbits:

Did you know. . . ?

U1 + U2 ∼ Triangular(0, 1, 2).
Pn
If X1 , . . . , Xn are iid Geom(p), then i=1 Xi ∼ NegBin(n, p).
Pn 2
If Z1 , . . . , Zn are iid Nor(0,1), then i=1 Zi ∼ χ2 (n).

Alexopoulos and Goldsman 5/21/10 25 / 73

 Acceptance-Rejection Method

Outline

1 Introduction
2 Inverse Transform Method
3 Cutpoint Method
4 Convolution Method
5 Acceptance-Rejection Method
6 Composition Method
7 Special-Case Techniques
8 Multivariate Normal Distribution
9 Generating Stochastic Processes

Alexopoulos and Goldsman 5/21/10 26 / 73

 Acceptance-Rejection Method

Acceptance-Rejection Method

Baby Example: Generate a U(2/3,1) RV. (You would usually do this

via inverse transform, but what the heck!)

Here’s the A-R algorithm:

1. Generate U ∼ U(0, 1).

2. If U ≥ 2/3, ACCEPT X ← U . Otherwise, REJECT and go to 1. 2

Alexopoulos and Goldsman 5/21/10 27 / 73

 Acceptance-Rejection Method

Motivation: The majority of cdf’s cannot be inverted efficiently. A-R

samples from a distribution that is “almost” the one we want, and then
adjusts by “accepting” only a certain proportion of those samples.

Suppose we want to simulate a continuous RV X with pdf f (x), but

that it’s difficult to generate directly. Also suppose that we can easily
generate a RV having pdf h(x) ≡ t(x)/c, where t(x) majorizes f (x),
i.e.,
t(x) ≥ f (x), x ∈ IR,
and Z ∞ Z ∞
c≡ t(x) dx ≥ f (x) dx = 1,
−∞ −∞

where we assume that c < ∞.

Alexopoulos and Goldsman 5/21/10 28 / 73

 Acceptance-Rejection Method

Notice that f (x) can be written as

f (x) t(x)
f (x) = c × × = cg(x)h(x),
t(x) c
where
Z ∞
h(x) dx = 1 (h is a density)
−∞

and

0 ≤ g(x) ≤ 1.

Alexopoulos and Goldsman 5/21/10 29 / 73

 Acceptance-Rejection Method

Theorem (von Neumann 1951): Let U ∼ U(0, 1), and let Y be a RV

(independent of U ) with pdf h(y). If U ≤ g(Y ), then Y has (conditional)
pdf f (y).

This suggests the following “acceptance-rejection” algorithm . . .

Algorithm A-R
Repeat
Generate U from U(0, 1)
Generate Y from h(y) (independent of U )
until U ≤ g(Y )
Return X ← Y

Alexopoulos and Goldsman 5/21/10 30 / 73

 Acceptance-Rejection Method

Proof that X has pdf f (x).

Let A be the “Acceptance” event. The cdf of X is

P (A, Y ≤ x)
P (X ≤ x) = P (Y ≤ x|A) = . (1)
P (A)

Then

P (A|Y = y) = P (U ≤ g(Y )|Y = y)

= P (U ≤ g(y)|Y = y)
= P (U ≤ g(y)) (U and Y are independent)
= g(y) (U is uniform). (2)

Alexopoulos and Goldsman 5/21/10 31 / 73

 Acceptance-Rejection Method

By the law of total probability,

Z ∞
P (A, Y ≤ x) = P (A, Y ≤ x|Y = y)h(y) dy
−∞
Z x
1
= P (A|Y = y)t(y) dy
c −∞
Z x
1
= g(y)t(y) dy (by (2))
c −∞
1 x
Z
= f (y) dy. (3)
c −∞
Letting x → ∞, we have
∞
1 1
Z
P (A) = f (y) dy = . (4)
c −∞ c
Then (1), (3), and (4) imply
x
P (A, Y ≤ x)
Z
P (X ≤ x) = = f (y) dy,
P (A) −∞
so that the pdf of X is f (x). 2
Alexopoulos and Goldsman 5/21/10 32 / 73
 Acceptance-Rejection Method

Alexopoulos and Goldsman 5/21/10 33 / 73

 Acceptance-Rejection Method

There are two main issues:

The ability to quickly sample from h(y).
c must be small (t(x) must be “close” to f (x)) since
1
P (U ≤ g(Y )) =
c
and the number of trials until “success” [U ≤ g(Y )] is Geom(1/c),
so that the mean number of trials is c.

Alexopoulos and Goldsman 5/21/10 34 / 73

 Acceptance-Rejection Method

Example (Law 2007): Generate a RV with pdf f (x) = 60x3 (1 − x)2 ,

0 ≤ x ≤ 1. Can’t invert this analytically.

Note that the maximum occurs at x = 0.6, and f (0.6) = 2.0736.

Using the majorizing function

t(x) = 2.0736,
0≤x≤1
R1
(which isn’t actually very efficient), we get c = 0 f (x) dx = 2.0736, so

h(x) = 1, 0≤x≤1 (i.e., a U(0,1) p.d.f.)

and
g(x) = 60x3 (1 − x)2 /2.0736.
E.g., if we generate U = 0.13 and Y = 0.25, then it turns out that
3 (1−Y )2
U ≤ g(Y ) = 60Y2.0736 , so we take X ← 0.25. 2

Alexopoulos and Goldsman 5/21/10 35 / 73

 Acceptance-Rejection Method

Example (Ross): Generate a standard half-normal RV, with pdf

2 2
f (x) = √ e−x /2 , x ≥ 0.
2π

Using the majorizing function

r
2e −x
t(x) = e ≥ f (x) for all x ≥ 0,
π
we get
∞
r r
2e 2e
Z
−x
c = e dx = = 1.3155,
π 0 π

h(x) = e−x (the Exp(1) pdf),

and
2 /2
g(x) = e−(x−1) . 2
Alexopoulos and Goldsman 5/21/10 36 / 73
 Acceptance-Rejection Method

We can use the half-normal result to generate a Nor(0, 1) variate.

Generate U from U(0, 1)

Generate X from the half-normal distribution
Return (
−X if U ≤ 1/2
Z =
X if U > 1/2.

We can then generate Nor(µ, σ 2 ) RV by using the obvious

transformation µ + σZ.

Alexopoulos and Goldsman 5/21/10 37 / 73

 Acceptance-Rejection Method

Example: The gamma distribution with density

xβ−1 −(x/α)β
f (x) = e , x > 0.
αβ Γ(β)

We’ll split the task of generating gamma RV’s via the A-R algorithm
into two cases depending on the magnitude of the shape parameter:
β < 1 and β ≥ 1. . .

Alexopoulos and Goldsman 5/21/10 38 / 73

 Acceptance-Rejection Method

If β < 1, we’ll use the following A-R algorithm with c ≤ 1.39:

Algorithm GAM1
b ← (e + β)/e (e is the base of the natural logarithm)
While (True)
Generate U from U(0, 1); W ← bU
If W < 1
Y ← W 1/β ; Generate V from U(0, 1)
If V ≤ e−Y : Return X = αY
Else
Y ← −ℓn[(b − W )/β]
Generate V from U(0, 1)
If V ≤ Y β−1 : Return X = αY
Alexopoulos and Goldsman 5/21/10 39 / 73
 Acceptance-Rejection Method

If β ≥ 1, the value
p of c for the following A-R algorithm decreases from
4/e = 1.47 to 4/π = 1.13 as β increases from 1 to ∞.

Algorithm GAM2
a ← (2β − 1)−1/2 ; b ← β − ℓn(4); c ← β + a−1 ; d ← 1 + ℓn(4.5)
While (True)
Generate U1 , U2 from U(0, 1)
V ← aℓn[U1 /(1 − U1 )]
Y ← βeV ; Z ← U12 U2
W ← b + cV − Y
If W + d − 4.5Z ≥ 0: Return X = αY
Else
If W ≥ ℓn(Z): Return X = αY

Alexopoulos and Goldsman 5/21/10 40 / 73

 Acceptance-Rejection Method

Example: The Poisson distribution with probability function

λn
P (X = n) = e−λ , n = 0, 1, . . .
n!
Define Ai as the ith interarrival time from a Pois(λ) process. Then
X = n ⇔ See exactly n PP(λ) arrivals by t = 1
Xn n+1
X
⇔ Ai ≤ 1 < Ai
i=1 i=1
n   n+1
X  −1 
X −1
⇔ ℓn(Ui ) ≤ 1 < ℓn(Ui )
λ λ
i=1 i=1
n n+1
! !
−1 Y −1 Y
⇔ ℓn Ui ≤ 1 < ℓn Ui
λ λ
i=1 i=1
n
Y n+1
Y
⇔ Ui ≥ e−λ > Ui . (5)
i=1 i=1

Alexopoulos and Goldsman 5/21/10 41 / 73

 Acceptance-Rejection Method

The following A-R algorithm samples U(0,1)’s

Qn+1 until (5) becomes true,
−λ
i.e., until the first time n such that e > i=1 Ui .

Algorithm POIS1
a ← e−λ ; p ← 1; X ← −1
Until p < a
Generate U from U(0, 1)
p ← pU ; X ← X + 1
Return X

Alexopoulos and Goldsman 5/21/10 42 / 73

 Acceptance-Rejection Method

Example (Banks et al.): Apply Algorithm POIS1 to obtain a Pois(2)

variate.
Qn+1
Sample until e−λ = 0.1353 > i=1 Ui .
Qn+1
n Un+1 i=1 Ui Stop?
0 0.3911 0.3911 No
1 0.9451 0.3696 No
2 0.5033 0.1860 No
3 0.7003 0.1303 Yes

Thus, we take X = 3. 2

Remark: How many U ’s are required to generate one realization of

X? An easy argument says that the expected number you’ll need is
E[X + 1] = λ + 1.

Alexopoulos and Goldsman 5/21/10 43 / 73

 Acceptance-Rejection Method

If λ ≥ 20, we can use the normal approximation

X −λ
√ ≈ Nor(0, 1).
λ

Algorithm POIS2 (for λ ≥ 20)

√
α← λ
Generate Z from Nor(0, 1)
Return X = max(0, ⌊λ + αZ + 0.5⌋) (Note that this employs a
“continuity correction.”)

Alexopoulos and Goldsman 5/21/10 44 / 73

 Composition Method

Outline

1 Introduction
2 Inverse Transform Method
3 Cutpoint Method
4 Convolution Method
5 Acceptance-Rejection Method
6 Composition Method
7 Special-Case Techniques
8 Multivariate Normal Distribution
9 Generating Stochastic Processes

Alexopoulos and Goldsman 5/21/10 45 / 73

 Composition Method

Composition Method

Idea: Suppose a RV actually comes from two RV’s (sort of on top of

each other). E.g., your plane can leave the airport gate late for two
reasons — air traffic delays and maintenance delays, which compose
the overall delay time.

The goal is to generate a RV with cdf

∞
X
F (x) = pj Fj (x),
j=1
P
where pj > 0 for all j, j pj = 1, and the Fj (x)’s are “easy” cdf’s to
generate from.

Generate a positive integer J such that P (J = j) = pj for all j.

Return X from cdf FJ (x).

Alexopoulos and Goldsman 5/21/10 46 / 73

 Composition Method

Proof that X has cdf F (x).

By the law of total probability,

∞
X
P (X ≤ x) = P (X ≤ x|J = j)P (J = j)
j=1
X∞
= Fj (x)pj
j=1
= F (x). 2

Alexopoulos and Goldsman 5/21/10 47 / 73

 Composition Method

Example: Laplace distribution.

(
1 x
2e if x < 0
f (x) ≡ 1 −x
2e if x > 0

After a little algebra,

(
1 x
F (x) ≡ 2e if x < 0
1 − 12 e−x if x > 0

Alexopoulos and Goldsman 5/21/10 48 / 73

 Composition Method

Meanwhile, let’s decompose X into “negative exponential” and regular

exponential distributions:
( (
ex if x < 0 0 if x < 0
F1 (x) ≡ and F2 (x) ≡
1 if x > 0 1 − e−x if x > 0

Then
1 1
F (x) = F1 (x) + F2 (x).
2 2

So by inverse transform using F1 (x) and F2 (x), let

(
ℓn(U ) w.p. 1/2
X ← 2
−ℓn(U ) w.p. 1/2

Alexopoulos and Goldsman 5/21/10 49 / 73

 Special-Case Techniques

Outline

1 Introduction
2 Inverse Transform Method
3 Cutpoint Method
4 Convolution Method
5 Acceptance-Rejection Method
6 Composition Method
7 Special-Case Techniques
8 Multivariate Normal Distribution
9 Generating Stochastic Processes

Alexopoulos and Goldsman 5/21/10 50 / 73

 Special-Case Techniques

Special-Case Techniques

Box–Muller Method: Here’s a nice, easy way to generate standard

normals.

Theorem: If U1 , U2 are iid U(0,1), then

p
Z1 = −2ℓn(U1 ) cos(2πU2 )
p
Z2 = −2ℓn(U1 ) sin(2πU2 )

are iid Nor(0,1).

Note that the trig calculations must be done in radians.

Proof Someday soon. 2

Alexopoulos and Goldsman 5/21/10 51 / 73

 Special-Case Techniques

Some interesting corollaries follow directly from Box–Muller.

Example: Note that

Z12 + Z22 ∼ χ2 (1) + χ2 (1) ∼ χ2 (2).

But

Z12 + Z22 = −2ℓn(U1 )(cos2 (2πU2 ) + sin2 (2πU2 ))

= −2ℓn(U1 )
∼ Exp(1/2).

Thus, we’ve just proven that

χ2 (2) ∼ Exp(1/2). 2

Alexopoulos and Goldsman 5/21/10 52 / 73

 Special-Case Techniques

Example: Note that

Z2 /Z1 ∼ Nor(0, 1)/Nor(0, 1) ∼ Cauchy ∼ t(1).

But
p
−2ℓn(U1 ) sin(2πU2 )
Z2 /Z1 = p = tan(2πU2 ).
−2ℓn(U1 ) cos(2πU2 )

Thus, we’ve just proven that

tan(2πU ) ∼ Cauchy (and, similarly, cot(2πU ) ∼ Cauchy).

Similarly,
Z22 /Z12 = tan2 (2πU ) ∼ t2 (1) ∼ F (1, 1).
(Did you know that?)

Alexopoulos and Goldsman 5/21/10 53 / 73

 Special-Case Techniques

Polar Method — a little faster than Box–Muller.

1. Generate U1 , U2 iid U(0,1).

Let Vi = 2Ui − 1, i = 1, 2, and W = V12 + V22 .

2. If W > 1, reject and go back to Step 1.

p
O’wise, let Y = −2(ℓn(W ))/W , and accept Zi ← Vi Y , i = 1, 2.

Then Z1 , Z2 are iid Nor(0,1).

Alexopoulos and Goldsman 5/21/10 54 / 73

 Special-Case Techniques

Order Statistics

Suppose that X1 , X2 , . . . , Xn are iid from some distribution, and let

Y = min{X1 , . . . , Xn }. (Y is called the first order stat.) Can we
generate Y using just one U(0,1)?

Example: Suppose X1 , . . . , Xn ∼ Exp(λ). Then

P (Y ≤ y) = 1 − P (Y > y)
= 1 − P (min Xi > y)
i
= 1 − P (all Xi ’s > y)
= 1 − (e−y )n .

This implies that Y = mini {Xi } ∼ Exp(nλ). So we can generate

1
Y = − ℓn(U ). 2
nλ
We can do the same kind of thing for Z = maxi Xi .
Alexopoulos and Goldsman 5/21/10 55 / 73
 Special-Case Techniques

Other Quickies

χ 2
P(n)n
distribution: If Z1 , Z2 , . . . , Zn are iid Nor(0,1), then
2 2
i=1 Zi ∼ χ (n).

t(n) distribution: If Z ∼ Nor(0, 1) and Y ∼ χ2 (n), and X and Y are

independent, then
Z
p ∼ t(n).
Y /n

Note that t(1) is the Cauchy distribution.

F (n, m) distribution: If X ∼ χ2 (n) and Y ∼ χ2 (m) and X and Y are

independent, then (X/n)/(Y /m) ∼ F (n, m).

Generating RV’s from continuous empirical distributions — no time

here. Can use the CONT function in Arena.

Alexopoulos and Goldsman 5/21/10 56 / 73

 Multivariate Normal Distribution

Outline

1 Introduction
2 Inverse Transform Method
3 Cutpoint Method
4 Convolution Method
5 Acceptance-Rejection Method
6 Composition Method
7 Special-Case Techniques
8 Multivariate Normal Distribution
9 Generating Stochastic Processes

Alexopoulos and Goldsman 5/21/10 57 / 73

 Multivariate Normal Distribution

Multivariate Normal Distribution

The random vector X = (X1 , . . . , Xk )T has the multivariate normal

distribution with mean vector µ = (µ1 , . . . , µk )T and k × k covariance
matrix Σ = (σij ) if it has pdf

(x − µ)T Σ−1 (x − µ)
 
1
f (x) = exp − , x ∈ IRk .
(2π)k/2 |Σ|1/2 2

It turns out that

E(Xi ) = µi , Var(Xi ) = σii , Cov(Xi , Xj ) = σij .

Notation: X ∼ Nork (µ, Σ).

Alexopoulos and Goldsman 5/21/10 58 / 73

 Multivariate Normal Distribution

In order to generate X, let’s start with a vector Z = (Z1 , . . . , Zk ) of iid

Nor(0,1) RV’s. That is, suppose Z ∼ Nork (0, I), where I is the k × k
identity matrix, and 0 is simply a vector of 0’s.

Suppose we can find a (lower triangular) matrix C such that Σ = CC T .

Then it can be shown that X = µ + CZ is multivariate normal with

mean E[X] and covariance matrix

Σ ≡ E[(CZ)(CZ)T ] = E[CZZ T C T ] = C(E[ZZ T ])C T = CC T .

That is, X ∼ Nork (µ, Σ).

Alexopoulos and Goldsman 5/21/10 59 / 73

 Multivariate Normal Distribution

The following algorithm computes the (Cholesky) matrix C:

Algorithm LTM

For i = 1, . . . , k,

For j = 1, . . . , i − 1,
 
cij ← σij − j−1
P
ℓ=1 ciℓ cjℓ /cjj

cji = 0
 1/2
cii = σii − i−1 2
P
ℓ=1 ciℓ

Alexopoulos and Goldsman 5/21/10 60 / 73

 Multivariate Normal Distribution

Once C has been computed, X is generated as follows:

Algorithm MN

i←1

Until i > k:

Generate Xi from Nor(0, 1)

Xi ← µi

j←1

Until j > i: Xi ← Xi + cij Xj ; j ← j + 1

i ←i+1

Alexopoulos and Goldsman 5/21/10 61 / 73

 Generating Stochastic Processes

Outline

1 Introduction
2 Inverse Transform Method
3 Cutpoint Method
4 Convolution Method
5 Acceptance-Rejection Method
6 Composition Method
7 Special-Case Techniques
8 Multivariate Normal Distribution
9 Generating Stochastic Processes

Alexopoulos and Goldsman 5/21/10 62 / 73

 Generating Stochastic Processes

Generating Stochastic Processes

Generating Poisson Arrivals

When the arrival rate is constant, say λ, the interarrival times of a

Poisson(λ) process are iid Exp(λ), and we can generate the arrival
times iteratively:

T0 ← 0
1
Ti ← Ti−1 − ℓn(Ui ), i ≥ 1.
λ

Alexopoulos and Goldsman 5/21/10 63 / 73

 Generating Stochastic Processes

Suppose that we want to generate a fixed number n of PP(λ) arrivals in

a fixed time interval [a, b]. To do so, we note a theorem stating that the
joint distribution of the n arrivals is the same as the joint distribution of
the order statistics of n iid Unif(a, b) RV’s.

Generate iid U1 , . . . , Un from U(0, 1)

Sort the Ui ’s: U(1) < U(2) < · · · < U(n)

Set the arrival times to Ti ← a + (b − a)U(i)

Alexopoulos and Goldsman 5/21/10 64 / 73

 Generating Stochastic Processes

Nonhomogeneous Poisson Process — nonstationary arrivals

Let
λ(t) = rate (intensity) function at time t,
N (t) = number of arrivals during [0, t].

Then Z s+t 
N (s + t) − N (s) ∼ Poisson λ(u) du .
s

Alexopoulos and Goldsman 5/21/10 65 / 73

 Generating Stochastic Processes

Incorrect NHPP Algorithm [it can “skip” intervals with large λ(t)]

T0 ← 0; i ← 0

Repeat

Generate U from U(0, 1)

1
Ti+1 ← Ti − λ(Ti ) ℓn(U )

i ←i+1

Don’t use this algorithm.

Alexopoulos and Goldsman 5/21/10 66 / 73

 Generating Stochastic Processes

Alexopoulos and Goldsman 5/21/10 67 / 73

 Generating Stochastic Processes

The following (good) algorithm assumes that λ∗ ≡ sup λ(t) < +∞,
generates potential arrivals with rate λ∗ , and accepts a potential arrival
at time t with probability λ(t)/λ∗ .

Thinning Algorithm
T0 ← 0; i ← 0
Repeat
t ← Ti
Repeat
Generate U , V from U(0, 1)
1
t←t− λ∗ ℓn(U )
until V ≤ λ(t)/λ∗
i ←i+1
Ti ← t
Alexopoulos and Goldsman 5/21/10 68 / 73
 Generating Stochastic Processes

First-Order Moving Average Process

An MA(1) process is defined by

Yi = εi + θεi−1 , for i = 1, 2, . . .,

where θ is a constant and the εi ’s are iid Nor(0, 1) RV’s that are
independent of Y0 .

The MA(1) is a popular tool for modeling and detecting trends.

The MA(1) has covariance function Var(Yi ) = 1 + θ 2 , Cov(Yi , Yi+1 ) = θ,

and Cov(Yi , Yi+k ) = 0 for k ≥ 2.

So the covariances die off pretty quickly.

How to generate? Start with ε0 ∼ Nor(0, 1). Then generate

ε1 ∼ Nor(0, 1) to get Y1 , ε2 ∼ Nor(0, 1) to get Y2 , etc.

Alexopoulos and Goldsman 5/21/10 69 / 73

 Generating Stochastic Processes

First-Order Autoregressive Process

An AR(1) process is defined by

Yi = φYi−1 + εi , for i = 1, 2, . . .,

where −1 < φ < 1; Y0 ∼ Nor(0, 1); and the εi ’s are iid Nor(0, 1 − φ2 )
RV’s that are independent of Y0 .

This is used to model lots of real-world stuff.

The AR(1) has covariance function Cov(Yi , Yi+k ) = φ|k| for all
k = 0, ±1, ±2, . . ..

If φ is close to one, you get highly positively correlated Yi ’s. If φ is close

to zero, the Yi ’s are nearly independent.

to generate? Start with Y0 ∼ Nor(0, 1) and

How p
ε1 ∼ 1 − φ2 Nor(0, 1). This gives Y1 . Then generate ε2 to get Y2 , etc.
Alexopoulos and Goldsman 5/21/10 70 / 73
 Generating Stochastic Processes

M/M/1 Queue

Consider a single-server queue with customers arriving according to a

Poisson(λ) process, standing in line with a FIFO discipline, and then
getting served in an Exp(µ) amount of time.

Let Ii+1 denote the interarrival time between the ith and (i + 1)st
customers; let Si be the ith customer’s service time; and let Wi denote
the ith customer’s waiting time before service.

Lindley gives a very nice way to generate a series of waiting times for
this simple example:

Wi+1 = max{Wi + Si − Ii+1 , 0}.

(You can model the time in the system with a similar equation.)

Alexopoulos and Goldsman 5/21/10 71 / 73

 Generating Stochastic Processes

Brownian Motion

The stochastic process {W(t), t ≥ 0} is a standard Brownian motion

process if:
1 W(0) = 0.
2 W(t) ∼ Nor(0, t).
3 {W(t), t ≥ 0} has stationary and independent increments.

Increments: Anything like W(b) − W(a).

Stationary increments: The distribution of W(t + h) − W(t) only

depends on h.

Independent increments: If a < b < c < d, then W(d) − W(c) is indep

of W(b) − W(a).

Discovered by Brown; analyzed rigorously by Einstein; mathematical

rigor established by Wiener (also called Wiener process).
Alexopoulos and Goldsman 5/21/10 72 / 73
 Generating Stochastic Processes

Here’s a way to construct BM:

Suppose Y1 , Y2 , . . . is any sequence of identically distributed RV’s with

mean zero and variance 1. (To some extent, the Yi ’s don’t even have to
be indep!) Donsker’s Central Limit Theorem says that
⌊nt⌋
1 X d
√ Yi −→ W(t) as n → ∞,
n
i=1

d
where −→ denotes convergence in distribution as n gets big, and ⌊·⌋
is the floor function, e.g., ⌊3.7⌋ = 3.

One choice that works well is to take Yi = ±1, each with probability
1/2. Take n at least 100, t = 1/n, 2/n, . . . , n/n, and calculate
W(1/n), W (2/n), . . . , W(n/n).

It really works!
Alexopoulos and Goldsman 5/21/10 73 / 73