Slides for Introduction to Stochastic Search

and Optimization (ISSO) by J. C. Spall

APPENDIX D
RANDOM NUMBER GENERATION
• Organization of chapter in ISSO*
– General description and linear congruential generators
• Criteria for “good” random number generator
– Random variates with general distribution
• Different types of random number generators

*Note: These slides cover some topics not included in ISSO

 Uniform Random Number Generators
• Want sequence of independent, identically distributed
uniform (U(0, 1)) random variables
– U(0, 1) random numbers of direct interest in some
applications
– More commonly, U(0, 1) numbers transformed to random
numbers having other distributions (e.g., in Monte Carlo
simulation)
• Computer-based random number generators (RNGs)
produce deterministic and periodic sequence of numbers
– Pseudo random numbers
• Want pseudo random numbers that “look” random
– Should be able to pass all relevant statistical tests for
randomness

D-2
 Overall Framework for Generating
Random Numbers
• State at step k given transition function fk:
Jk  fk (Jk 1, Jk 2 ,..., Jk r ), r  1
• Output function, g, produces pseudo random
numbers as
Uk  g  J k 

• Output sequence of RNG is {Jk , k  1}

• Period of an RNG is number of iterations before
RNG output (Uk) repeats itself

D-3
Criteria for Good Random Number Generators
• Long period
• Strong theoretical foundation
• Able to pass empirical statistical tests for independence and
distribution (next slide)
• Speed/efficiency
• Portability: can be implemented easily using different
languages and computers
• Repeatability: should be able to generate same sequence
from same seed
• Be cryptographically strong to external observer: unable to
predict next value from past values
• Good distribution of points throughout domain (low
discrepancy) (also related to quasi-random sequences, not
covered here)
D-4
 Criteria for Good Random Number
Generators (cont’d): Statistical Tests
• Ideal aim is that no statistical test can distinguish RNG
output from i.i.d. U(0, 1) sequence
– Not possible in practice due to limits of testing and limits of
finite-period generators
• More realistic goal is passing only key (relevant) tests
• Null hypothesis: sequence of random numbers is
realization of i.i.d. U(0, 1) stochastic process
– Almost limitless number of possible tests of this hypothesis
• Failing to reject null hypothesis improves confidence in
generator but does not guarantee random numbers will be
appropriate for all applications
• Bad RNGs fail simple tests; good RNGs fail only
complicated and/or obscure tests
D-5
 Types of Random Number Generators

• Linear: commonly used

• Combined: can increase period and improve
statistical properties
• Nonlinear: structure is less regular than linear
generators but more difficult to implement and
analyze
• Physical processes (e.g., timing in atomic decay,
internal system noise, etc.)
– Not as widely used as computer-based generators due
to costliness of implementation, lack of speed, and
inability to reproduce same sequence

D-6
 Linear Congruential Generators
• Linear congruential generators (LCG) produce U(0, 1)
numbers via
Jk   aJk 1  c  mod m
Jk
Uk  ,
m
where a, c, and m are user-specified constants
• LCG appears to be most widely used and studied random
number generator
• Values a, c, and m should be carefully chosen:
0  a  m, 0  c  m
0  J0  m, Jk  0,1, , m  1
(LCG output may be modified to avoid 0 values for Uk)
D-7
 Linear Congruential Generators

• Some famous values for a and m (assuming c = 0)

– a = 23, m = 108 + 1 (first LCG; original 1951
implementation)
– a = 65539, m = 231 – 1 (RANDU generator of 1960s; poor
because of correlated output)
– a = 16807, m = 231 – 1 (has been discussed as minimum
standard for RNGs; used in Matlab version 4)

Lehmer, D. H. (1951), “Mathematical Methods in Large-Scale

Computing Units,” Annals of the Computation Laboratory of
Harvard University, no. 26, pp. 141–146.

D-8
Example of “Minimal” Statistical Test for LCG:
Is Sample Mean Close to 0.5?
0.58

0.56

0.54
Sample Mean

0.52 m = 231 – 1, a = 4, c = 1

0.5
m = 482, a = 13, c = 14
0.48
m = 27, a = 26, c = 5
0.46

0.44
m = 9, a = 4, c = 1
0.42
0 500 1000 1500 2000
Number of Samples
D-9
 Fibonacci Generators

• These are generators where current value is sum (or

difference, or XOR) of two preceding elements
• Lagged Fibonacci generators use two numbers
earlier in sequence

Jk  (Jk  p  Jk r ) mod m
Jk
Uk 
m
p, q are the lags

D-10
 Multiple Recursive Generators

• Multiple recursive generators (MRGs) are defined by

Jk   a1Jk 1   ak Jk r  mod m
Jk
Uk  ,
m
where the ai belong to {0,1,…,m – 1}
• Maximal period is m r – 1 for prime m and properly
chosen ai
• For r = 1, MRG reduces to LCG

D-11
 Nonlinear Generators
• Nonlinearity sometimes used to enhance performance of
RNGs
– Nonlinearity may appear in transition function fn and/or in
output function g (see earlier slide “Overall Framework for
Generating Random Numbers”)
– Have some advantage in reducing lattice structure
(Exercise D.2) and in reducing discrepancy
• Two examples (L’Ecuyer, 1998)
– Nonlinear f = fk via quadratic recursion:
Jk  (aJk21  bJk 1  c ) mod m
Uk  J k M
– Nonlinear fk via inversive generator :
m 2
Jk   ak  c  mod m
Uk  J k M
D-12
 Combining Generators

• Used to increase period length and improve

statistical properties
• Shuffling: uses second generator to choose
random order for numbers produced by final
generator
• Bit mixing: combines numbers in two sequences
using some logical or arithmetic operation (addition
and subtraction are preferred)

D-13
 Random Number Generators Used in
Common Software Packages
• Important to understand types of generators used in
statistical software packages and their limitations
• MATLAB:
– Versions earlier than 5: LCG with a = 75 = 16807, c = 0, m
= 231 – 1
– Versions 5 to 7.3: lagged Fibonacci generator combined
with shift register random integer generator with period
21492 (“ziggurat algorithm”)
– Versions 7.4 and later: “Mersenne twister” (sophisticated
linear algorithm with huge period 219937 )
• EXCEL: Uk = fractional part (9821Uk –1 + 0.211327);
period 223
• SAS (vers. 6): LCG with period 231 – 1
D-14
 Inverse-Transform Method for Generating
Non-U(0,1) Random Numbers

• Let F(x) be distribution function of X

• Define inverse function of F by

F 1( y )  inf x : F ( x )  y ,0  y  1.

• Generate X by
X  F 1(U )
• Example: exponential distribution
F ( x )  1  e x
1 1
X  F (U )   log(1  U )

D-15
 AcceptReject Method
• Let pX(x) be density function of X
• Find function (x) that majorizes pX(x)
– Have (x) = Cq(x), C  1, q is density function that is “easy”
to generate outcomes from
• Acceptreject method generates X by following steps:
Generate U from U(0,1) (*)
Generate Y from q(y), independent of U
p (Y )
If U  X , then set X = Y. Otherwise, go back to (*)
(Y )
• Probability of acceptance (efficiency) = 1/C
• Related to Markov chain Monte Carlo (MCMC) (see
Exercise 16.4 of ISSO)
• Example to follow next two slides (pX(x) = beta density)….
D-16
 Y ~ q( y )  U (0,1)
60 x (1  x )
3 2
if 0  x  1
pX ( x )   60Y 3 (1  Y )2
0 otherwise U
2.0736
2.5
( x )  Cq( x )  2.0736  U (0,1)
2.0

1.5
pX(x)

1.0
q(x) = U(0,1)
Note: This
example
0.5 adapted from
Law (2007,
p.438)
0 0.2 0.4 0.6 0.8 1.0 1.2

D-17
 U ~ U(0,1): 0.9501, 0.2311, 0.6068, 0.4860, 0.8913, 

Y ~ q(y)  U(0,1): 0.7621, 0.4565, 0.0185, 0.8214, 0.4447, 

p X (Y )
: 0.7249, 0.8131, 0.00018,
Cq(Y )

Accept/reject: Is U value  above ratio?

X ~ pX(x): 0.7621, 0.4565, 0.0185,

reject accept reject

Accepted values represent realization of random

numbers from pX(x)

D-18
 References for Further Study
• Law, A. M. (2007), Simulation Modeling and Analysis (4th ed.),
McGraw-Hill, New York, Chap. 8.
• L’Ecuyer, P. (1998), “Random Number Generation,” in
Handbook of Simulation: Principles, Methodology, Advances,
Applications, and Practice (J. Banks, ed.), Wiley, New York,
Chap. 4.
• L'Ecuyer, P. (2004), “Random Number Generation,” in
Handbook of Computational Statistics (J. E. Gentle, W. Härdle,
and Y. Mori, eds.), Springer, Chap. II.2 (pp. 3570).
• Moler, C. (2004), Numerical Computing with MATLAB (Chap.
9: Random Numbers), SIAM, Philadelphia (online at
www.mathworks.com/moler/chapters.html).
• Neiderreiter, H. (1992), Random Number Generation and
Quasi-Monte Carlo Methods, SIAM, Philadelphia.
D-19