Chapter 5

Generating Random numbers

Topics
Exercises

Copyright ©2010 by K. Plantenberg

Restricted use only
 Generating Random numbers
Summary

Summary Properties of Random Numbers

5.1) Properties of Random Numbers

Generation of Pseudo-Random Numbers (PRN)

5.2) Generation of Pseudo-Random Numbers (PRN)

Techniques for Generating Random Numbers

5.3) Techniques for Generating Random Numbers

Tests for Random Numbers

5.4) Tests for Random Numbers

Caution

5.5) Caution
Caution

Caution

Copyright ©2010 by K. Plantenberg

Restricted use only
 Generating Random numbers

5.1) Properties of Random

Numbers

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.1) Properties of Random Numbers
A sequence of random numbers R1, R2, …, must have two important
statistical properties:
→ Uniformity
→ Independence.
Random Number, Ri, must be independently drawn from a uniform
distribution with pdf:
pdf for random
1, 0  x  1 numbers
f ( x) = 
0, otherwise
1
1 2


x 1
E ( R) = xdx = =
0 2 2
0
3 1 2
1 1 1 1
V ( R) =  x dx − E (R )
1 x
= −  = − =
2 2

Copyright ©2010 by K. Plantenberg

0 3 0  2  3 4 12 4
Restricted use only
 5.1) Properties of Random Numbers
Uniformity: If the interval [0,1] is divided into n
classes, or subintervals of equal length, the expected
number of observations in each interval is N/n, where N
is the total number of observations
Independence: The probability of observing a value in
a particular interval is independent of the previous
value drawn

Copyright ©2010 by K. Plantenberg

Restricted use only
 Generating Random numbers

5.2) Generation of Pseudo-

Random Numbers (PRN)

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.2) Generation of Pseudo-Random Numbers
(PRN)
“Pseudo”, because generating numbers using a known
method removes the potential for true randomness.
• If the method is known, the set of random numbers can be
replicated!!
Goal: To produce a sequence of numbers in [0,1] that
simulates, or imitates, the ideal properties of random
numbers (RN) - uniform distribution and independence.

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.2) Generation of Pseudo-Random Numbers
(PRN)
Problems that occur in generation of pseudo-random
numbers (PRN)
• Generated numbers might not be uniformly distributed
• Generated numbers might be discrete-valued instead of
continuous-valued
• Mean of the generated numbers might be too low or too
high
• Variance of the generated numbers might be too low or too
high
• There might be dependence (i.e., correlation)

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.2) Generation of Pseudo-Random Numbers
(PRN)
Important considerations in RN routines:
• The routine should be fast. Individual computations are
inexpensive, but a simulation may require many millions of
random numbers
• Portable to different computers – ideally to different
programming languages. This ensures the program produces
same results
• Have sufficiently long cycle. The cycle length, or period
represents the length of random number sequence before
previous numbers begin to repeat in an earlier order.
• Replicable. Given the starting point, it should be possible to
generate the same set of random numbers, completely
independent of the system that is being simulated
• Closely approximate the ideal statistical properties of
uniformity and independence.

Copyright ©2010 by K. Plantenberg

Restricted use only
 Generating Random numbers

5.3) Techniques for Generating

Random Numbers

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3) Techniques for Generating Random
Numbers
3.1 MidSquare
3.2 Linear Congruential Method (LCM).
→ Most widely used technique for generating random numbers
3.3 Combined Linear Congruential Generators (CLCG).
→ Extension to yield longer period (or cycle)
3.4 Random-Number Streams.

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3.1) MidSquare

MidSquare
Example:
X0 = 7182 (seed)
= 51581124
==> R1 = 0.5811
= (5811) 2 = 33767721
==> R2 = 0.7677
etc.

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3.1) MidSquare

Note: Cannot choose a seed that guarantees that

the sequence will not degenerate and will have a
long period. Also, zeros, once they appear, are
carried in subsequent numbers.
Ex1: X0 = 5197 (seed) = 27008809
==> R1 = 0.0088 = 00007744
==> R2 = 0.0077
Ex2: X0 = 4500 (seed) = 20250000
==> R1 = 0.2500 = 06250000
==> R2 = 0.2500
Copyright ©2010 by K. Plantenberg
Restricted use only
 5.3.1) MidSquare

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3.2) Linear Congruential Method (LCM)
To produce a sequence of integers, X1, X2, … between 0 and m-1
by following a recursive relationship:

X i +1 = (aX i + c) mod m, i = 0,1,2,...

The The The

multiplier increment modulus

X0 is called the seed

The selection of the values for a, c, m, and X0 drastically affects
the statistical properties and the cycle length.
If c 0 then it is called mixed congruential method
When c=0 it is called multiplicative congruential method

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3.2) Linear Congruential Method (LCM)
The random integers are being generated in the
range [0,m-1], and to convert the integers to
random numbers:

Xi
Ri = , i = 1,2,...
m

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3.2) Linear Congruential Method (LCM)

EXAMPLE: Use X0 = 27, a = 17, c = 43, and m = 100.

The Xi and Ri values are:
X1 = (17*27+43) mod 100 = 502 mod 100 = 2, R1 = 0.02;
X2 = (17*2+43) mod 100 = 77 mod 100 =77, R2 = 0.77;
X3 = (17*77+43) mod 100 = 1352 mod 100 = 52 R3 = 0.52;
…

Notice that the numbers generated assume values only from the set
I = {0,1/m,2/m,….., (m-1)/m} because each Xi is an integer in
the set {0,1,2,….,m-1}
Thus each Ri is discrete on I, instead of continuous on interval [0,1]

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3.2) Linear Congruential Method (LCM)

• Parameters m = 63, a = 22, c = 4, X0 = 19:

Xi = (22 Xi–1 + 4) (mod 63), seed with X0 = 19
i 22 Xi–1+4 Xi Ri = Xi / m
0 19
1
2
422
972
44
27
0.6984
0.4286
• Cycling — will repeat forever
3 598 31 0.4921 • Cycle length  m
4 686 56 0.8889
: : : : (could be < m depending
61 158 32 0.5079
62 708 15 0.2381 on parameters)
63
64
334
422
19
44
0.3016
0.6984
• Result: Pick m BIG
65 972 27 0.4286
66 598 31 0.4921
: : : :

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3.2) Linear Congruential Method (LCM)

Maximum Density
→ Such that the values assumed by Ri, i = 1,2,…, leave no
large gaps on [0,1]
→ Problem: Instead of continuous, each Ri is discrete
→ Solution: a very large integer for modulus m (e.g., 231-1, 248)
Maximum Period
→ To achieve maximum density and avoid cycling.
→ Achieved by: proper choice of a, c, m, and X0.
Most digital computers use a binary representation of numbers
→ Speed and efficiency are aided by a modulus, m, to be (or
close to) a power of 2.

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3.2) Linear Congruential Method (LCM)
Maximum Period or Cycle Length:
For m a power of 2, say m=2b, and c0, the longest possible period is
P=m=2b, which is achieved when c is relatively prime to m
(greatest common divisor of c and m is 1) and a=1+4k, where k is
an integer
For m a power of 2, say m=2b, and c=0, the longest possible period is
P=m/4=2b-2, which is achieved if the seed X0 is odd and if the
multiplier a is given by a=3+8k or a=5+8k for some k=0,1,….
For m a prime number and c=0, the longest possible period is P=m-
1, which is achieved whenever the multiplier a has the property
that the smallest integer k such that ak-1 is divisible by m is
k=m-1

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3.2) Linear Congruential Method (LCM)
Example: Using the multiplicative congruential method, find the period of the
generator for a=13, m=26=64 and X0=1,2,3 and 4

i 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

Xi 1 13 41 21 17 29 57 37 33 45 9 53 49 61 25 5 1

Xi 2 26 18 42 34 58 50 10 2

Xi 3 39 59 63 51 23 43 47 35 7 27 31 19 55 11 15 3

Xi 4 52 36 20 4

m=64, c=0; Maximal period P=m/4 = 16 is achieved by using odd seeds X0=1 and
X0=3 (a=13 is of the form 5+8k with k=1)
With X0=1, the generated sequence {1,5,9,13,…,53,57,61} has large gaps
Not a viable generator !! Density insufficient, period too short
Copyright ©2010 by K. Plantenberg
Restricted use only
 5.3.3) Combined Linear Congruential
Generators :
With increased computing power, the complexity of
simulated systems is increasing, requiring longer
period generator.
• Examples: 1) highly reliable system simulation
requiring hundreds of thousands of elementary
events to observe a single failure event;
• 2) A computer network with large number of nodes,
producing many packets
Approach: Combine two or more multiplicative
congruential generators in such a way to produce a
generator with good statistical properties

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3.3) Combined Linear Congruential
Generators :
L’Ecuyer suggests how this can be done:
• If Wi,1, Wi,2,….,Wi,k are any independent, discrete
valued random variables (not necessarily identically
distributed)
• If one of them, say Wi,1 is uniformly distributed on
the integers from 0 to m1-2, then
 k 
Wi =   Wi , j  mod m1 − 1
 j =1 
is uniformly distributed on the integers from 0 to m1-2

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3.3) Combined Linear Congruential
Generators :
Let Xi,1, Xi,2, …, Xi,k, be the ith output from k different
multiplicative congruential generators.
• The jth generator:
–Has prime modulus mj and multiplier aj and
period is mj-1
–Produced integers Xi,j is approx ~ Uniform
on integers in [1, mj-1]
–Wi,j = Xi,j -1 is approx ~ Uniform on integers
in [0, mj-2]

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.3.3) Combined Linear Congruential
Generators :
Suggested form:

 Xi
 m , Xi  0
 k 
X i =   (−1) X i , j  mod m1 − 1
 j −1
Hence, Ri =  1
 j =1  m −1
 1 , Xi = 0
 m1

•The maximum possible period for such a

generator is:
(m1 − 1)(m2 − 1)...(mk − 1)
P=
2k −1
Copyright ©2010 by K. Plantenberg
Restricted use only
 5.3.3) Combined Linear Congruential
Generators :
Example: For 32-bit computers, L’Ecuyer [1988] suggests combining k = 2
generators with m1 = 2,147,483,563, a1 = 40,014, m2 = 2,147,483,399 and
a2 = 20,692. The algorithm becomes:
Step 1: Select seeds
– X1,0 in the range [1, 2,147,483,562] for the 1st generator
– X2,0 in the range [1, 2,147,483,398] for the 2nd generator.

Step 2: For each individual generator,

X1,j+1 = 40,014 X1,j mod 2,147,483,563
X2,j+1 = 40,692 X1,j mod 2,147,483,399.
Step 3: Xj+1 = (X1,j+1 - X2,j+1 ) mod 2,147,483,562.
Step 4: Return  X j +1
 , X j +1  0
R j +1 =  2,147,483, 563
 2,147,483,562 , X j +1 = 0
 2,147,483,563

Step 5: Set j = j+1, go back to step 2.

• Combined generator has period: (m1 – 1)(m2 – 1)/2 ~ 2 x 1018
Copyright ©2010 by K. Plantenberg
Restricted use only
 5.3.4) Random-Numbers Streams

The seed for a linear congruential random-number generator:

• Is the integer value X0 that initializes the random-number
sequence.
• Any value in the sequence can be used to “seed” the generator.
A random-number stream:
• Refers to a starting seed taken from the sequence X0, X1, …,
XP.
• If the streams are b values apart, then stream i could defined
by starting seed:

Si = X b (i −1) for i = 1,2,, P b

• Older generators: b = 105; Newer generators: b = 1037.

Copyright ©2010 by K. Plantenberg

Restricted use only
 Generating Random numbers

5.4) Tests for Random Numbers

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4) Tests for Random Numbers

Desirable properties of random numbers: Uniformity and

Independence
Number of tests can be performed to check these properties been
achieved or not
Two type of tests:
• Frequency Test: Uses the Kolmogorov-Smirnov or the Chi-
square test to compare the distribution of the set of numbers
generated to a uniform distribution
• Autocorrelation test: Tests the correlation between numbers
and compares the sample correlation to the expected
correlation, zero

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4) Tests for Random Numbers

Two categories:
• Testing for uniformity. The hypotheses are:
H0: Ri ~ U[0,1]
H1: Ri ~ U[0,1]
– Failure to reject the null hypothesis, H0, means
that evidence of non-uniformity has not been
detected.
• Testing for independence. The hypotheses are:
H0: Ri ~ independently distributed
H1: Ri ~ independently distributed
– Failure to reject the null hypothesis, H0, means
that evidence of dependence has not been detected.
Copyright ©2010 by K. Plantenberg
Restricted use only
 5.4) Tests for Random Numbers

For each test, a Level of significance  must be stated.

The level   is the probability of rejecting the null hypothesis H0
when the null hypothesis is true:
 = P(reject H0|H0 is true)
The decision maker sets the value of  for any test
Frequently  is set to 0.01 or 0.05

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4) Tests for Random Numbers

When to use these tests:

• If a well-known simulation languages or random-number
generators is used, it is probably unnecessary to test
• If the generator is not explicitly known or documented, e.g.,
spreadsheet programs, symbolic/numerical calculators, tests
should be applied to many sample numbers.
Types of tests:
• Theoretical tests: evaluate the choices of m, a, and c without
actually generating any numbers
• Empirical tests: applied to actual sequences of numbers
produced. Our emphasis.

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4) Tests for Random Numbers

Test of uniformity
Two different methods:
• Kolmogorov-Smirnov test
• Chi-square test
Both these tests measure the degree of agreement between the
distribution of a sample of generated random numbers and the
theoretical uniform distribution
Both tests are based on null hypothesis of no significant difference
between the sample distribution and the theoretical distribution

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4.1) Kolmogorov-Smirnov Test
Compares the continuous cdf, F(x), of the uniform distribution with
the empirical cdf, SN(x), of the N sample observations.
• We know: F ( x) = x, 0  x  1
• If the sample from the RN generator is R1, R2, …, RN, then the
empirical cdf, SN(x) is:

number of R1 , R2 ,..., Rn which are  x

S N ( x) =
N
The cdf of an empirical distribution is a step function with jumps at
each observed value.

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4.1) Kolmogorov-Smirnov Test

Test is based on the largest absolute deviation statistic between F(x) and
SN(x) over the range of the random variable:
D = max| F(x) - SN(x)|
The distribution of D is known and tabulated (A.8) as function of N
Steps:
1. Rank the data from smallest to largest. Let R(i) denote ith smallest
observation, so that R(1)R(2)…R(N)
2. Compute i   i − 1
D + = max  − R(i ) ; D − = max  R(i ) − 

1i  N N
 1i  N
 N 
3. Compute D= max(D+, D-)
4. Locate in Table A.8 the critical value D, for the specified
significance level  and the sample size N
5. If the sample statistic D is greater than the critical value D, the
null hypothesis is rejected. If D D, conclude there is no difference

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4.1) Kolmogorov-Smirnov Test

Example: Suppose 5 generated numbers are 0.44, 0.81, 0.14, 0.05,

0.93.
Arrange R(i) from
R(i) 0.05 0.14 0.44 0.81 0.93
Step 1: smallest to largest
i/N 0.20 0.40 0.60 0.80 1.00

i/N – R(i) 0.15 0.26 0.16 - 0.07 D+ = max {i/N – R(i)}

Step 2: R(i) – (i-1)/N 0.05 - 0.04 0.21 0.13
D- = max {R(i) - (i-1)/N}

Step 3: D = max(D+, D-) = 0.26

Step 4: For  = 0.05,
D = 0.565 > D

Hence, H0 is not rejected.

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4.2) Chi-Square Test
Chi-square test uses the sample statistic:

n is the # of classes Ei is the expected

n
(Oi − Ei ) 2
 02 = 
# in the ith class

i =1 Ei Oi is the observed
# in the ith class

• Approximately the chi-square distribution with n-1 degrees of

freedom (where the critical values are tabulated in Table A.6)
• For the uniform distribution, Ei, the expected number in the
each class is:
N
Ei = , where N is the total # of observation
n
Valid only for large samples, e.g. N >= 50
Reject H0 if 02 > ,N-12

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4.2) Chi-Square Test
Example 7.7: Use Chi-square test for the data shown below with
=0.05. The test uses n=10 intervals of equal length, namely
[0,0.1),[0.1,0.2), …., [0.9,1.0)

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4.2) Chi-Square Test
The value of 02=3.4; The critical value from table A.6 is
0.05,92=16.9. Therefore the null hypothesis is not rejected

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4.3) Tests for Autocorrelation
Testing the autocorrelation between every m numbers (m is a.k.a.
the lag), starting with the ith number
• The autocorrelation im between numbers: Ri, Ri+m, Ri+2m,
Ri+(M+1)m
i + (M + 1 )m  N
• M is the largest integer such that
Hypothesis:H :  = 0, if numbers are independent
0 im

H1 : im  0, if numbers are dependent

If the values are uncorrelated: ̂im

• For large values of M, the distribution of the estimator of im,
denoted is approximately normal.

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4.3) Tests for Autocorrelation

Test statistics is: ˆ im

Z0 =
ˆ ˆ im

• Z0 is distributed normally with mean = 0 and variance = 1, and:

1 M 
ρˆ im =  
M + 1  k =0
Ri + km i +(k +1 )m  − 0.25
R

13M + 7
σˆ ρim =
12(M + 1 )

If im > 0, the subsequence has positive autocorrelation

• High random numbers tend to be followed by high ones, and vice
versa.
If im < 0, the subsequence has negative autocorrelation
• Low random numbers tend to be followed by high ones, and vice
versa.
Copyright ©2010 by K. Plantenberg
Restricted use only
 5.4.3) Tests for Autocorrelation
After computing Z0, do not reject the hypothesis of independence if
–z/2Z0  z/2
 is the level of significance and z/2 is obtained from table A.3

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4.3) Tests for Autocorrelation

Example: Test whether the 3rd, 8th, 13th, and so on, for the output on
Slide 37 are auto-correlated or not.
• Hence,  = 0.05, i = 3, m = 5, N = 30, and M = 4. M is the
largest integer such that 3+(M+1)530.

1 (0.23)(0.28) + (0.25)(0.33) + (0.33)(0.27)

ρˆ 35 =   − 0.25
4 + 1 + (0.28)(0.05) + (0.05)(0.36) 
= −0.1945
13(4) + 7
σˆ ρ35 = = 0.128
12( 4 + 1 )
0.1945
Z0 = − = −1.516
0.1280

• From Table A.3, z0.025 = 1.96. Hence, the hypothesis is not

rejected.

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.4.3) Tests for Autocorrelation

Shortcoming:
The test is not very sensitive for small values of M, particularly
when the numbers being tested are on the low side.
Problem when “fishing” for autocorrelation by performing numerous
tests:
• If  = 0.05, there is a probability of 0.05 of rejecting a true
hypothesis.
• If 10 independent sequences are examined,
– The probability of finding no significant
autocorrelation, by chance alone, is 0.9510 = 0.60.
– Hence, the probability of detecting significant
autocorrelation when it does not exist = 40%

Copyright ©2010 by K. Plantenberg

Restricted use only
 5.5) Caution

Caution:
• Even with generators that have been used for years, some of
which still in use, are found to be inadequate.
• This chapter provides only the basics
• Also, even if generated numbers pass all the tests, some
underlying pattern might have gone undetected.

Copyright ©2010 by K. Plantenberg

Restricted use only