Chapter 7

Random-Number
Generation

Banks, Carson, Nelson & Nicol

Discrete-Event System Simulation
Purpose & Overview
 Discuss the generation of random numbers.

 Introduce the subsequent testing for

randomness:
 Frequency test
 Autocorrelation test.

2
Properties of Random Numbers
 Two important statistical properties:
 Uniformity
 Independence.
 Random Number, Ri, must be independently drawn from a
uniform distribution with pdf:

1, 0  x  1
f ( x)  
0, otherwise
1
2
1 x 1

E ( R )  xdx 
0 2
0

2
Figure: pdf for
random numbers
3
Generation of Pseudo-Random Numbers
 “Pseudo”, because generating numbers using a known
method removes the potential for true randomness.
 Goal: To produce a sequence of numbers in [0,1] that
simulates, or imitates, the ideal properties of random numbers
(RN).
 When Generating pseudo Random Numbers, Certain errors
or problems can occur including:
 Departure from uniformity
 Discrete instead of continuous
 Too high or too low mean / variance
 Dependency
 Autocorrelation,
 trend,
 several point above the mean followed by points lower the mean

4
Generation of Pseudo-Random Numbers
 Important considerations in RN routines:
 Fast
 Portable to different computers
 Have sufficiently long cycle (avoid cycling degenerating)
 Replicable
 Closely approximate the ideal statistical properties of uniformity
and independence.

5
Techniques for Generating Random
Numbers

 Linear Congruential Method (LCM).

 Combined Linear Congruential Generators (CLCG).

6
Linear Congruential Method [Techniques]

 To produce a sequence of integers, X1, X2, … between 0

and m-1 by following a recursive relationship:

X i 1  (aX i  c) mod m, i  0,1,2,...

The The The

multiplier increment modulus

 The selection of the values for a, c, m, and X0 drastically

affects the statistical properties and the cycle length.
 When c=0, the form is known as multiplicative Congruential
Method
 The random integers are being generated [0,m-1], and to
convert the integers to random numbers:
Xi
Ri  , i  1,2,...
m 7
Example [LCM]

 Use X0 = 27, a = 17, c = 43, and m = 100.

 The Xi and Ri values are:
X1 = (17*27+43) mod 100 = 502 mod 100 = 2, R1 = 0.02;
X2 = (17*2+43) mod 100 = 77, R2 = 0.77;
X3 = (17*77+43) mod 100 = 52, R3 = 0.52;
…

8
Characteristics of a Good Generator
[LCM]
 Maximum Density
 Such that he values assumed by Ri, i = 1,2,…, leave no large
gaps on [0,1]
 Problem: Instead of continuous, each Ri is discrete
 Solution: a very large integer for modulus m
 Maximum Period
 To achieve maximum density and avoid cycling.
 Achieve by: proper choice of a, c, m, and X0.
 Most digital computers use a binary representation of
numbers
 Speed and efficiency are aided by a modulus, m, to be (or close
to) a power of 2.

9
Combined Linear Congruential Generators
[Techniques]
 Reason: Longer period generator is needed because of the
increasing complexity of stimulated systems.
 Approach: Combine two or more multiplicative congruential
generators.
 Let Xi,1, Xi,2, …, Xi,k, be the ith output from k different
multiplicative congruential generators.
 The jth generator:
 Has prime modulus mj and multiplier aj and period is mj-1

 Produces integers Xi,j is approx ~ Uniform on integers in [1,

m-1]
 Wi,j = Xi,j -1 is approx ~ Uniform on integers in [1, m-2]

10
Combined Linear Congruential Generators
[Techniques]
 Suggested form:
 Xi
 k   m , Xi  0
X i    (1) X i , j  mod m1  1
 j 1
Hence, Ri   1
 j 1  m 1
 1 , Xi  0
 m1
The coefficient:
Performs the
subtraction Xi,1-1

 The maximum possible period is:

(m1  1)(m2  1)...(mk  1)
P
2 k 1

11
Combined Linear Congruential Generators
[Techniques]
 Example: For 32-bit computers, L’Ecuyer [1988] suggests combining
k = 2 generators with m1 = 2,147,483,563, a1 = 40,014, m2 =
2,147,483,399 and a2 = 20,692. The algorithm becomes:
Step 1: Select seeds
 X1,0 in the range [1, 2,147,483,562] for the 1st generator

 X2,0 in the range [1, 2,147,483,398] for the 2nd generator.

Step 2: For each individual generator,
X1,j+1 = 40,014 X1,j mod 2,147,483,563
X2,j+1 = 40,692 X1,j mod 2,147,483,399.
Step 3: Xj+1 = (X1,j+1 - X2,j+1 ) mod 2,147,483,562.
Step 4: Return  X j 1
 , X j 1  0
R j 1   2,147,483, 563
 2,147,483,562 , X j 1  0
 2,147,483,563
Step 5: Set j = j+1, go back to step 2.
 Combined generator has period: (m1 – 1)(m2 – 1)/2 ~ 2 x 1018
12
Random-Numbers Streams [Techniques]

 The seed for a linear congruential random-number generator:

 Is the integer value X0 that initializes the random-number sequence.
 Any value in the sequence can be used to “seed” the generator.
 A random-number stream:
 Refers to a starting seed taken from the sequence X0, X1, …, XP.
 If the streams are b values apart, then stream i could defined by starting
seed: Si  X b (i 1)

 Older generators: b = 105; Newer generators: b = 1037.

 A single random-number generator with k streams can act like k
distinct virtual random-number generators
 To compare two or more alternative systems.
 Advantageous to dedicate portions of the pseudo-random number
sequence to the same purpose in each of the simulated systems.

13
Tests for Random Numbers
 Two categories:
 Testing for uniformity:
H0: Ri ~ U[0,1]
H1: Ri ~/ U[0,1]
 Failure to reject the null hypothesis, H0, means that evidence of
non-uniformity has not been detected.
 Testing for independence:
H0: Ri ~ independently
H1: Ri ~/ independently
 Failure to reject the null hypothesis, H0, means that evidence of
dependence has not been detected.
 Level of significance the probability of rejecting H0 when it
is true:  = P(reject H0|H0 is true)
14
Tests for Random Numbers
 When to use these tests:
 If a well-known simulation languages or random-number
generators is used, it is probably unnecessary to test
 If the generator is not explicitly known or documented, e.g.,
spreadsheet programs, symbolic/numerical calculators, tests
should be applied to many sample numbers.
 Types of tests:
 Theoretical tests: evaluate the choices of m, a, and c without
actually generating any numbers
 Empirical tests: applied to actual sequences of numbers
produced. Our emphasis.

15
 Test for Random Numbers

1. Frequency test. Uses the Kolmogorov-Smirnov or the

chi-square test to compare the distribution of the set of
numbers generated to a uniform distribution.

2. Runs test. Tests the runs up and down or the runs

above and below the mean by comparing the actual
values to expected values. The statistic for
comparison is the chi-square.

3. Autocorrelation test. Tests the correlation between

numbers and compares the sample correlation to the
expected correlation of zero.

16 Random Numbers
 Test for Random Numbers (cont.)

4. Gap test. Counts the number of digits that appear

between repetitions of a particular digit and then uses
the Kolmogorov-Smirnov test to compare with the
expected number of gaps.

5. Poker test. Treats numbers grouped together as a

poker hand. Then the hands obtained are compared
to what is expected using the chi-square test.

17 Random Numbers
 Test for Random Numbers (cont.)

Level of significance 
 = P(reject H0 | H0 true)
Frequently,  is set to 0.01 or 0.05
(Hypothesis)
Actually True Actually False
Accept 1 - 
(Type II error)
Reject  1 – 
(Type I error)

18 Random Numbers
Frequency Tests [Tests for RN]

 Test of uniformity
 Two different methods:
 Kolmogorov-Smirnov test
 Chi-square test

19
Kolmogorov-Smirnov Test [Frequency Test]

 Compares the continuous cdf, F(x), of the uniform

distribution with the empirical cdf, SN(x), of the N sample
observations.
 We know: F ( x )  x, 0  x  1
 If the sample from the RN generator is R1, R2, …, RN, then the
empirical cdf, SN(x) is:
number of R1 , R2 ,..., Rn which are  x
S N ( x) 
N

 Based on the statistic: D = max| F(x) - SN(x)|

 Sampling distribution of D is known (a function of N, tabulated in Table
A.8.)
 A more powerful test, recommended.

20
Kolmogorov-Smirnov Test [Frequency Test]

 Example: Suppose 5 generated numbers are 0.44, 0.81, 0.14,

0.05, 0.93.
Arrange R(i) from
Step 1: R(i) 0.05 0.14 0.44 0.81 0.93 smallest to largest

i/N 0.20 0.40 0.60 0.80 1.00

D+ = max {i/N – R(i)}
i/N – R(i) 0.15 0.26 0.16 - 0.07
Step 2:
R(i) – (i-1)/N 0.05 - 0.04 0.21 0.13
D- = max {R(i) - (i-1)/N}

Step 3: D = max(D+, D-) = 0.26

Step 4: For  = 0.05,
D = 0.565 > D

Hence, H0 is not rejected.

21
Chi-square test [Frequency Test]

 Chi-square test uses the sample statistic:

n is the # of classes Ei is the expected
(Oi  Ei )
n 2 # in the ith class
 
2
0
i 1 Ei Oi is the observed
# in the ith class

 Approximately the chi-square distribution with n-1 degrees of

freedom (where the critical values are tabulated in Table A.6)
 For the uniform distribution, Ei, the expected number in the each
class is: N
Ei  , where N is the total # of observation
n

 Valid only for large samples, e.g. N >= 50

22
Test for Random Numbers (Run Tests )

 Run Tests (Up and Down)

Consider the 40 numbers; both the Kolmogorov-Smirnov
and Chi-square would indicate that the numbers are
uniformly distributed. But, not so.

0.08 0.09 0.23 0.29 0.42 0.55 0.58 0.72 0.89 0.91
0.11 0.16 0.18 0.31 0.41 0.53 0.71 0.73 0.74 0.84
0.02 0.09 0.30 0.32 0.45 0.47 0.69 0.74 0.91 0.95
0.12 0.13 0.29 0.36 0.38 0.54 0.68 0.86 0.88 0.91

23 Random Numbers
Run test (Up and Down)

Now, rearrange and there is less reason to doubt

independence.

0.41 0.68 0.89 0.84 0.74 0.91 0.55 0.71 0.36

0.30
0.09 0.72 0.86 0.08 0.54 0.02 0.11 0.29 0.16
0.18
0.88 0.91 0.95 0.69 0.09 0.38 0.23 0.32 0.91
0.53
0.31 0.42 0.73 0.12 0.74 0.45 0.13 0.47 0.58
0.29

24 Random Numbers
Run test (Up and Down)

Concerns:
 Number of runs
 Length of runs
Note: If N is the number of numbers in a sequence,
the maximum number of runs is N-1, and the
minimum number of runs is one.
If “a” is the total number of runs in a sequence, the
mean and variance of “a” is given by

25 Random Numbers
Run test (Up and Down)

a = (2n - 1) / 3
 2
a = (16N - 29) / 90
For N > 20, the distribution of “a” approximated
by a normal distribution, N(a ,  a ).
2

This approximation can be used to test the

independence of numbers from a generator.
Z0= (a - a) / a
26 Random Numbers
Run test (Up and Down)

Substituting for aanda ==>

Za = {a - [(2N-1)/3]} / {(16N-29)/90},
where Z ~ N(0,1)
Acceptance region for hypothesis of
independence -Z Z0  Z
 / 2  / 2

-Z  / 2
Z  / 2

27 Random Numbers
Run test (Up and Down)

(Example)
Based on runs up and runs down, determine
whether the following sequence of 40 numbers is
such that the hypothesis of independence can be
rejected where  = 0.05.

0.41 0.68 0.89 0.94 0.74 0.91 0.55 0.62 0.36 0.27
0.19 0.72 0.75 0.08 0.54 0.02 0.01 0.36 0.16 0.28
0.18 0.01 0.95 0.69 0.18 0.47 0.23 0.32 0.82 0.53
0.31 0.42 0.73 0.04 0.83 0.45 0.13 0.57 0.63 0.29

28 Random Numbers
Run test (Up and Down)

The sequence of runs up and down is as follows:

+ + + +  +  + +  + + + + + + + + + + + + 

There are 26 runs in this sequence. With N=40 and a=26,

a= {2(40) - 1} / 3 = 26.33 and
 2
=
a {16(40) - 29} / 90 = 6.79
Then,
Z0 = (26 - 26.33) / 
Now, the critical value is Z0.025 = 1.96, so the
independence of the numbers cannot be rejected on the
basis of this test.

29 Random Numbers
Run test (above and below the mean)
 Run Tests (above and below the mean)
 Consider the following 40 numbers, what do you notice about
them?:

0.63 0.72 0.79 0.81 0.52 0.94 0.83 0.93 0.87 0.67

0.54 0.83 0.89 0.55 0.88 0.77 0.74 0.95 0.82 0.86

0.43 0.32 0.36 0.18 0.08 0.19 0.18 0.27 0.36 0.34

0.31 0.45 0.49 0.43 0.46 0.35 0.25 0.39 0.47 0.41
•Expected mean = (0.00 + 0.99)/2 = 0.495
•The first, and last 20 numbers are above and
below the mean respectively.

30
Run test (above and below the mean)
 Let n1, n2 be the number of values above and
below the mean respectively.
 maximum number of runs = N = n1, n2. and the
minimum number of runs =1.
2n1n2 1
 b 
N 2
2n1n2 (2n1n2  N )
b 
2

N 2 ( N  1)

•For either n1 or n2 >20, b is approximately

normally distributed. b  b
Zo 
b 31
Run test (above and below the mean)
 Example 7.9
For the same sequence in example 7.8:
0.41 0.68 0.89 0.94 0.74 0.91 0.55 0.62 0.36 0.27
0.19 0.72 0.75 0.08 0.54 0.02 0.01 0.36 0.16 0.28
0.18 0.01 0.95 0.69 0.18 0.47 0.23 0.32 0.82 0.53
0.31 0.42 0.73 0.04 0.83 0.45 0.13 0.57 0.63 0.29
To test number of runs above and below the mean, we
assign (-) and (+) as follows:
-+++++++---++-+-------++----++-
-+-+--++-
n1 = 18 and n2 = 22.
number of runs b = 17.
32
Run test (above and below the mean)

2(18)(22) 1
 b   20.3
40 2
2(18)(22)[(18)(22)  40)
b 
2
 9.54
40 (40  1)
2

17  20.3
Z0   1.07
9.54

Since Z0.025 = 1.96 > 1.07, the hypothesis of

independence cannot be rejected by this test.

33
Run test (length of runs)
 For the following set of numbers:
0.16 0.27 0.58 0.63 0.45 0.21 0.72 0.87
0.27 0.87 0.27 0.15 0.92 0.85 …

 Consider the runs above and below the mean:

--++--++--++--++…

34
 Run test (length of runs)
 Let Yi be the number of runs of length I in a
sequence of N numbers, then..
 For runs up and down:

E(Y ) : 2 
2 3 2 
 N (i  3i 1) - (i  3i  i  4) : when..i  N  2
 
i (i  3)!  

E(Y ) : 2 : when..i  N 1
i N!

35
Run test (length of runs)
 For runs above and below the mean:
Nwi
E (Yi )  , , N  20
E(I )
n1 i n2 n1 n2 i
wi  ( )  ( )  ( )( ) , , N  20
N N N N
n1 n2
E(I )   , , N  20
n2 n1

 Where, wi is the approximate probability that a

run has a length i
 E(I) is the approximate expected length of a run.
36
Run test (length of runs)
 The approximate expected TOTAL number of runs of all
lengths in a sequence of length N:
 E(Ai)=N/E(I), N>20 (for runs above and below the mean).
 μa=(2N-1)/3 (for runs up and down).
 Chi square test is the most appropriate:
L
[O  E (Y )]2
 02   i i

i 1 E (Yi )
 L=N-1 for runs up and down, and N for runs above and
below the mean.
 If the numbers are independent, is chi square
distributed with L-1 degrees of freedom.
 02

37
Run test (length of runs)
 Example 7.10 (length of run up and down)

38
Run test (length of runs)
 Example 7.10 (length of run up and down)

The mean number of total runs is:

μa= 2(N-1)/3 = 2(60-1)/3 =39.67.
Thus, E(Yi) i>3 = 39.67 – SUM(E(Yi), i=1,2,3)
39
= 0.78.
Run test (length of runs)

40
Run test (length of runs)
 Example 7.11 (length of runs above and below the mean)

41
Run test (length of runs)

42
Run test (length of runs)

43
Tests for Autocorrelation [Tests for RN]

 Testing the autocorrelation between every m numbers

starting with the ith number
 The autocorrelation im between numbers: Ri, Ri+m, Ri+2m, Ri+(M+1)m
 M is the largest integer such that
i  (M  1 )m  N
 Hypothesis:

H 0 :  im  0, if numbers are independent

H1 :  im  0, if numbers are dependent
 If the values are uncorrelated:
 For large values of M, the distribution of the estimator of im,
denoted is approximately normal.
̂ im

44
Tests for Autocorrelation [Tests for RN]

 Test statistics is: ˆ im

Z0 
ˆ ˆim
 Z0 is distributed normally with mean = 0 and variance = 1, and:
1 M 
ˆρim   
M  1  k 0
Ri  km Ri (k 1 )m   0.25

13M  7
σˆ ρim 
12(M  1 )
 If im > 0, the subsequence has positive autocorrelation
 High random numbers tend to be followed by high ones, and vice versa.
 If im < 0, the subsequence has negative autocorrelation
 Low random numbers tend to be followed by high ones, and vice versa.

45
Example [Test for Autocorrelation]

 Test whether the 3rd, 8th, 13th, and so on, for the following
output on P. 265.
 Hence,  = 0.05, i = 3, m = 5, N = 30, and M = 4

1 (0.23)(0.28)  (0.25)(0.33)  (0.33)(0.27)

ρˆ 35    0.25
4  1  (0.28)(0.05)  (0.05)(0.36) 
 0.1945
13(4)  7
σˆ ρ35   0.128
12( 4  1 )
0.1945
Z0    1.516
0.1280
 From Table A.3, z0.025 = 1.96. Hence, the hypothesis is not
rejected.
46
Shortcomings [Test for Autocorrelation]

 The test is not very sensitive for small values of M,

particularly when the numbers being tests are on the low
side.
 Problem when “fishing” for autocorrelation by performing
numerous tests:
 If  = 0.05, there is a probability of 0.05 of rejecting a true
hypothesis.
 If 10 independence sequences are examined,
 The probability of finding no significant autocorrelation, by

chance alone, is 0.9510 = 0.60.

 Hence, the probability of detecting significant autocorrelation

when it does not exist = 40%

47
Gap test

 Consider the gap between the digit 3

 There is eighteenth 3’s and seventeenth gap
between digit 3.
 What is the probability of getting a gap length
of 10? 10 times

 P(gap of 10) = p(no 3) .. P(no 3) P(3)

= (0.9)10 (0.1)
48
Gap test

 In general, if we have a range of 10 digits

 P(t followed by exactly x non-t digits) = (0.9)10(0.1)
 The theoretical CDF for randomly ordered
digits is: x
p( gap  x)  F ( x)  0.1 (0.9) n  1  0.9 x 1
n 0
 The theoretical CDF is compared with the
observed cumulative frequency of gap lengths
using Kolmogorov-Smirnov test.

49
Steps for the Gap test

 1. specify the CDF for the theoretical frequency

distribution based on the selected class interval
width.
 2. Arrange the observed sample of gaps in a
cumulative distribution with these classes.
 3.Find D = maximum deviation between F(x) and
the cumulative frequency SN(X)
 4. determine Dα from the table A.8
 5.Reject the null hypothesis if D>Dα.

50
Gap test - Example

51
Gap test - Example

52
Poker test

 The poker test is based on the frequency with

which a certain digit is repeated in a series of
number.
e.g. 0.255 0.577 0.331 0.414 0.828 0.909 0.001
 In a three-digit numbers, there are only three
possible outcomes:
 The individual numbers can all be different
 The individual numbers can all be the same
 Their can be one pair of like digits.

 What is the probability of each outcome?

53
Poker test

 P(three digits are different) = P(2nd is different from 1st ) *

P(3rd is different from 1st and 2nd ) = 0.9 * 0.8 = 0.72

 P( three like digits) = P(2nd same as 1st digit) * P(3rd same

as 1st ) = 0.1* 0.1 = 0.01

 P(exactly one pair) = 1- [P(three different digits)+P(three

like digits)] = 1- (0.72+0.01) = 0.27.

 Chi-square test is used to compare expected with

observed frequencies of outcomes.

54
Poker test - Example

55
Summary
 In this chapter, we described:
 Generation of random numbers
 Testing for uniformity and independence

 Caution:
 Even with generators that have been used for years, some of
which still in used, are found to be inadequate.
 This chapter provides only the basic
 Also, even if generated numbers pass all the tests, some
underlying pattern might have gone undetected.

56