THE CROSS-ENTROPY METHOD IN

OPTIMIZATION AND MONTE-CARLO

SIMULATION
Reuven Y. Rubinstein

Faculty of Industrial Engineering and Management, Technion, Israel

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 1/48

 Contents

1. Introduction
2. Methodology: Partially Connected TSP
3. A Motivating Example
4. Trajectory Generation
5. The Rare Events Framework
6. Calculating the Permanent, etc
7. The Minimum Cross-Entropy (MCE) Method
8. Some Theory on CE and MCE
9. Conclusion

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 2/48

 CE Matters

Book: R.Y. Rubinstein and D.P. Kroese.

The Cross-Entropy Method: A Unified
Approach to Combinatorial Optimization,
Monte Carlo Simulation and Machine Learn-
ing, Springer-Verlag, New York, 2004.

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 3/48

 CE Matters

Book: R.Y. Rubinstein and D.P. Kroese.

The Cross-Entropy Method: A Unified
Approach to Combinatorial Optimization,
Monte Carlo Simulation and Machine Learn-
ing, Springer-Verlag, New York, 2004.

Special Issue: Annals of Operations Research (Jan 2005).

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 3/48

 CE Matters

Book: R.Y. Rubinstein and D.P. Kroese.

The Cross-Entropy Method: A Unified
Approach to Combinatorial Optimization,
Monte Carlo Simulation and Machine Learn-
ing, Springer-Verlag, New York, 2004.

Special Issue: Annals of Operations Research (Jan 2005).

The CE home page:

http://www.cemethod.org

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 3/48

 Introduction

The Cross-Entropy Method was originally developed as a

simulation method for the estimation of rare event probabilities:
Estimate P(S(X) ≥ γ)

X: random vector/process taking values in some set X .

S: function on X .

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 4/48

 Introduction

The Cross-Entropy Method was originally developed as a

simulation method for the estimation of rare event probabilities:
Estimate P(S(X) ≥ γ)

X: random vector/process taking values in some set X .

S: function on X .

It was soon realised that the CE Method could also be used as an

optimization method:
Determine maxx∈X S(x)

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 4/48

 Applications

Rare-Event Probability Estimation (Queueing Models),

Estimation of Normalization Constant and Counting
Problems (Permanent Calculation, Self-Avoiding Random
Walks).

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 5/48

 Applications

Rare-Event Probability Estimation (Queueing Models),

Estimation of Normalization Constant and Counting
Problems (Permanent Calculation, Self-Avoiding Random
Walks).
Combinatorial Optimization, like TSP, Maximal Cut,
Scheduling and Production Lines.

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 5/48

 Applications

Rare-Event Probability Estimation (Queueing Models),

Estimation of Normalization Constant and Counting
Problems (Permanent Calculation, Self-Avoiding Random
Walks).
Combinatorial Optimization, like TSP, Maximal Cut,
Scheduling and Production Lines.
Machine Learning

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 5/48

 Applications

Rare-Event Probability Estimation (Queueing Models),

Estimation of Normalization Constant and Counting
Problems (Permanent Calculation, Self-Avoiding Random
Walks).
Combinatorial Optimization, like TSP, Maximal Cut,
Scheduling and Production Lines.
Machine Learning
Pattern Recognition, Clustering and Image Analysis

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 5/48

 Applications

Rare-Event Probability Estimation (Queueing Models),

Estimation of Normalization Constant and Counting
Problems (Permanent Calculation, Self-Avoiding Random
Walks).
Combinatorial Optimization, like TSP, Maximal Cut,
Scheduling and Production Lines.
Machine Learning
Pattern Recognition, Clustering and Image Analysis
DNA Sequence Alignment

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 5/48

 Applications

Rare-Event Probability Estimation (Queueing Models),

Estimation of Normalization Constant and Counting
Problems (Permanent Calculation, Self-Avoiding Random
Walks).
Combinatorial Optimization, like TSP, Maximal Cut,
Scheduling and Production Lines.
Machine Learning
Pattern Recognition, Clustering and Image Analysis
DNA Sequence Alignment
Simulation-based (noisy) Optimization, like Optimal Buffer
Allocation and Optimization in Finance Engineering
THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 5/48
 Applications

Rare-Event Probability Estimation (Queueing Models),

Estimation of Normalization Constant and Counting
Problems (Permanent Calculation, Self-Avoiding Random
Walks).
Combinatorial Optimization, like TSP, Maximal Cut,
Scheduling and Production Lines.
Machine Learning
Pattern Recognition, Clustering and Image Analysis
DNA Sequence Alignment
Simulation-based (noisy) Optimization, like Optimal Buffer
Allocation and Optimization in Finance Engineering
Multi-extremal Continuous
THE CROSS-ENTROPY Optimization
METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 5/48
 Applications

Rare-Event Probability Estimation (Queueing Models),

Estimation of Normalization Constant and Counting
Problems (Permanent Calculation, Self-Avoiding Random
Walks).
Combinatorial Optimization, like TSP, Maximal Cut,
Scheduling and Production Lines.
Machine Learning
Pattern Recognition, Clustering and Image Analysis
DNA Sequence Alignment
Simulation-based (noisy) Optimization, like Optimal Buffer
Allocation and Optimization in Finance Engineering
Multi-extremal Continuous
THE CROSS-ENTROPY Optimization
METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 5/48
 A Multi-extremal function

25

20

15

10

1
0.5
-1 0
-0.5
0 -0.5
0.5
1 -1

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 6/48

 A Maze Problem

The Optimal Trajectory

20

18

16

14

12

10

0
0 2 4 6 8 10 12 14 16 18 20

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 7/48

 A Maze Problem

Iteration 1:
20

18

16

14

12

10

0
0 2 4 6 8 10 12 14 16 18 20

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 8/48

 A Maze Problem

Iteration 2:
20

18

16

14

12

10

0
0 2 4 6 8 10 12 14 16 18 20

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 9/48

 A Maze Problem

Iteration 3:
20

18

16

14

12

10

0
0 2 4 6 8 10 12 14 16 18 20

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 10/48

 A Maze Problem

Iteration 4:
20

18

16

14

12

10

0
0 2 4 6 8 10 12 14 16 18 20

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 11/48

 Calculating Number of Trajectories
in a Partially Connected TSP

A Motivating Example
Consider the following distance matrix C of partially connected
TSP with n = 4.
0 1
0 c12 c13 c14
B C
B c 0 c23 c13 C
B 21 C
C=B C. (1)
B ∞ c32 0 c34 C
@ A
c41 ∞ c43 0

The associated 0-1 distance matrix D is

0 1
0 1 1 1
B C
B 1 0 1 0 C
B C
D=B C (2)
B 0 1 0 1 C
@ A
1 0 1 0

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 12/48

 Partially Connected TSP

Valid and Not Valid Trajectories

Define a valid TSP trajectory as one which visits each city only
once and goes trough all n cities with dij = 1. In our example
with 0 1
0 1 1 1
B C
B 1 0 1 0 C
B C
D=B C
B 0 1 0 1 C
@ A
1 0 1 0

we have a total of (n − 1)! = 3! = 6 trajectories, three of wich,

1 → 2 → 3 → 4 → 1, 1 → 3 → 2 → 4 → 1 and
1 → 4 → 3 → 2 → 1 are valid and the remaining 3 are invalid.
Denote |X | = (n − 1)! for a fully connected TSP. Let X − be a
subset of |X | and let |X − | be the desired cordinality of X − ,
respectively. We have |X | = 3! = 6 and |X − | = 3, respectively.
THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 13/48
 Partially Connected TSP

Trajectory Generation
We shall associate with each cost matrix D a probability matrix.
For our D matrix 0 1
0 1 1 1
B C
B 1 0 1 0 C
B C
D=B C
B 0 1 0 1 C
@ A
1 0 1 0

we can take, as example, the following initial one

0 1
0 1/3 1/3 1/3
B C
B 1/3 0 1/3 1/3 C
B C
P0 = B C. (3)
B 1/3 1/3 0 1/3 C
@ A
1/3 1/3 1/3 0

We shall call such TSP problem the 0-1-TSP.

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 14/48
 Partially Connected TSP

Trajectory Generation in a TSP

1. Define P (1) = P and X1 = 1. Let k = 1.

2. Obtain P (k+1) from P (k) by first setting the Xk -th column
of P (k) to 0 and then normalizing the rows to sum up to 1.
Generate Xk+1 from the distribution formed by the Xk -th
row of P (k) .
3. If k = n − 1 then stop; otherwise set k = k + 1 and
reiterate from step 2.

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 15/48

 Partially Connected TSP

The Rare Event Approach

Define
` = EX I{X∈A} , (4)
where |X | = (n − 1)! and A denotes the event that X is a valid
TSP trajectory in the sample space X and X in EX means that
the expectation is taken with respect to the uniform pdf f (x)
over X , that is
1 1
f (x) = = . (5)
|X | n − 1!

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 16/48

 Partially Connected TSP

The Rare Event Approach

Clearly, that
|X − | = `|X |.
It is obvious that the naive Monte-Carlo estimates of `, that is

XN
è = 1
I{X i ∈A} (6)
N i=1

in |Xe− | = è|X | is useless.

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 17/48

 Partially Connected TSP

The Rare Event Approach:

Importance Sampling
To speed up the simulation process we use importance sampling
(IS). In particular, we write ` as

` = EX {I{X∈A} W (X)}, (7)

where X in EX means that the random trajectory X is generated

according to the above TSP generation Algorithm.

f (x, P 0 )
W (x) =
f (x, P )

is the likelihood ratio with f (x) being uniformly distributed over

X , and f (x, P ) being the IS pdf.
THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 18/48
 Partially Connected TSP

The Rare Event Approach

The IS estimates of ` in and |X − | are

XN
ˆ 1 f (X i , P 0 )
`= I{X i ∈A} W (X i ), W (X i ) = (8)
N i=1 f (X i , P )

and
XN
b − b̀ 1 1
|X | = |X | = I{X i ∈A} , (9)
N i=1 f (X i , P )
respectively. The second equality holds since f (x) is uniformly
distributed over the set X , that is
1 1
f (x, P 0 ) = f1 (x1 )f2 (x2 |x1 ) · · · fn (xn |x1 , . . . , xn−1 ) = ···1
n−1n−2
THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 19/48
 Partially Connected TSP

The Rare Event Approach

We shall use interchangeable both events {X ∈ A} and
{S(X) = n} in (8) and (9), since

{X ∈ A} ≡ {S(X) = n}. (10)

Here S(X) is the length of the 0-1-TSP trajectory. Clearly, that

0 ≤ S(X) ≤ n and for a valid trajectory, that is for {X i ∈ A},
we have that S(X) = n. The optimal in the cross-entropy sense
matrix P is

P ∗ = argmax IEP 0 I{S(X)≥γ} log f (X; P ). (11)

P

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 20/48

 Partially Connected TSP

Example
For our example with
0 1
0 1 1 1
B C
B 1 0 1 0 C
B C
D=B C
B 0 1 0 1 C
@ A
1 0 1 0

and any initial P0 , the optimal P ∗ is

0 1
0 1/3 1/3 1/3
B C
B 1/2 0 1/2 0 C
∗ B C
P =B C. (12)
B 0 1/2 0 1/2 C
@ A
1/2 0 1/2 0

For example, for the trajectory 1 → 3 → 2 → 4 → 1 we have

f (x, P ) = 1/3 · 1 · 1 = 1/3.
THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 21/48
 Partially Connected TSP

Updating P and γ
Adaptive updating of γt . Let γt be the (1 − ρ)-quantile of
S(X) under P t−1 . A simple estimator, denoted γ bt , of γt can be
obtained by drawing a random sample X 1 , . . . , X N from
f (x, P t−1 ), calculating the associated function values
S(X 1 ), . . . , S(X N ) and their order statistics S(1) , . . . , S(N ) and
assigning γ bt to be these order statistics’ (1 − ρ)-quantile, that is,

γ
bt = S(1−ρ)N +1 . (13)

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 22/48

 Partially Connected TSP

Updating P and γ
Adaptive updating of P t . For fixed γt and P t−1 , derive P t
from the solution of the program

max D(P t ) = max IEP t−1 I{S(X)≥γt } log f (x, P t ) . (14)

Pt Pt

The stochastic counterpart of (14) is as follows: for fixed γ

bt and
e t−1 , derive P
P e t from the following program

XN
b e 1 e t) .
max D(P t ) = max I{S(X i )≥bγt } log f (X i ; P (15)
et
P et N
P
i=1

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 23/48

 Partially Connected TSP

The Rare Event Frame

Solving(14) and (15) for the TSP problem we obtain the
following simple analytic expressions

IEP t−1 I{X∈Xij } I{S(X)≥γt }

pt,ij = (16)
IEP t−1 I{S(X)≥γt }

and
N
X
I{X k ∈Xij } I{S(X k )≤bγt }
k=1
pet,ij = N
(17)
X
I{S(X k )≤bγt }
k=1

e t.
for updating the parameters of the matrices P t and P
THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 24/48
 The Main Algorithm

The 0-1-TSP Algorithm

b 0 , say p0,ij =
1. Choose some P 1
. Set t = 1.
n−1

2. Generate a sample X 1 , . . . , X N from the density

b t−1 ) and compute the sample (1 − ρ)-quantile γ
f (·; P bt
of the performances according to γ bt = S(1−ρ)N +1 .
3. Use the same sample X 1 , . . . , X N and solve the
stochastic program (15). Denote the solution by P b t.

4. Apply smooting P e t + (1 − α)P

b t = αP e t−1 , 0 < α < 1.

5. If γ̂t < n, set t = t + 1 and reiterate from step 2. Else

proceed with step 6.
6. Estimate |X − | according to (9).
THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 25/48
 Calculating the Permanent and More

The permanent problem can be stated as follows: Given an n x n

matrix A = (aij ) , where each element aij is either 0 or 1, and a
set of all permutations Π of the sequence {1, . . . , n}, find the
permanent of A, which is
n
XY
perm(A) = aiσ(i) .
σ∈Π i=1

Here σ ∈ Π is a permutation of {1, . . . , n}, which can be

represented as
 
1, 2, ..., n
 , (18)
σ(1), σ(2), . . . , σ(n)

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 26/48

 Calculating the Permanent

n
Y
RA = {σ : S(σ) = 1}, S(σ) = aiσ(i) , (19)
i=1

where σ(i) records the new position of label i after the

permutation.
Note that if one of the elements aiσ(i) , i = 1, . . . , n is zero, then
S(σ) is zero.
A 3 x 3 permanent matrix
 
1 1 1
 
A=  1 1 0 .

0 1 1

For theTHE
matrix A there are 3! = 6 permutations. They are:
CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 27/48
 Permanent Example

0 1
1 2 3
B C
B 1 3 2 C
B C
B C
B 2 1 3 C
σ=B
B
C,
C (20)
B 2 3 1 C
B C
B 3 1 2 C
@ A
3 2 1

with the corresponding S(σ) values

0 1
a11 a22 a33 =1
B C
B a11 a23 a32 =0 C
B C
B C
B a12 a21 a33 =1 C
S(σ) = B
B
C,
C (21)
B a12 a23 a31 =0 C
B C
B a13 a21 a32 =1 C
@ A
a13 a22 a33 =0

and the permanent perm(A) = 3.

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 28/48
 Calculating the Permanent

A naive way for approximating perm(A) is to generate the

permutations uniformly on the set Π and then to estimate it as
PN
j=1 S(σj )
[
perm(A) = n! (22)
N
However, for sparse matrix A we typically have perm(A) << n!,
that is under the uniform sampling over Π, the probability

` = P {S(σ) = 1} = IEU {IQni=1 aiσ(i) = 1} (23)

is very small and {S(σ) = 1} is rare event. Clearly that

perm(A) = `n! (24)

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 29/48

 Permanent Calculation

Note that ` can be also written equivalently as

` = P {S(σ) = n} = IEU {IPni=1 aiσ(i) = n}, (25)

where
n
X
S(σ) = aiσ(i) , (26)
i=1

which as more suitable for CE.

Here ` defines a probability that the length of a random
permutation (trajectory) equals exactly n. Thus we have a
longest tour TSP-like model, where (a) the distances between
and within cities are either zeros or ones; (b) TSP tour is defined
as follows: as soon as all cities are visited from some starting
city, one does not loop back, but rather remains there for ever.
THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 30/48
 Numerical Results

The Permanent Problem

As an illustration of the dynamics of the CE algorithm, we
display below the sequence of matrices P b 1, . . . , P
b 0, P b T (the
estimate of the true unknown matrix P t ) for the following
permanent matrix D

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 31/48

 Numerical Results

The Permanent Problem: Toy Example

D:

1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 1 1 1 1 1
1 1 1 1 1 0 0 0 0 0
1 1 1 1 1 0 0 0 0 0
1 1 1 1 1 0 0 0 0 0
1 1 1 1 1 0 0 0 0 0
1 1 1 1 1 0 0 0 0 0

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 32/48

 Numerical Results

The Permanent Problem

b1
P

0.079 0.065 0.037 0.044 0.044 0.142 0.135 0.142 0.135 0.177
0.044 0.079 0.058 0.058 0.051 0.142 0.121 0.107 0.149 0.191
0.037 0.037 0.44 0.072 0.037 0.191 0.177 0.093 0.177 0.135
0.058 0.072 0.044 0.044 0.051 0.128 0.135 0.177 0.142 0.149
0.044 0.051 0.037 0.044 0.051 0.128 0.205 0.142 0.163 0.135
0.180 0.166 0.208 0.145 0.201 0.006 0.006 0.062 0.006 0.020
0.173 0.166 0.208 0.145 0.201 0.125 0.006 0.006 0.006 0.006
0.145 0.173 0.166 0.194 0.117 0.006 0.083 0.104 0.006 0.006
0.152 0.201 0.166 0.166 0.243 0.006 0.006 0.006 0.048 0.006
0.208 0.131 0.180 0.159 0.166 0.006 0.006 0.041 0.048 0.055

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 33/48

 Numerical Results

The Permanent Problem

b2
P

0.0237 0.0195 0.0111 0.0132 0.0132 0.1756 0.2085 0.1406 0.1945 0.2001
0.0132 0.0237 0.0174 0.0174 0.0153 0.1896 0.1343 0.1511 0.1637 0.2743
0.0111 0.0111 0.0132 0.0216 0.0111 0.2463 0.1721 0.1329 0.2631 0.1175
0.0174 0.0216 0.0132 0.0132 0.0153 0.1784 0.2435 0.1686 0.1119 0.2295
0.0132 0.0153 0.0111 0.0132 0.0153 0.1784 0.2435 0.1686 0.1119 0.2295
0.1240 0.1968 0.3214 0.1625 0.1653 0.0018 0.0018 0.0186 0.0018 0.0060
0.2829 0.1065 0.1310 0.2122 0.2227 0.0375 0.0018 0.0018 0.0018 0.0018
0.2045 0.2339 0.1338 0.2262 0.1401 0.0018 0.0249 0.0312 0.0018 0.0018
0.1716 0.2353 0.1758 0.1618 0.2339 0.0018 0.0018 0.0018 0.0144 0.0018
0.1744 0.1723 0.2080 0.1947 0.2038 0.0018 0.0018 0.0123 0.0144 0.0165

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 34/48

 Numerical Results

The Permanent Problem

b 3:
P

0.0071 0.0058 0.0033 0.0039 0.0039 0.1891 0.2260 0.1421 0.1959 0.2223
0.0039 0.0071 0.0052 0.0052 0.0045 0.2015 0.1602 0.1629 0.1749 0.2740
0.0033 0.0033 0.0039 0.0064 0.0033 0.2644 0.1751 0.1539 0.2636 0.1223
0.0052 0.0064 0.0039 0.0039 0.0045 0.1423 0.1744 0.3115 0.2083 0.1391
0.0039 0.0045 0.0033 0.0039 0.0045 0.1782 0.2436 0.1988 0.1359 0.2229
0.1572 0.1708 0.2940 0.1911 0.1778 0.0005 0.0005 0.0055 0.0005 0.0018
0.2742 0.1413 0.1557 0.1824 0.2326 0.0112 0.0005 0.0005 0.0005 0.0005
0.2213 0.2289 0.1342 0.2219 0.1549 0.0005 0.0074 0.0093 0.0005 0.0005
0.1726 0.2505 0.1703 0.1708 0.2289 0.0005 0.0005 0.0005 0.0043 0.0005
0.1617 0.1916 0.2365 0.2007 0.1952 0.0005 0.0005 0.0036 0.0043 0.0049

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 35/48

 Numerical Results

The Permanent Problem

b 4:
P

0.0021 0.0017 0.0009 0.0011 0.0011 0.1784 0.2377 0.1570 0.1976 0.2219
0.0011 0.0021 0.0015 0.0015 0.0013 0.2017 0.1583 0.1779 0.1774 0.2766
0.0009 0.0009 0.0011 0.0019 0.0009 0.2533 0.1750 0.1687 0.2702 0.1265
0.0015 0.0019 0.0011 0.0011 0.0013 0.1635 0.1903 0.2796 0.2038 0.1552
0.0011 0.0013 0.0009 0.0011 0.0013 0.1955 0.2323 0.2074 0.1445 0.2139
0.1582 0.1786 0.2932 0.1945 0.1726 0.0001 0.0001 0.0016 0.0001 0.0005
0.2725 0.1494 0.1471 0.1780 0.2486 0.0033 0.0001 0.0001 0.0001 0.0001
0.2273 0.2149 0.1472 0.2441 0.1608 0.0001 0.0022 0.0028 0.0001 0.0001
0.1808 0.2499 0.1654 0.1786 0.2230 0.0001 0.0001 0.0001 0.0012 0.0001
0.1571 0.2020 0.2441 0.2007 0.1917 0.0001 0.0001 0.0011 0.0012 0.0014

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 36/48

 Numerical Results

The Permanent Problem: The optimal matrix P ∗0

P ∗0 :

0 0 0 0 0 0.2 0.2 0.2 0.2 0.2

0 0 0 0 0 0.2 0.2 0.2 0.2 0.2
0 0 0 0 0 0.2 0.2 0.2 0.2 0.2
0 0 0 0 0 0.2 0.2 0.2 0.2 0.2
0 0 0 0 0 0.2 0.2 0.2 0.2 0.2
0.2 0.2 0.2 0.2 0.2 0 0 0 0 0
0.2 0.2 0.2 0.2 0.2 0 0 0 0 0
0.2 0.2 0.2 0.2 0.2 0 0 0 0 0
0.2 0.2 0.2 0.2 0.2 0 0 0 0 0
0.2 0.2 0.2 0.2 0.2 0 0 0 0 0

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 37/48

 Numerical Results

The Permanent Problem: The Initial Matrix

P 0:

0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
0.18 0.18 0.18 0.18 0.18 0.02 0.02 0.02 0.02 0.02
0.18 0.18 0.18 0.18 0.18 0.02 0.02 0.02 0.02 0.02
0.18 0.18 0.18 0.18 0.18 0.02 0.02 0.02 0.02 0.02
0.18 0.18 0.18 0.18 0.18 0.02 0.02 0.02 0.02 0.02
0.18 0.18 0.18 0.18 0.18 0.02 0.02 0.02 0.02 0.02

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 38/48

 Numerical Results

The Permanent Problem

We set ρ = 0.1, α = 0.7, N = 10 ∗ n2 . In our tables below we
list besides |X̂T− |, (the (1 − ρ)-quantile of the performances) also
the best sample performance obtained at each iteration and
denoted by |Xˆt,(N
−
) |.

To indicate the convergence of P b t to the optimal P ∗ we

introduce the following quantities

Ptmm = min max pbt,ij , (27)

1≤i≤n 1≤j≤n

t = 1, 2, . . ., which corresponds to the min max value of the

elements of the matrix P b t,ij at iteration t, and, similarly, the max
min value
P METHOD
THE CROSS-ENTROPYt,mm
= INmax min pbAND. MONTE-CARLO SIMULATION
OPTIMIZATION t,ij
(28) – p. 39/48
1≤i≤n 1≤j≤n
 Numerical Results

The Permanent Problem

Table 1: A typical performance of Algo-

rithm 25 for the permanent matrix Dmn with
m = 10 and n = 100.
t |X̂t− | −
|X̂t,(N )
| ||P̂ t − P ∗ ||

1 0 5.50E-07 0.721
2 2.35E-17 4.28E-06 0.519
3 6.62E-14 0.003 0.427
4 6.14E-11 0.013 0.355
5 1.81E-08 0.131 0.296
6 2.24E-06 1.692 0.251
7 19.158 28.520 0.116
8 100 100 0.036

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 40/48

 Numerical Results

The Permanent Problem

Table 2: A typical performance of Algo-

rithm 25 for the permanent matrix Dmn with
m = 10 and n = 100.
t |X̂t− | −
|X̂t,(N )
| ||P̂ t − P ∗ ||

1 0 1.664 0.199
2 1.435E-10 0.753 0.160
3 3.592E-07 1.793 0.137
4 0.052 1.825 0.130
5 11.073 18.453 0.101
6 100 100 0.067
7 100 100 0.035

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 41/48

 Background on MCE

The MinxEnt Program (MCE)

n R o
minf (x) D(f |h) = ln fh(x)
(x)
f (x)dx = IEf ln fh(X)
(X)

R
(P0 ) s.t. Sj (x)f (x)dx = IEf Sj (X) = γj , j = 1, . . . , k,
R R
f (x)dx = 1, h(x)dx = 1.
(29)
Here f and h are joint n-dimensional pdf ’s or
n-dimensional pmf ’s, Sj (x), j = 1, . . . , k, are well
defined functions and x is an n-dimensional vector.
The pdf h is typically assumed to be known and is
called the prior pdf.

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 42/48

 Background on MCE

The MaxEnt Program

When h is unknown, it is taken as a uniform (continuous or
discrete) pdf. In this case
 Z 
f (x) f (x)
min D(f |h) = ln f (x)dx = IEf ln (30)
f (x) h(x) h(x)

reduces to
 Z 
max H(f ) = − f (x) ln f (x)dx = −IEf ln f (x) . (31)
f (x)

The original program (P0 ) is called the Kullback MinxEnt

program, while the program (P0 ) with (30) replaced by (31) is
called the Janes MaxEnt program.
THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 43/48
 Background on MCE

The Discrete Program:

Pm pi
minp D(p|u) = minp i=1 p i ln ui

Pm
(P1 ) s. t. i=1 Sj (xi )pi = IEp Sj (X) = γj , j = 1, . . . , k,

Pm Pm
i=1 pi = 1, i=1 ui = 1, pi ≥ 0, ui > 0.
(32)

X denotes a random variable with m possible outcomes.

u = (u1 , . . . , um ) denotes the prior probability vector of X.
p = (p1 , . . . , pm ) is the decision probability vector.

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 44/48

 Background on MCE

The Solution of the Discrete Program (P1 )

Pk ∗
ui exp{− j=1 λj Sj (xi )}
p∗i = Pm
u exp{−
Pk ∗ =
r=1 r j=1 λj Sj (xr )}

(33)
P
IEu I{X=xi } exp{− kj=1 λ∗j Sj (X)}
= P
IEu exp{− kj=1 λ∗j Sj (X)}
, i = 1, . . . , m,

where λ∗j , j = 1, . . . , n are obtained from the solution of

Pm Pk
i=1 S j (x i )u i exp { − r=1 Sr (xi )λr }
− Pm P + γj =
i=1 ui exp{− r=1 Sr (xi )λr }
k

(34)
P
IEu Sj (X) exp{− kr=1 Sr (X)λr }
= − IE exp − Pk S (X)λ + γj = 0.
u { r=1 r r}

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 45/48

 Background on MCE

A Toy Example Associated with Die Tossing

P6 pi
minp D(p|u) = minp p
i=1 i ln ui

P6
s. t. i=1 ipi = γ, (35)

P6 P6
i=1 pi = 1, i=1 ui = 1, pi ≥ 0, ui > 0.

The solution:
∗ ∗
∗ u i exp {−iλ } IE u I {X=i} exp {−Xλ }
p i = P6 = ∗
, i = 1, . . . , 6,
r=1 ur exp {−rλ }
∗ IEu exp {−Xλ }
(36)
and similarly λ∗ .
THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 46/48
 Conclusions

Advantages:
Universally applicable (discrete/continuous/mixed
problems)
Very easy to implement
Easy to adapt, e.g. when constraints are added
Works generally well
Disadvantages:
Performance function should be relatively cheap
Tweaking (modifications) may be required

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 47/48

 THANK YOU

THE CROSS-ENTROPY METHOD IN OPTIMIZATION AND MONTE-CARLO SIMULATION – p. 48/48