Unbiased Statistics of a Constraint Satisfaction

Problem – a Controlled-Bias Generator

Denis Berthier
Institut Telecom ; Telecom & Management SudParis
9 rue Charles Fourier, 91011 Evry Cedex, France

Abstract: We show that estimating the complexity (mean and It should be clear that the above question is independent of
distribution) of the instances of a fixed size Constraint Satisfaction a widely investigated problem, the NP-completeness of some
Problem (CSP) can be very hard. We deal with the main two types of CSPs. NP-completeness [7] supposes the CSP has a
aspects of the problem: defining a measure of complexity and parameter (such as the size of a Sudoku grid: 9x9, 16x16; or
generating random unbiased instances. For the first problem, we the number of resources and tasks in a resource allocation
rely on a general framework and a measure of complexity we problem) and one concentrates on worst case analysis as a
presented at CISSE08. For the generation problem, we restrict our function of this parameter. Here, on the contrary, we fix this
analysis to the Sudoku example and we provide a solution that also parameter (if any), we consider the various instances of this
explains why it is so difficult. fixed size CSP (e.g. all the 9x9 Sudoku puzzles) and we are
Keywords: constraint satisfaction problem, modelling and simulation, more interested in mean case than in worst case analysis.
unbiased statistics, Sudoku puzzle generation, Sudoku rating.
II. MINIMAL INSTANCES
I. INTRODUCTION
Instances of a fixed size CSP are defined by their givens (or
Constraint Satisfaction Problems (CSP) constitute a very clues): a given is a value pre-assigned to a variable of the
general class of problems. A finite CSP is defined by a finite CSP.
number of variables with values in fixed finite domains and a Instances of a CSP with several solutions cannot be solved
finite set of constraints (i.e. of relations they must satisfy); it in a purely constructive way: at some point, some choice must
consists of finding a value for each of these variables, such that be made. Such under-constrained instances can be considered
they globally satisfy all the constraints. General solving methods as ill-posed problems. We therefore concentrate on instances
are known [1, 2]. Most of these methods combine a blind search with a single solution.
algorithm (also called depth-first or breadth-first structured It should also be obvious that, given an instance of a CSP,
search, Trial and Error with Guessing, …) with some form of the more givens are added to it, the easier the resulting
pattern-based pruning of the search graph. instances should become – the limit being when all the non
In [3, 4, 5], we introduced a new general framework, based on given variables have only one possible value. This leads to the
the idea of a constructive, fully pattern-based solution and on the following definition: an instance of a CSP is called minimal if
concepts of a candidate (a value not yet known to be impossible) it has one and only one solution and it would have several
and a resolution rule (which allows to progressively eliminate solutions if any of its givens was deleted.
candidates). In [6], we introduced several additional notions, In statistical analyses, only samples of minimal instances
also valid for any CSP, such as those of a chain and a whip, and are meaningful because adding extra givens would multiply
we showed how these patterns lead to general and powerful the number of easy instances. We shall show that building
kinds of resolution rules. random unbiased samples of minimal instances may be very
The present paper relies on these general concepts (that are hard.
briefly recalled in order to make it as self-contained as possible)
to analyse another question: how can we define a measure of III. ZT-WHIPS AND THE ASSOCIATED MEASURE OF COMPLEXITY
complexity for the instances of a given “fixed size” CSP and
The following definitions were introduced in [3], in the
how can we estimate the statistical distribution of this Sudoku context, and generalised to the general CSP in [6].
complexity measure? As yet, this question has received little
Definition: two different candidates of a CSP are linked by
interest and it could hardly have, because any method allowing a direct contradiction (or simply linked) if one of the
blind search will rely on chance and hide the complexity of the constraints of the CSP directly prevents them from being true
various instances. With our constructive resolution approach, we at the same time in any state in which they are present (the
can define a realistic mesure of complexity.
fact that this notion does not depend on the state is fundamental). – as there can be no confusion between sets of rules and
If two candidates are not linked, they are said compatible. sets of instances, Ln is also used to name the set of minimal
For any CSP, two different candidates for the same variable instances of the CSP that can be solved with rules in Ln;
are always linked; but there may be additional direct – given an instance of a CSP, its ZT rating is defined as
contradictions; as expliciting them is part of modelling the CSP, the smallest n such that this instance is in Ln.
we consider them as given with the CSP.
In Sudoku, two different candidates n1r1c1 and n2r2c2 are In Sudoku, the zt-rating has a nice structural property: it is
linked if: (n1 ≠ n2 & r1c1 = r2c2) or (n1 = n2 & share-a-unit(r1c1, invariant under the (n, r, c) natural super-symmetries of the
r2c2)), where “share-a-unit” means “in the same row or in the game, i.e two puzzles that are isomorphic under any of these
same column or in the same block”. symmetries have the same zt-rating. For this reason, we
named zt-whips nrczt-whips [4, 5] and the zt-rating NRCZT.
A. zt-whips in a general CSP There was an anterior measure of complexity, the SER,
Definition: given a candidate Z (which will be called the based on a very different approach and compatible with the
target), a zt-whip of length n built on Z is a sequence L1, R1, L2, players intuition of complexity, but not invariant under
R2, … Ln, of 2n-1 (notice that there is no Rn) different candidates symmetries. It appears that the correlation coefficient
(alternatively called left-linking and right-linking candidates) for (computed on several collections of a million puzzles each)
possibly different variables, such that, additionally: between the NRCZT and the SER is always high: 0.895.
– for any 1 ≤ k ≤ n, Lk is linked to Rk-1 (setting R0 = Z), Finally, there is also a very good correlation between the
– for any 1 ≤ k < n, Lk and Rk are candidates for the same NRCZT and the logarithm of the number of partial whips
variable (and they are therefore linked), used in the resolution process: 0.946. This number is an
intuitive measure of complexity, because it indicates among
– Rk is the only candidate for this variable compatible with
Z and with the previous right-linking candidates (i.e. with all the how many useless whips the useful ones must be found.
Rj, for j < k), These two properties show that the NRCZT rating is a good
(logarithmic) measure of complexity, from both theoretical
– for the same variable as Ln, there is no candidate
and pragmatic points of view. We can therefore conclude that
compatible with the target and the previous right-linking
the first task we had set forth is accomplished.
candidates.
zt-whip theorem for a general CSP: in any knowledge state
C. First statistical results for the Sudoku nrczt-whips
of any CSP, if Z is a target of a zt- whip of any length, then it
In the Sudoku case, we have programmed all the rules for
can be eliminated (formally, this rule concludes ¬Z). The proof
nrczt-whips in our SudoRules solver, a knowledge based
was given in [6].
system, running indifferently on the CLIPS [8] or the JESS
[9] inference engine.
B. The ZT measure of complexity
The following statistics are relative to a sample of one
For any CSP, we are now in a position to define an increasing
million puzzles obtained with the suexg [10] top-down
sequence of theories (i.e. sets of resolution rules) based on zt-
random generator. This was our first, naive approach to the
whips, an increasing sequence of sets of minimal puzzles solved
generation problem: using a generator of random minimal
by these theories and a rating for these instances:
puzzles widely available and used by the Sudoku community.
– L0 is the set of resolution rules expressing the Row 2 of Table 1 below gives the number of puzzles with
propagation of constraints (elimination of candidates due to the
NRCZT rating n. Row 3 gives the total number of puzzles
presence of a value for a variable) and of resolution rules
solved when whips of length ≤ n (corresponding to resolution
asserting values for variables that have a unique candidate left;
theory Ln) are allowed. This shows that more than 99% of the
– for any n>0, Ln is the union of L0 with set of resolution puzzles can be solved with whips of length ≤ 5 and more than
rules for whips of length ≤ n. 99.9% with whips of length ≤ 7. But there remain a few
exceptional cases with much larger complexity.

0 1 2 3 4 5 6 7 8 9 10 11 12 13
417,624 120,618 138,371 168,562 122,946 24,187 5,511 1,514 473 130 38 15 9 2
417,624 538,242 676,613 845,175 968,121 992,308 997,819 999,333 999,806 999,936 999,974 999,989 999,998 1,000,000

Table 1: Number of puzzles in 1,000,000 with NRCZT rating n (row2) and solved with nrczt-whips of length ≤ n (row 3).
 IV. STANDARD TOP-DOWN AND BOTTOM-UP GENERATORS V. A CONTROLLED-BIAS GENERATOR
A little after the above results were obtained, additional No generator of minimal puzzles is currently guaranteed to
statistics led to suspect that the top-down suexg generator may have no bias and building such a generator with reasonable
have some bias. There is a very simple procedure for generating computation times seems out of reach.
an unbiased Sudoku puzzle: We therefore decided to proceed differently: taking the
1) generate a random complete grid P;
2) for each cell in P, delete its value with
generators (more or less) as they are and applying corrections
probability ½, thus obtaining a puzzle Q; for the bias, if we can estimate it.
3)if Q is minimal, return it, otherwise goto 1. The method was inspired by what is done in cameras:
Unfortunately, the probability of getting a valid puzzle this instead of complex optimisations of the lenses to reduce
way is infinitesimal and one has to rely on other generators. typical anomalies (such as chromatic aberration, purple
Before going further, we introduce the two classical algorithms fringing, barrel or pincushion distortion…) – optimisations
for generating minimal puzzles: bottom-up and top-down. that lead to large and expensive lenses –, some camera makers
now accept a small amount of these in the lenses and they
A. The classical bottom-up and top-down generators [12] correct the result in real time with dedicated software before
A standard bottom-up generator works as follows to produce recording the photo.
one minimal puzzle (it has to be iterated n times to produce n The main question was then: can we determine the bias of
minimal puzzles): the classical top-down or bottom-up generators? Once again,
1) start from an empty grid P
2a) in P, randomly choose an undecided cell and a the answer was negative. But there appears to be a medium
value for it, thus getting a puzzle Q;
2b) if Q is minimal, return it and exit;
way between “improving the lens” and “correcting its small
2b) if Q has several solutions, set P = Q and GOTO 2a; defects by software”: we devised a modification of the top-
2c) if Q has no solutions, then goto 2a (i.e. down generators such that it allows a precise mathematical
backtrack, forget Q and try another cell).
computation of the bias.
A standard top-down generator works as follows to produce
one minimal puzzle (it has to be iterated n times to produce n
A. Definition of the controlled-bias generator
minimal puzzles):
1) choose randomly a complete grid P; Consider the following, modified top-down generator, the
2a) choose one clue randomly from P and delete it, controlled-bias generator; the procedure described below
thus obtaining a puzzle P2;
2b) if P2 has several solutions, GOTO 2a (i.e. produces one minimal puzzle (it has to be iterated n times to
reinsert the clue just deleted and try deleting produce n minimal puzzles):
another); 1) choose randomly a complete grid P;
2c) if P2 is minimal, printout P2 and exit the whole 2a) choose one clue randomly from P and delete it,
procedure; set P2 = the resulting puzzle;
2d) otherwise (the puzzle has more than one solution), 2b) if P2 has several solutions, GOTO 1 (i.e.
set P=P2 and GOTO 2a. restart with another complete grid);
Clause 2c in the bottom-up case and clause 2b in the top-down 2c) if P2 is minimal, printout P2 and exit the whole
procedure;
case make any analysis very difficlut. Moroever, it seems that 2d) otherwise (the puzzle has more than one
they also cause the generator to look for puzzles with fewer solution), set P=P2 and GOTO 2a
clues. It may thus be suspected of introducing a strong,
uncontrolled bias with respect to the number of clues. The only difference with the top-down algorithm is in
clause 2b: if a multi-solution puzzle is encountered, instead of
C. Existence of a bias and a (weak) correlation backtracking to the previous state, the current complete grid is
The existence of a (as yet non measurable) bias in the merely discarded and the search for a minimal puzzle is
number-of-clues distribution may in itself introduce a bias in the restarted with another complete grid.
distribution of complexities (measured by the NRCZT or SER Notice that, contrary to the standard bottom-up or top-down
ratings). This bias may not be very large, as the correlation generators, which produce one minimal puzzle per complete
coefficient between the number of clues and the NRCZT or the grid, the controlled-bias generator will generally use several
SER was estimated (on our 1,000,000-puzzle sample) to be only complete grids before it outputs a minimal puzzle. The
0.12. But it cannot be completely neglected either because it is efficiency question is: how many? Experimentations show
an indication that other kinds of bias, with a potentially larger that many complete grids (approximately 250,000 in the
impact, may be present in these generators. mean) are necessary before a minimal puzzle is reached. But
this question is about the efficiency of the generator, it is not a - recursive step: given floor n+1 (each doubly indexed
conceptual problem. puzzle of which has n+1 clues and is indexed by a complete
The controlled-bias generator has the same output and will grid that solves it and by a sequence of length 81-(n+1)), build
therefore produce minimal puzzles according to the same floor n as follows:
probability distribution as its following “virtual” counterpart: each doubly indexed puzzle Q at floor n+1 sprouts n+1
Repeat until a minimal puzzle has been printed: branches; for each clue C in Q, there is a branch leading to a
1) choose randomly a complete grid P;
2) repeat while P has at least one clue: doubly indexed puzzle R at floor n: R is obtained from Q by
2a) choose one clue randomly from P and delete it, removing clue C; its first index is identical to that of Q and its
thus obtaining a puzzle P2;
2b) if P2 is minimal, print P2 (but do not exit the second index is the (81-n)-element sequence obtained by
procedure); appending C to the end of the second index of Q; notice that
2c) set P=P2.
all the doubly indexed puzzles at floor n have n clues and the
The only difference with the controlled-bias generator is that,
length of their second index is equal to 1 + (81-(n+1)) = 81-n.
once it has found a minimal or a multi-solution puzzle, instead
It is easy to see that, at floor n, each doubly indexed puzzle
of exiting, this virtual generator continues along a useless path
has an underlying singly indexed puzzle identical to that of
until it reaches the empty grid.
(81 - n)! doubly indexed puzzles with the same first index at
But this virtual generator is interesting theoretically because it
the same floor (including itself).
works similarly to the random uniform search defined in the next
This is equivalent to saying that, at any floor n < 81, any
section and according to the same transition probabilities and it
singly indexed puzzle Q can be reached by exactly (81 - n)!
outputs minimal puzzles according to the probability Pr on the
different paths from the top (all of which start necessarily
set B of minimal puzzles defined below.
from the complete grid defined as the first index of Q). These
B. Analysis of the controlled-bias generator paths are the (81 - n)! different ways of deleting one by one its
We now build our formal model of this generator. missing 81-n clues from its solution grid.
Let us introduce the notion of a doubly indexed puzzle. We Notice that this would not be true for non indexed puzzles
consider only (single or multi solution) consistent puzzles P. The that have multiple solutions. This is where the first index is
double index of a doubly indexed puzzle P has a clear intuitive useful.
meaning: the first index is one of its solution grids and the Let N be the number of complete grids (N is known to be
second index is a sequence (notice: not a set, but a sequence, i.e. close to 6.67x1021, but this is pointless here). At each floor n,
an ordered set) of clue deletions leading from this solution to P. there are N . 81! / n! doubly indexed puzzles and N . 81! / (81-
In a sense, the double index keeps track of the full generation n)! / n! singly indexed puzzles. For each n, there is therefore a
process. uniform probability P(n) = 1/N . 1/81! . (81-n)! . n! that a
Given a doubly indexed puzzle Q, there is an underlying singly indexed puzzle Q at floor n is reached by a random
singly-indexed puzzle: the ordinary puzzle obtained by (uniform) search starting from one of the complete grids.
forgetting the second index of Q, i.e. by remembering the What is important here is the ratio:
solution grid from which it came and by forgetting the order of P(n+1) / P(n) = (n + 1) / (81 - n).
the deletions leading from this solution to Q. This formula is valid globally if we start from all the
Given a doubly indexed puzzle Q, there is also a non indexed complete grids, as above, but it is also valid for all the single
puzzle, obtained by forgetting the two indices. solution puzzles if we start from a single complete grid (just
Notice that, for a single solution doubly indexed puzzle, the forget N in the proof above). (Notice however that it is not
first index is useless as it can be computed from the puzzle; in valid if we start from a subgrid instead of a complete grid.)
this case singly indexed and non indexed are equivalent. In terms
of the generator, it could equivalently output minimal puzzles or Now, call B the set of (non indexed) minimal puzzles. On
couples (minimal-puzzle, solution). B, all the puzzles are minimal. Any puzzle strictly above B
Consider now the following layered structure (a forest, in the has redundant clues and a single solution. Notice that, for all
graph-theoretic sense, i.e. a set of disjoint trees, with branches the puzzles on B and above B, singly indexed and non
pointing downwards), the nodes being (single or multi solution) indexed puzzles are in one-to-one correspondence.
doubly indexed puzzles: On the set B of minimal puzzles there is a probabily Pr
- floor 81 : the N different complete solution grids (considered naturally induced by the different Pn's and it is the probability
as puzzles), each indexed by itself and by the empty sequence; that a minimal puzzle Q is output by our controlled-bias
notice that all the puzzles at floor 81 have 81 clues; generator. It depends only on the number of clues and it is
defined, up to a multiplicative constant k, by Pr(Q) = k P(n),
if Q has n clues. k must be chosen so that the probabilities of all The corrected, unbiased mean of X must be estimated as
the minimal puzzles sum up to 1. (this is a mere weighted average):
But we need not know k. What is important here is that, by unbiased-mean(X) =
construction of Pr on B (a construction which models the sum[E(X, n).on(n).cf(n)] / sum[on(n).cf(n)].
workings of the virtual controlled bias generator), the Similarly, the raw standard deviation of X is classically
fundamental relation Pr(n+1) / Pr(n) = (n + 1) / (81 - n) holds estimated as: sqrt{sum[sd(X, n)2 . on(n)] / sum[on(n)]}.
for any two minimal puzzles, with respectively n+1 and n clues. And the unbiased standard deviation of X must be
For n < 41, this relation means that a minimal puzzle with n estimated as (this is merely the standard deviation for a
clues is more likely to be reached from the top than a minimal weighted average):
puzzle with n+1 clues. More precisely, we have: unbiased-sd(X) =
Pr(40) = Pr(41), sqrt{sum[sd(X, n)2.on(n).cf(n)] / sum[on(n).cf(n)]}.
Pr(39) = 42/40 . Pr(40), These formulæ show that the cf(n) sequence needs to be
Pr(38) = 43/39 . Pr(39). defined only modulo a multiplicative factor.
Repeated application of the formula gives Pr(24) = It is convenient to choose cf(26) = 1. This gives the
61.11 Pr(30) : a puzzle with 24 clues has ~ 61 more chances of following sequence of correction factors (in the range n = 19-
being output than a puzzle with 30 clues. This is indeed a strong 31, which includes all the puzzles of all the samples we have
bias. obtained with all the random generators considered here):
[0.00134 0.00415 0.0120 0.0329 0.0843 0.204 0.464 1
A non-biased generator would give the same probability to all 2.037 3.929 7.180 12.445 20.474]
the minimal puzzles. The above relation shows that the It may be shocking to consider that 30-clue puzzles in a
controlled bias generator: sample must be given a weight 61 times greater than a 24-clue
- is unbiased when restricted (by filtering its output) to n- puzzle, but it is a fact. As a result of this strong bias of the
clue puzzles, for any fixed n, controlled-bias generator (strong but known and smaller than
- is biased towards puzzles with fewer clues, the other generators), unbiased statistics for the mean number
- this bias is well known. of clues of minimal puzzles (and any variable correlated with
As we know precisely the bias with respect to uniformity, we this number) must rely on extremely large samples with
can correct it easily by applying correction factors cf(n) to the sufficiently many 29-clue and 30-clue puzzles.
probabilities on B. Only the relative values of the cf(n) is
important: they satisfy cf(n+1) / cf(n) = (81 - n) / (n + 1). D. Implementation, experimentations and optimisations of the
Mathematically, after normalisation, cf is just the relative controlled-bias generator
density of the uniform distribution on B with respect to the Once this algorithm was defined, it was easily implemented
probability distribution Pr. by a simple modification of the top-down suexg – call it
This analysis also shows that a classical top-down generator suexg-cb. The modified generator, even after some
is still more strongly biased towards puzzles with fewer clues optimisations, is much slower than the original one, but the
because, instead of discarding the current path when it meets a purpose here is not speed, it is controlled bias.
multi-solution puzzle, it backtracks to the previous floor and
tries again to go deeper. V. COMPARISON OF RESULTS FOR DIFFERENT GENERATORS
All the results below rely on very large samples. Real
C. Computing unbiased means and standard deviations using a values are estimated according to the controlled-bias theory.
controlled-bias generator
In practice, how can one compute unbiased statistics of A. Complexity as a function of the generator
minimal puzzles based on a (large) sample produced by a
controlled-bias generator? Consider any random variable X Generator bottom-up top-down ctr-bias real
defined (at least) on minimal puzzles. Define: on(n) = the sample size 1,000,000 1,000,000 5,926,343
number of n-clue puzzles in the sample, E(X, n) = the mean
SER mean 3.50 3.77 4.29 4.73
value of X for n-clue puzzles in the sample and sd(X, n) = the
std dev 2.33 2.42 2.48 2.49
standard deviation of X for n-clue puzzles in the sample.
NRCZT mean 1.80 1.94 2.22 2.45
The (raw) mean of X is classically estimated as: sum[E(X, n) .
std dev 1.24 1.29 1.35 1.39
on(n)] / sum[on(n)].
Table 2: SER and NRCZT means and standard deviations for bottom-up,
top-down, controlled-bias generators and real estimated values.
 0 1 2 3 4 5 6 7 8 9 10 11 12-16
bottom-up 46.27 13.32 12.36 15.17 10.18 1.98 0.49 0.19 0.020 0.010 0* 0.01 * 0*
top-down 41.76 12.06 13.84 16.86 12.29 2.42 0.55 0.15 0.047 0.013 3.8e-3 1.5e-3 1.1e-3
ctr-bias 35.08 9.82 13.05 20.03 17.37 3.56 0.79 0.21 0.055 0.015 4.4e-3 1.2e-3 4.3e-4
real 29.17 8.44 12.61 22.26 21.39 4.67 1.07 0.29 0.072 0.020 5.5e-3 1.5e-3 5.4e-4
Table 3: The NRCZT-rating distribution (in %) for different kinds of generators, compared to the real distribution.

Table 2 shows that the mean (NRCZT or SER) complexity of

minimal puzzles depends strongly on the type of generator used VI. STABILITY OF THE CLASSIFICATION RESULTS
to produce them and that all the generators give rise to mean
complexity below the real values. A. Insensivity of the controlled-bias generator to the source of
Table 3 expresses the NRCZT complexity bias of the three complete grids
kinds of generators. All these distributions have the same two There remains a final question: do the above results depend
modes, at levels 0 and 3, as the real distribution. But, when one on the source of complete grids. Until now, we have done as if
moves from bottom-up to top-down to controlled-bias to real, this was not a problem. But producing unbiased collections of
the mass of the distribution moves progressively to the right. complete grids, necessary in the first step of all the puzzle
This displacement towards higher complexity occurs mainly at generators, is all but obvious. It is known that there are 6.67 x
the first nrczt-levels, after which it is only very slight. 1021 complete grids; it is therefore impossible to have a
generator scan them all. Up to isomorphisms, there are “only”
B. Number-of-clues distribution as a function of the generator 5.47 x 109 complete grids, but this remains a very large
Table 4 partially explains Tables 2 and 3. More precisely, it number and storing them would require about 500 Gigabytes.
explains why there is a strong complexity bias in the samples Very recently, a collection of all the (equivalence classes
produced by the bottom-up and top-down generators, in spite of of) complete grids became available in a compressed format
the weak correlation coefficient between the number of clues (6 Gb); at the same time, a real time decompressor became
and the (SER or NRCZT) complexity of a puzzle: the bias with available. Both were provided by Glenn Fowler. All the
respect to the number of clues is very strong in these generators; results reported above for the controlled bias generator were
controlled-bias, top-down and bottom-up are increasingly obtained with this a priori unbiased source of complete grids.
biased towards easier puzzles with fewer clues. Before this, all the generators we tried had a first phase
consisting of creating a complete grid and this is where some
#clues bottom-up % top-down % ctr-bias % real % type of bias could slip in. Nevertheless, several sources of
20 0.028 0.0044 0.0 0.0 complete grids based on very different generation principles
21 0.856 0.24 0.0030 0.000034 were tested and the classification results remained very stable.
The insensitivity of the controlled-bias generator to the
22 8.24 3.45 0.11 0.0034
source of complete grids can be understood intuitively: it
23 27.67 17.25 1.87 0.149
deletes in the mean two thirds of the initial grid data and any
24 36.38 34.23 11.85 2.28
structure that might be present in the complete grids and cause
25 20.59 29.78 30.59 13.42 a bias is washed away by the deletion phase.
26 5.45 12.21 33.82 31.94
27 0.72 2.53 17.01 32.74 B. Insensivity of the classification results to the choice of
28 0.054 0.27 4.17 15.48 whips or braids
29 0.0024 0.017 0.52 3.56 In [6], in addition to the notion of a zt-whip, we introduced
30 0 0.001 0.035 0.41 the apparently much more general notion of a zt-braid, to
31 0 0 0.0012 0.022 which a B-NRCZT rating can be associated in the same way
as the NRCZT rating was associated to zt-whips. The above
mean 23.87 24.38 25.667 26.577
statistical results are unchanged when NRCZT is replaced by
std-dev 1.08 1.12 1.116 1.116
B-NRCZT. Indeed, in 10,000 puzzles tested, only 20 have
different NRCZT and B-NRCZT ratings. The NRCZT rating
Table 4: Number-of-clues distribution (%) for the bottom-up, top-down and
is thus a good approximation of the (harder to compute) B-
controlled-bias generators and real estimated values.
NRCZT rating.
 VII. COLLATERAL RESULTS One can also get, still with 0.065% relative error: after
multiplying by the number of complete grids (known to be
The number of minimal Sudoku puzzles has been a
6,670,903,752,021,072,936,960 [13]), the total number of
longstanding open question. We can now provide precise
minimal Sudoku puzzles: 3.1055e+37; after multiplying by
estimates for the mean number of n-clue minimal puzzles per
the number of non isomorphic complete grids (known to be
complete grid and for the total number (Table 5).
5,472,730,538 [14]), the total number of non isomorphic
minimal Sudoku puzzles: 2.5477e+25.
number of clues number of n-clue number of n-clue
minimal puzzles minimal puzzles
VIII. CONCLUSION
per complete grid: per complete grid:
mean relative error The results reported in this paper rely on several months of
(= 1 std dev) (3 GHz) CPU time. They show that building unbiased samples
20 6.152e+6 70.7% of a CSP and obtaining unbiased statistics can be very hard.
21 1.4654e+9 7.81% Although we presented them, for definiteness, in the
22 1.6208e+12 1.23% specific context of the Sudoku CSP, the sample generation
23 6.8827e+12 0.30% methods described here (bottom-up, top-down and controlled-
24 1.0637e+14 0.12% bias) could be extended to many CSPs. The specific
24 6.2495e+14 0.074% P(n+1)/P(n) formula proven for the controlled-bias generator
will not hold in general, but the general approach can in many
26 1.4855e+15 0.071%
cases help understand the existence of a very strong bias in the
27 1.5228e+15 0.10%
samples. Even in the very structured and apparently simple
28 7.2063e+14 0.20%
Sudoku domain, none of this was clear before the present
29 1.6751e+14 0.56%
analysis.
30 1.9277e+13 2.2%
31 1.1240e+12 11.6% REFERENCES
32 4.7465e+10 70.7%
[1] E.P.K. Tsang, Foundations of Constraint Satisfaction, Academic Press,
Total 4.6655e+15 0.065%
1993.
Table 5: Mean number of n-clue minimal puzzles per complete grid
[2] H.W. Guesgen & J. Herztberg, A Perspective of Constraint-Based
Reasoning, Lecture Notes in Artificial Intelligence, Springer, 1992.
Another number of interest is the mean proportion of n-clue [3] D. Berthier: The Hidden Logic of Sudoku, Lulu.com, May 2007.
minimal puzzles among the n-clue subgrids of a complete [4] D. Berthier: The Hidden Logic of Sudoku (Second Edition), Lulu.com
grids. Its inverse is the mean number of tries one should do to Publishers, December 2007.
find an n-clue minimal by randomly deleting 81-n clues from a [5] D. Berthier: From Constraints to Resolution Rules; Part I: conceptual
complete grid. It is given by Table 6. framework, CISSE08/SCSS, International Joint Conferences on
Computer, Information, and System Sciences, and Engineering,
number of clues mean number of tries December 4-12, 2009.
20 7.6306e+11 [6] D. Berthier: From Constraints to Resolution Rules; Part II: chains, braids,
confluence and T&E, CISSE08/SCSS, International Joint Conferences on
21 9.3056e+9
Computer, Information, and System Sciences, and Engineering,
22 2.2946e+8
December 4-12, 2009.
23 1.3861e+7 [7] M. Gary & D. Johnson, Computers and Intractability: A Guide to the
24 2.1675e+6 Theory of NP-Completeness, Freeman, 1979.
24 8.4111e+5 [10] G. Riley, CLIPS online documentation, http://clipsrules.sourceforge.net/
26 7.6216e+5 OnlineDocs.html, 2008.
27 1.5145e+6 [11] E.J. Friedmann-Hill, JESS Manual, http://www.jessrules.com/jess/docs/
28 6.1721e+6 71, 2008.
[12]G. Stertenbrink, suexg, http://www.setbb.com/phpb/viewtopic.php?t=206
29 4.8527e+7
&mforum=sudoku, 2005.
30 7.3090e+8
[13] B. Felgenhauer & F. Jarvis, Sudoku enumeration problems,
31 2.0623e+10 http://www.afjarvis.staff.shef.ac.uk/sudoku/, 2006.
32 7.6306e+11 [14] E. Russell & F. Jarvis, There are 5472730538 essentially different
Table 6: Inverse of the proportion of n-clue minimals among n-clue subgrids Sudoku grids, http://www.afjarvis.staff.shef.ac.uk/sudoku/sudgroup.html,
2006.