Lecture notes on descriptional complexity

and randomness

Peter Gács
Boston University
[email protected]

A didactical survey of the foundations of Algorithmic Information

Theory. These notes are short on motivation, history and background
but introduce some of the main techniques and concepts of the field.
The “manuscript” has been evolving over the years. Please, look at
“Version history” below to see what has changed when.
CONTENTS

Contents iii

1 Complexity 1
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Formal results . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.1.3 History of the problem . . . . . . . . . . . . . . . . . . . . 9
1.2 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.3 Kolmogorov complexity . . . . . . . . . . . . . . . . . . . . . . . . 12
1.3.1 Invariance . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.3.2 Simple quantitative estimates . . . . . . . . . . . . . . . . 15
1.4 Simple properties of information . . . . . . . . . . . . . . . . . . 17
1.5 Algorithmic properties of complexity . . . . . . . . . . . . . . . . 21
1.6 The coding theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 26
1.6.1 Self-delimiting complexity . . . . . . . . . . . . . . . . . . 26
1.6.2 Universal semimeasure . . . . . . . . . . . . . . . . . . . . 29
1.6.3 Prefix codes . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
1.6.4 The coding theorem for H(x) . . . . . . . . . . . . . . . . 32
1.6.5 Algorithmic probability . . . . . . . . . . . . . . . . . . . . 33
1.7 The statistics of description length . . . . . . . . . . . . . . . . . 34

2 Randomness 41
2.1 Uniform distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.2 Computable distributions . . . . . . . . . . . . . . . . . . . . . . . 44
2.2.1 Two kinds of test . . . . . . . . . . . . . . . . . . . . . . . . 44
2.2.2 Randomness via complexity . . . . . . . . . . . . . . . . . 45
2.2.3 Conservation of randomness . . . . . . . . . . . . . . . . 47
2.3 Infinite sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

iii
CONTENTS

2.3.1 Null sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

2.3.2 Probability space . . . . . . . . . . . . . . . . . . . . . . . . 57
2.3.3 Computability . . . . . . . . . . . . . . . . . . . . . . . . . 59
2.3.4 Integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
2.3.5 Randomness tests . . . . . . . . . . . . . . . . . . . . . . . 61
2.3.6 Randomness and complexity . . . . . . . . . . . . . . . . 62
2.3.7 Universal semimeasure, algorithmic probability . . . . . 63
2.3.8 Randomness via algorithmic probability . . . . . . . . . 65

3 Information 67
3.1 Information-theoretic relations . . . . . . . . . . . . . . . . . . . . 67
3.1.1 The information-theoretic identity . . . . . . . . . . . . . 67
3.1.2 Information non-increase . . . . . . . . . . . . . . . . . . 77
3.2 The complexity of decidable and enumerable sets . . . . . . . . 79
3.3 The complexity of complexity . . . . . . . . . . . . . . . . . . . . 83
3.3.1 Complexity is sometimes complex . . . . . . . . . . . . . 83
3.3.2 Complexity is rarely complex . . . . . . . . . . . . . . . . 84

4 Generalizations 87
4.1 Continuous spaces, noncomputable measures . . . . . . . . . . . 87
4.1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.1.2 Uniform tests . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.1.3 Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
4.1.4 Conservation of randomness . . . . . . . . . . . . . . . . 92
4.2 Test for a class of measures . . . . . . . . . . . . . . . . . . . . . . 94
4.2.1 From a uniform test . . . . . . . . . . . . . . . . . . . . . . 94
4.2.2 Typicality and class tests . . . . . . . . . . . . . . . . . . . 96
4.2.3 Martin-Löf’s approach . . . . . . . . . . . . . . . . . . . . 99
4.3 Neutral measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
4.4 Monotonicity, quasi-convexity/concavity . . . . . . . . . . . . . . 106
4.5 Algorithmic entropy . . . . . . . . . . . . . . . . . . . . . . . . . . 109
4.5.1 Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
4.5.2 Algorithmic entropy . . . . . . . . . . . . . . . . . . . . . . 110
4.5.3 Addition theorem . . . . . . . . . . . . . . . . . . . . . . . 112
4.5.4 Information . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
4.6 Randomness and complexity . . . . . . . . . . . . . . . . . . . . . 116
4.6.1 Discrete space . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.6.2 Non-discrete spaces . . . . . . . . . . . . . . . . . . . . . . 119

iv
 Contents

4.6.3 Infinite sequences . . . . . . . . . . . . . . . . . . . . . . . 120

4.6.4 Bernoulli tests . . . . . . . . . . . . . . . . . . . . . . . . . 121
4.7 Cells . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.7.1 Partitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
4.7.2 Computable probability spaces . . . . . . . . . . . . . . . 128

5 Exercises and problems 131

A Background from mathematics 137

A.1 Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
A.1.1 Topological spaces . . . . . . . . . . . . . . . . . . . . . . . 137
A.1.2 Continuity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
A.1.3 Semicontinuity . . . . . . . . . . . . . . . . . . . . . . . . . 140
A.1.4 Compactness . . . . . . . . . . . . . . . . . . . . . . . . . . 142
A.1.5 Metric spaces . . . . . . . . . . . . . . . . . . . . . . . . . . 142
A.2 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
A.2.1 Set algebras . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
A.2.2 Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
A.2.3 Integral . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
A.2.4 Density . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
A.2.5 Random transitions . . . . . . . . . . . . . . . . . . . . . . 154
A.2.6 Probability measures over a metric space . . . . . . . . . 155

B Constructivity 163
B.1 Computable topology . . . . . . . . . . . . . . . . . . . . . . . . . 163
B.1.1 Representations . . . . . . . . . . . . . . . . . . . . . . . . 163
B.1.2 Constructive topological space . . . . . . . . . . . . . . . 165
B.1.3 Computable functions . . . . . . . . . . . . . . . . . . . . 167
B.1.4 Effective compactness . . . . . . . . . . . . . . . . . . . . . 168
B.1.5 Computable elements and sequences . . . . . . . . . . . 169
B.1.6 Semicomputability . . . . . . . . . . . . . . . . . . . . . . . 170
B.1.7 Computable metric space . . . . . . . . . . . . . . . . . . 172
B.2 Constructive measure theory . . . . . . . . . . . . . . . . . . . . . 175
B.2.1 Space of measures . . . . . . . . . . . . . . . . . . . . . . . 176
B.2.2 Computable and semicomputable measures . . . . . . . 177
B.2.3 Random transitions . . . . . . . . . . . . . . . . . . . . . . 178

Bibliography 181

v
CONTENTS

VERSION HISTORY

February 2010: chapters introduced, and recast using the memoir class.
April 2009: besides various corrections, a section is added on infinite
sequences. This is not new material, just places the most classical results on
randomness of infinite sequences before the more abstract theory.
January 2008: major rewrite.
• Added formal definitions throughout.
• Made corrections in the part on uniform tests and generalized complexity,
based on remarks of Hoyrup, Rojas and Shen.
• Rearranged material.
• Incorporated material on uniform tests from the work of Hoyrup-Rojas.
• Added material on class tests.

vi
1 COMPLEXITY

Ainsi, au jeu de croix ou pile,

l’arrivée de croix cent fois de
suite, nous paraît extraordinaire;
parce que le nombre presque
infini des combinaisons qui
peuvent arriver en cent coups
étant partagé en séries
regulières, ou dans lesquelles
nous voyons régner un ordre
facile à saisir, et en séries
irregulières; celles-ci sont
incomparablement plus
nombreuses.

Laplace

1.1 INTRODUCTION
The present section can be read as an independent survey on the problems
of randomness. It serves as some motivation for the dryer stuff to follow.
If we toss a coin 100 times and it shows each time Head, we feel lifted to
a land of wonders, like Rosencrantz and Guildenstern in [48]. The argument
that 100 heads are just as probable as any other outcome convinces us only
that the axioms of Probability Theory, as developed in [27], do not solve all
mysteries they are sometimes supposed to. We feel that the sequence con-
sisting of 100 heads is not random, though others with the same probability
are. Due to the somewhat philosophical character of this paradox, its history
is marked by an amount of controversy unusual for ordinary mathematics.

1
1. COMPLEXITY

Before the reader hastes to propose a solution, let us consider a less trivial
example, due to L. A. Levin.
Suppose that in some country, the share of votes for the ruling party in
30 consecutive elections formed a sequence 0.99x i where for every even i,
the number x i is the i-th digit of π = 3.1415 . . .. Though many of us would
feel that the election results were manipulated, it turns out to be surprisingly
difficult to prove this by referring to some general principle.
In a sequence of n fair elections, every sequence ω of n digits has ap-
proximately the probability Q n (ω) = 10−n to appear as the actual sequence
of third digits. Let us fix n. We are given a particular sequence ω and want
to test the validity of the government’s claim that the elections were fair. We
interpret the assertion “ω is random with respect to Q n ” as a synonym for
“there is no reason to challenge the claim that ω arose from the distribution
Q n ”.
How can such a claim be challenged at all? The government, just like the
weather forecaster who announces 30% chance of rain, does not guarantee
any particular set of outcomes. However, to stand behind its claim, it must
agree to any bet based on the announced distribution. Let us call a payoff
function
P with respect the distribution P any nonnegative function t(ω) with
ω P(ω)t(ω) ¶ 1. If a “nonprofit” gambling casino asks 1 dollar for a game
and claims that each outcome has probability P(ω) then it must agree to
pay t(ω) dollars on outcome ω. We would propose to the government the
following payoff function t 0 with respect to Q n : let t 0 (ω) = 10n/2 for all
sequences ω whose even digits are given by π, and 0 otherwise. This bet
would cost the government 10n/2 − 1 dollars.
Unfortunately, we must propose the bet before the elections take place
and it is unlikely that we would have come up exactly with the payoff funct-
ion t 0 . Is then the suspicion unjustifiable?
No. Though the function t 0 is not as natural as to guess it in advance, it is
still highly “regular”. And already Laplace assumed in [15] that the number
of “regular” bets is so small we can afford to make them all in advance and
still win by a wide margin.
Kolmogorov discovered in [26] and [28] (probably without knowing
about [15] but following a four decades long controversy on von Mises’ con-
cept of randomness, see [51]) that to make this approach work we must
define “regular” or “simple” as “having a short description” (in some formal
sense to be specified below). There cannot be many objects having a short
description because there are not many short strings of symbols to be used

2
 1.1. Introduction

as descriptions.
We thus come to the principle saying that on a random outcome, all suf-
ficiently simple payoff functions must take small values. It turns out below
that this can be replaced by the more elegant principle saying that a random
outcome itself is not too simple. If descriptions are written in a 2-letter alpha-
bet then a typical sequence of n digits takes n log 10 letters to describe (if not
stated otherwise, all logarithms in these notes are to the base 2). The digits
of π can be generated by an algorithm whose description takes up only some
constant length. Therefore the sequence x 1 . . . x n above can be described
with approximately (n/2) log 10 letters, since every other digit comes from
π. It is thus significantly simpler than a typical sequence and can be declared
nonrandom.

1.1.1 FORMAL RESULTS

The theory of randomness is more impressive for infinite sequences than

for finite ones, since sharp distinction can be made between random and
nonrandom infinite sequences. For technical simplicity, first we will confine
ourselves to finite sequences, especially a discrete sample space Ω, which we
identify with the set of natural numbers. Binary strings will be identified
with the natural numbers they denote.
Definition 1.1.1. A Turing machine is an imaginary computing device con-
sisting of the following. A control state belonging to a finite set A of possible
control states. A fixed number of infinite (or infinitely extendable) strings of
cells called tapes. Each cell contains a symbol belonging to a finite tape al-
phabet B. On each tape, there is a read-write head observing one of the tape
cells. The machine’s configuration (global state) is determined at each instant
by giving its control state, the contents of the tapes and the positions of the
read-write heads. The “hardware program” of the machine determines its
configuration in the next step as a functon of the control state and the tape
symbols observed. It can change the control state, the content of the ob-
served cells and the position of the read-write heads (the latter by one step
to the left or right). Turing machines can emulate computers of any known
design (see for example [58]). The tapes are used for storing programs,
input data, output and as memory. ù
The Turing machines that we will use to interpret descriptions will be
somewhat restricted.

3
1. COMPLEXITY

Definition 1.1.2. Consider a Turing machine F which from a binary string p

and a natural number x computes the output F (p, x) (if anything at all). We
will say that F interprets p as a description of F (p, x) in the presence of the
side information x. We suppose that if F (p, x) is defined then F (q, x) is not
defined for any prefix q of p. Such machines are called self-delimiting (s.d.).
The conditional complexity H F (x | y) of the number x with respect to
the number y is the length of the shortest description p for which F (p, y) =
x. ù

Kolmogorov and Solomonoff observed that the function H F (x | y) de-

pends only weakly on the machine F , because there are universal Turing
machines capable of simulating the work of any other Turing machine whose
description is supplied. More formally, the following theorem holds:

Theorem 1.1.1 (Invariance Theorem). There is a s.d. Turnig machine T such

that for any s.d. machine F a constant c F exists such that for all x, y we have
H T (x | y) ¶ H F (x | y) + c F .

This theorem motivates the following definition.

Definition 1.1.3. Let us fix T and define H(x | y) = H T (x | y) and H(x) =

H(x | 0). ù

The function H(x | y) is not computable. We can compute a nonincreas-

ing, convergent sequence of approximations to H(x) (it is semicomputable
from above), but will not know how far to go in this sequence for some
prescribed accuracy.
If x is a binary string of length n then H(x) ¶ n + 2 log n + c for some
constant c. The description of x with this length gives x bit-for-bit, along
with some information of length 2 log n which helps the s.d. machine find the
end of the description. For most binary strings of lenght n, no significantly
shorter description exists, since the number of short descriptions is small.
Below, this example is generalized and sharpened for the case when instead
of counting the number of simple sequences, we measure their probability.

Definition 1.1.4. Denote by x ∗ the first one among the shortest descriptions
of x. ù

The correspondence x → x ∗ is a code in which no codeword is the prefix

of another one. This implies by an argument well-known in Information

4
 1.1. Introduction

Theory the inequality X

2−H(x| y) ¶ 1,
x

hence only a few objects x can have small complexity. The converse of the
same argument goes as follows. Let µ be a computable probability distribu-
tion, one for which there is a binary program computing µ(ω) for each ω to
any given degree of accuracy. Let H(µ) be the length of the shortest one of
these programs. Then

H(ω) ¶ − log µ(ω) + H(µ) + c.

Here c is a universal constant. These two inequalities are the key to the esti-
mates of complexity and the characterization of randomness by complexity.
Denote
dµ (ω) = − log µ(ω) − H(ω).
Inequality (1.1.1) implies
t µ (ω) = 2dµ (ω)
can be viewed as a payoff function. Now we are in a position to solve the
election paradox. We propose the payoff function

2− log Q n (ω)−H(ω|n)

to the government. (We used the conditional complexity H(ω | n) because

the uniform distribution Q n depends on n.) If every other digit of the out-
come x comes from π then H(x | n) ¶ (n/2) log 10 + c0 hence we win a sum
2 t(x) ¾ c1 10n/2 from the government (for some constants c0 , c1 > 0), even
though the bet does not contain any reference to the number π.
The fact that t µ (ω) is a payoff function implies by Markov’s Inequality for
any k > 0
µ{ ω : H(ω) < − log µ(ω) − k } < 2−k . (1.1.1)
Inequalities (1.1.1) and (1.1.1) say that with large probability, the complex-
ity H(ω) of a random outcome ω is close to its upper bound − log µ(ω) +
H(µ). This law occupies distinguished place among the “laws of probability”,
because if the outcome ω violates any such law, the complexity falls far be-
low the upper bound. Indeed, a proof of some “law of probability” (like the
law of large numbers, the law of iterated logarithm, etc.) always gives rise to
some simple computable payoff function t(ω) taking large values on the out-
comes violating the law, just as in the election example. Let m be some large

5
1. COMPLEXITY

number, suppose that t(ω) has complexity < m/2, and that t(ω0 ) > 2m .
Then inequality (1.1.1) can be applied to ν(ω) = µ(ω)t(ω), and we get

H(ω) ¶ − log µ(ω) − m + H(ν) + c0

¶ − log µ(ω) − m/2 + H(µ) + c1

for some constants c0 , c1 .

More generally, the payoff function t µ (ω) is maximal (up to a multiplica-
tive constant) among all payoff functions that are semicomputable (from
below). Hence the quantity − log µ(ω) − H(ω) is a universal test of random-
ness. Its value measures the deficiency of randomness in the outcome ω with
respect to the distribution µ, or the extent of justified suspicion against the
hypothesis µ given the outcome ω.

1.1.2 APPLICATIONS
Algorithmic Information Theory (AIT) justifies the intuition of random se-
quences as nonstandard analysis justifies infinitely small quantities. Any
statement of classical probability theory is provable without the notion of
randomness, but some of them are easier to find using this notion. Due to
the incomputability of the universal randomness test, only its approximations
can be used in practice.
Pseudorandom sequences are sequences generated by some algorithm,
with some randomness properties with respect to the coin-tossing distribu-
tion. They have very low complexity (depending on the strength of the tests
they withstand, see for example [14]), hence are not random. Useful pseu-
dorandom sequences can be defined using the notion of computational com-
plexity, for example the number of steps needed by a Turing machine to
compute a certain function. The existence of such sequences can be proved
using some difficult unproven (but plausible) assumptions of computation
theory. See [6], [57], [24], [33].

INDUCTIVE INFERENCE

The incomputable “distribution” m(ω) = 2−H(ω) has the remarkable property

that, the test d(ω | m), shows all outcomes ω “random” with respect to it.
Relations (1.1.1) and (1.1.1) can be read as saying that if the real distribu-
tion is µ then µ(ω) and m(ω) are close to each other with large probability.

6
 1.1. Introduction

Therefore if we know that ω comes from some unknown simple distribution

µ then we can use m(ω) as an estimate of µ(ω). This suggests to call m
the “apriori probability” (but we will not use this term much). The random-
ness test dµ (ω) can be interpreted in the framework of hypothesis testing:
it is the likelihood ratio between the hypothesis µ and the fixed alternative
hypothesis m.
In ordinary statistical hypothesis testing, some properties of the unknown
distribution µ are taken for granted. The sample ω is generally a large in-
dependent sample: µ is supposed to be a product distribution. Under these
conditions, the universal test could probably be reduced to some of the tests
used in statistical practice. However, these conditions do not hold in other
applications: for example testing for independence in a proposed random
sequence, predicting some time series of economics, or pattern recognition.
If the apriori probability m is a good estimate of the actual probability
then we can use the conditional apriori probability for prediction, without
reference to the unknown distribution µ. For this purpose, one first has to
define apriori probability for the set of infinite sequences of natural numbers,
as done in [59]. We denote this function by M . For any finite sequence x, the
number M (x) is the apriori probability that the outcome is some extension
of x. Let x, y be finite sequences. The formula

M (x y)
M (x)

is an estimate of the conditional probability that the next terms of the out-
come will be given by y provided that the first terms are given by x. It con-
verges to the actual conditional probability µ(x y)/µ(x) with µ-probability 1
for any computable distribution µ (see for example [45]). To understand the
surprising generality of the formula (1.1.2), suppose that some infinite se-
quence z(i) is given in the following way. Its even terms are the subsequent
digits of π, its odd terms are uniformly distributed, independently drawn
random digits. Let
z(1 : n) = z(1) · · · z(n).
Then M (z(1 : 2i)a/M (z(1 : 2i)) converges to 0.1 for a = 0, . . . , 9, while
M (z(1 : 2i + 1)a)/M (z(1 : 2i + 1)) converges to 1 if a is the i-th digit of π,
and to 0 otherwise.
The inductive inference formula using conditional apriori probability can
be viewed as a mathematical form of “Occam’s Razor”: the advice to predict

7
1. COMPLEXITY

by the simplest rule fitting the data. It can also viewed as a realization of
Bayes’Rule, with a universally applicable apriori distribution. Since the dis-
tribution M is incomputable, we view the main open problem of inductive
inference to find maximally efficient approximations to it. Sometimes, even
a simple approximation gives nontrivial results (see [3]).

INFORMATION THEORY

P with large probability, H(ω) is close to − log µ(ω), the entropy

Since
P ω µ(ω) log µ(ω) of the distribution µ is close to the average complexity
−
ω µ(ω)H(ω). The complexity H(x) of an object x can indeed be inter-
preted as the distribution-free definition of information content. The corre-
spondence x 7→ x ∗ is a sort of universal code: its average (even individual)
“rate”, or codeword length is almost equally good for any simple computable
distribution.
It is of great conceptual help to students of statistical physics that en-
tropy can be defined now not only for ensembles (probability distributions),
but for individual states of a physical system. The notion of an individual
incompressible sequence, a sequence whose complexity is maximal, proved
also extremely useful in finding information-theoretic lower bounds on the
computing speed of certain Turing machines (see [38]).
The conditional complexity H(x | y) obeys identities analogous to the
information-theoretical identities for conditional entropy, but these identi-
ties are less trivial to prove in AIT. The information I(x : y) = H(x) + H( y) −
H(x, y) has several interpretations. It is known that I(x : y) is equal, to
within an additive constant, to H( y) − H( y | x ∗ ), the amount by which the
object x ∗ (as defined in the previous section) decreases our uncertainty about
y. But it can be written as − log m2 (x, y) − H(x, y) = d((x, y) | m2 ) where
m2 = m × m is the product distribution of m with itself. It is thus the defi-
ciency of randomness of the pair (x, y) with respect to this product distribu-
tion. Since any object is random with respect to m, we can view the random-
ness of the pair (x, y) with respect to the product m2 as the independence
of x and y from each other. Thus “information” measures the “deficiency of
independence”.

8
 1.1. Introduction

LOGIC

Some theorems of Mathematical Logic (in particular, Gödel’s theorem) have a

strong quantitative form in AIT, with new philosophical implications (see [8],
[9], [30], [32]). Levin based a new system of intuitionistic analysis on his
Independence Principle (see below) in [32].

1.1.3 HISTORY OF THE PROBLEM

P. S. Laplace thought that the number of “regular” sequences (whatever “reg-

ular” means) is much smaller than the number of irregular ones (see [15]).
In the first attempt at formalization hundred years later, R. von Mises de-
fined an infinite binary sequence as random (a “Kollektiv”) if the relative
frequencies converge in any subsequence selected according to some (non-
anticipating) “rule” (whatever “rule” means, see [51]). As pointed out by
A. Wald and others, Mises’s definitions are sound only if a countable set of
possible rules is fixed. The logician A. Church, in accordance with his fa-
mous thesis, proposed to understand “rule” here as “recursive (computable)
function”.
The Mises selection rules can be considered as special randomness tests.
In the ingenious work [50], J. Ville proved that they do not capture all rele-
vant properties of random sequences. In particular, a Kollektiv can violate the
law of iterated logarithm. He proposed to consider arbitrary payoff functions
(a countable set of them), as defined on the set of infinite sequences—these
are more commonly known as martingales.
For the solution of the problem of inductive inference, R. Solomonoff
introduced complexity and apriori probability in [44] and proved the Invari-
ance Theorem. A. N. Kolmogorov independently introduced complexity as a
measure of individual information content and randomness, and proved the
Invariance Theorem (see [26] and [28]). The incomputability properties of
complexity have noteworthy philosophical implications (see [4], [8], [9].
P. Martin-Löf defined in [37] randomness for infinite sequences. His con-
cept is essentially the synthesis of Ville and Church (as noticed in [40]). He
recognized the existence of a universal test, and pointed out the close con-
nection between the randomness of an infinite sequence and the complexity
of its initial segments.
L. A. Levin defined the apriori probability M as a maximal (to within a
multiplicative constant) semicomputable measure. With the help of a modi-

9
1. COMPLEXITY

fied complexity definition, he gave a simple and general characterization of

random sequences by the behavior of the complexity of their initial segments
(see [59], [29]). In [17] and [30], the information-theoretical properties of
the self-delimiting complexity (as defined above) are exactly described. See
also [10], [41] and [56].
In [32], Levin defined a deficiency of randomness d(ω | µ) in a uniform
manner for all (computable or incomputable) measures µ. He proved that
all outcomes are random with respect to the apriori probability M . In this
and earlier papers, he also proved the Law of Information Conservation, stat-
ing that the information I(α; β) in a sequence α about a sequence β cannot
be significantly increased by any algorithmic processing of α (even using
random number generators). He derived this law from a so-called Law of
Randomness Conservation via the definition of information I(α; β) as de-
ficiency of randomness with respect to the product distribution M 2 . Levin
suggested the Independence Principle saying that any sequence α arising in
nature contains only finite information I(α; β) about any sequence β defined
by mathematical means. With this principle, he showed that the use of more
powerful notions of definability in randomness tests (or the notion of com-
plexity) does not lead to fewer random sequences among those arising in
nature.
The monograph [16] is very useful as background information on the
various attempts in this century at solving the paradoxes of probability the-
ory. The work [32] is quite comprehensive but very dense; it can be rec-
ommended only to devoted readers. The work [59] is comprehensive and
readable but not quite up-to-date. The surveys [12], [40] and [42] can be
used to complement it.
The most up-to-date and complete survey, which subsumes most of these
notes, is [35].
AIT created many interesting problems of its own. See for example [10],
[11], [17], [18], [19], [20], [34], [32], [36], [40], [43], [46], and the
technically difficult results [47] and [53].

ACKNOWLEDGMENT
The form of exposition of the results in these notes and the general point
of view represented in them were greatly influenced by Leonid Levin. More
recent communication with Paul Vitányi, Mathieu Hoyrup, Cristóbal Rojas
and Alexander Shen has also been very important.

10
 1.2. Notation

1.2 NOTATION
When not stated otherwise, log means base 2 logarithm. The cardinality
of a set A will be denoted by |A|. (Occasionally there will be inconsistency,
sometimes denoting it by |A|, sometimes by #A.) If A is a set then 1A(x) is its
indicator function:
(
1 if x ∈ A,
1A(x) =
0 otherwise.

The empty string is denoted by Λ. The set A∗ is the set of all finite strings
of elements of A, including the empty string. Let l(x) denote the length of
string x. (Occasionally there will be inconsistency, sometimes denoting it by
|x|, sometimes by l(x)). For sequences x and y, let x v y denote that x is
a prefix of y. For a string x and a (finite or infinite) sequence y, we denote
their concatenation by x y. For a sequence x and n ¶ l(x), the n-th element
of x is x(n), and
x(i : j) = x(i) · · · x( j).
Sometimes we will also write

x ¶n = x(1 : n).

The string x 1 · · · x n will sometimes be written also as (x 1 , . . . , x n ). For na-

tural number m let β(m) be the binary sequence denoting m in the binary
notation. We denote by X N the set of all infinite sequences of elements of X .
The sets of natural numbers, integers, rational numbers, real numbers
and complex numbers will be denoted respectively by N, Z, Q, R, C. The set
of nonnegative real numbers will be denoted by R+ . The set of real numbers
with −∞, ∞ added (with the appropriate topology making it compact) will
be denoted by R. Let

S r = {0, . . . , r − 1}∗ , S = N∗ , B = {0, 1}.

We use ∧ and ∨ to denote min and max, further

|x|+ = x ∨ 0, |x|− = | − x|+

for real numbers x.

11
1. COMPLEXITY

Let 〈·〉 be some standard one-to-one encoding of N∗ to N, with partial

inverses [·]i where [〈x〉]i = x(i) for i ¶ l(x). For example, we can have
1
〈i, j〉 = (i + 1)(i + j + 1) + j, 〈n1 , . . . , nk+1 〉 = 〈〈n1 , . . . , nk 〉, nk+1 〉.
2
We will use 〈·, ·〉 in particular as a pairing function over N. Similar pairing
functions will be assumed over other domains, and they will also be denoted
by 〈·, ·〉.
Another use of the notation (·, ·) may arise when the usual notation (x, y)
of an ordered pair of real numbers could be confused with the same notation
of an open interval. In this case, we will use (x, y) to denote the pair.
The relations
+ ∗
f < g, f < g
mean inequality to within an additive constant and multiplicative constant
respectively. The first is equivalent to f ¶ g + O(1), the second to f = O(g).
∗ ∗
The relation f = g means f < g and f > g.
∗

1.3 KOLMOGOROV COMPLEXITY

1.3.1 INVARIANCE
It is natural to try to define the information content of some text as the size
of the smallest string (code) from which it can be reproduced by some de-
coder, interpreter. We do not want too much information “to be hidden in the
decoder” we want it to be a “simple” function. Therefore, unless stated other-
wise, we require that our interpreters be computable, that is partial recursive
functions.

Definition 1.3.1. A partial recursive function A from S2 ×S to S will be called

a (binary) interpreter, or decoder. We will often refer to its first argument as
the program, or code. ù

Partial recursive functions are relatively simple; on the other hand, the
class of partial recursive functions has some convenient closure properties.

Definition 1.3.2. For any binary interpreter A and strings x, y ∈ S, the num-
ber
KA(x | y) = min{ l(p) : A(p, y) = x }

12
 1.3. Kolmogorov complexity

is called the conditional Kolmogorov-complexity of x given y, with respect to

the interpreter A . (If the set after the “min” is empty, then the minimum is
∞). We write KA(x) = KA(x | Λ). ù

The number KA(x) measures the length of the shortest description for x
when the algorithm A is used to interpret the descriptions.
The value KA(x) depends, of course, on the underlying function A. But, as
Theorem 1.3.1 shows, if we restrict ourselves to sufficiently powerful inter-
preters A, then switching between them can change the complexity function
only by amounts bounded by some additive constant. Therefore complexity
can be considered an intrinsic characteristic of finite objects.

Definition 1.3.3. A binary p.r. interpreter U is called optimal if for any binary
p.r. interpreter A there is a constant c < ∞ such that for all x, y we have

KU (x | y) ¶ KA(x | y) + c. (1.3.1)

Theorem 1.3.1 (Invariance Theorem). There is an optimal p.r. binary inter-

preter.

Proof. The idea is to use interpreters that come from universal partial re-
cursive functions. However, not all such functions can be used for universal
interpreters. Let us introduce an appropriate pairing function. For x ∈ Bn ,
let
x o = x(1)0x(2)0 . . . x(n − 1)0x(n)1
where x o = x1 for l(x) = 1. Any binary sequence x can be uniquely repre-
sented in the form p = a o b.
We know there is a p.r. function V : S2 × S2 × S → S that is universal: for
any p.r. binary interpreter A, there is a string a such that for all p, x, we have
A(p, x) = V (a, p, x). Let us define the function U(p, x) as follows. We repre-
sent the string p in the form p = uo v and define U(p, x) = V (u, v, x). Then
U is a p.r. interpreter. Let us verify that it is optimal. Let A be a p.r. binary
interpreter, a a binary string such that A(p, x) = U(a, p, x) for all p, x. Let
x, y be two strings. If KA(x | y) = ∞, then (1.3.1) holds trivially. Otherwise,
let p be a binary string of length KA(x | y) with A(p, y) = x. Then we have

U(a o p, y) = V (a, p, y) = A(p, y) = x,

13
1. COMPLEXITY

and
KU (x | y) ¶ 2l(a) + KA(x | y).

The constant 2l(a) is in the above proof a bound on the complexity of

description of the interpreter A for the optimal interpreter U. Let us note
that for any two optimal interpreters U (1) , U (2) , there is a constant c such
that for all x, y, we have

|KU (1) (x | y) − KU (2) (x | y)| < c. (1.3.2)

Hence the complexity KU (x) of description of an object x does not depend

strongly on the interpreter U. Still, for every string x, there is an optimal
interpreter U with KU (x) = 0. Imposing a universal bound on the table size
of the Turing machines used to implement optimal interpreters, we can get a
universal bound on the constants in (1.3.2).
The theorem motivates the following definition.

Definition 1.3.4. We fix an optimal binary p.r. interpreter U and write K(x |
y) for KU (x | y). ù

Theorem 1.3.1 (as well as other invariance theorems) is used in AIT for
much more than just to show that KA(x) is a proper concept. It is the principal
tool to find upper bounds on K(x); this is why most such upper bounds are
proved to hold only to within an additive constant.
The optimal interpreter U(p, x) defined in the proof of Theorem 1.3.1 is
obviously a universal partial recursive function. Because of its convenient
properties we will use it from now on as our standard universal p.r. function,
and we will refer to an arbitrary p.r. function as U p (x) = U(p, x).

Definition 1.3.5. We define U p (x 1 , . . . , x k ) = U p (〈x 1 . . . , x k 〉). Similarly, we

will refer to an arbitrary computably enumerable set as the range Wp of some
U p . We often do not need the second argument of the function U(p, x). We
therefore define
U(p) = U(p, Λ).

14
 1.3. Kolmogorov complexity

It is of no consequence that we chose binary strings as as descriptions. It is

rather easy to define, (Exercise) for any two natural numbers r, s, a standard
encoding cnvsr of base r strings x into base s strings with the property

log r
l(cnvsr (x)) ¶ l(x) + 1.
log s

Now, with r-ary strings as descriptions, we must define KA(x) as the mini-
mum of l(p) log r over all programs p in S r with A(p) = x. The equivalence
of the definitions for different values of r is left as an exercise.

1.3.2 SIMPLE QUANTITATIVE ESTIMATES

We found it meaningful to speak about the information content, complexity

of a finite object. But how to compute this quantity? It turns out that com-
plexity is not a computable function, but much is known about its statistical
behavior.
The following notation will be useful in the future.
+
Definition 1.3.6. The relation f < g means inequality to within an additive
constant, that there is a constant c such that for all x, f (x) ¶ g(x). We can
write this also as f ¶ g + O(1). We also say that g additively dominates f .
+ + + ∗
The relation f = g means f < g and f > g. The relation f < g among non-
negative functions means inequality to within a multiplicative constant. It is
+
equivalent to log f < log g and f = O(g). We also say that g multiplicatively
∗ ∗
dominates f . The relation f = g means f < g and f > g.
∗
ù

With the above notation, here are the simple properties.

Theorem 1.3.2. The following simple upper bound holds.

(a) For any natural number m, we have
+
K(m) < log m. (1.3.3)

(b) For any positive real number v and string y, every finite set E of size m has
at least m(1 − 2−v+1 ) elements x with K(x | y) ¾ log m − v.

Corollary 1.3.1. limn→∞ K(n) = ∞.

15
1. COMPLEXITY

Theorem 1.3.2 suggests that if there are so few strings of low complexity,
then K(n) converges fast to ∞. In fact this convergence is extremely slow, as
shown in a later section.

Proof of Theorem 1.3.2. First we prove (a). Let the interpreter A be defined
such that if β(m) = p then A(p, y) = m. Since |β(m)| = dlog me, we have
+
K(m) < KA(m) < log m + 1.
Part (b) says that the trivial estimate (1.3.3) is quite sharp for most num-
bers under m. The reason for this is that since there are only few short
programs, there are only few objects of low complexity. For any string y, and
any positive real natural number u, we have

|{ x : K(x | y) ¶ u }| < 2u+1 . (1.3.4)

To see this, let n = blog uc. The number 2n+1 − 1 of different binary strings of
length ¶ n is an upper bound on the number of different shortest programs
of length ¶ u. Now (b) follows immediately from (1.3.4).

The three properties of complexity contained in Theorems 1.3.1 and 1.3.2

are the ones responsible for the applicabiliy of the complexity concept. Sev-
eral important later results can be considered transformations, generaliza-
tions or analogons of these.
Let us elaborate on the upper bound (1.3.3). For any string x ∈ S r , we
have
+
K(x) < n log r + 2 log r. (1.3.5)
In particular, for any binary string x, we have
+
K(x) < l(x).

Indeed, let the interpreter A be defined such that if p = β(r)o cnv2r (x) then
A(p, y) = x. We have KA(x) ¶ 2l(β(r)) + n log r + 1 ¶ (n + 2) log r + 3. Since
+
K(x) < KA(x), we are done.
We will have many more upper bound proofs of this form, and will not
+
always explicitly write out the reference to K(x | y) < KA(x | y), needed for
the last step.
Apart from the conversion, inequality (1.3.5) says that since the begin-
ning of a program can command the optimal interpreter to copy the rest, the
complexity of a binary sequence x of length n is not larger than n.

16
 1.4. Simple properties of information

Inequality (1.3.5) contains the term 2 log r instead of log r since we have
to apply the encoding w o to the string w = β(r); otherwise the interpreter
cannot detach it from the program p. We could use the code β(|w|)o w, which
is also a prefix code, since the first, detachable part of the codeword tells its
length.
For binary strings x, natural numbers n and real numbers u ¾ 1 we define
J(u) = u + 2 log u,
ι(x) = β(|x|)o x,
ι(n) = ι(β(n)).
+ +
We have l(ι(x)) < J(l(x)) for binary sequences x, and l(ι(r)) < J(log r)
for numbers r. Therefore (1.3.5) is true with J(log r) in place of 2 log r.
Of course, J(x) could be replaced by still smaller functions, for example
x + J(log x). We return to this topic later.

1.4 SIMPLE PROPERTIES OF INFORMATION

If we transform a string x by applying to it a p.r. function, then we cannot

gain information over the amount contained in x plus the amount needed to
describe the transformation. The string x becomes easier to describe, but it
helps less in describing other strings y.
Theorem 1.4.1. For any partial recursive function Uq , over strings, we have
+
K(Uq (x) | y) < K(x | y) + J(l(q)),
+
K( y | Uq (x)) > K( y | x) − J(l(q)).
+
Proof. To define an interpreter A with KA(Uq (x) | y) < K(x | y) + J(l(q)),
we define A(ι(q)p, y) = Uq (U(p, y)). To define an interpreter B with KB ( y |
+
x) < K( y | Uq (x)) + J(q), we define B(ι(q)p, x) = U(p, Uq (x)).
The following notation is natural.
Definition 1.4.1. The definition of conditional complexity is extended to
pairs, etc. by
K(x 1 , . . . , x m | y1 , . . . , yn ) = K(〈x 1 , . . . , x m 〉 | 〈 y1 , . . . , yn 〉).
ù

17
1. COMPLEXITY

With the new notation, here are some new results.

Corollary 1.4.1. For any one-to-one p.r. function U p , we have
+
|K(x) − K(U p (x))| < J(l(p)).

Further,
+
K(x | y, z) < K(x | U p ( y), z) + J(l(p)),
+
K(U p (x)) < K(x) + J(l(p)),
+
K(x | z) < K(x, y | z), (1.4.1)
+
K(x | y, z) < K(x | y),
+
K(x, x) = K(x), (1.4.2)
+
K(x, y | z) = K( y, x | z),
+
K(x | y, z) = K(x | z, y),
+
K(x, y | x, z) = K( y | x, z),
+ +
K(x | x, z) = K(x | x) = 0.

We made several times implicit use of a basic additivity property of com-

plexity which makes it possible to estimate the joint complexity of a pair by
the complexities of its constituents. As expressed in Theorem 1.4.2, it says
essentially that to describe the pair of strings, it is enough to know the de-
scription of the first member and of a method to find the second member
using our knowledge of the first one.
Theorem 1.4.2.
+
K(x, y) < J(K(x)) + K( y | x).
Proof. We define an interpreter A as follows. We decompose any binary string
p into the form p = ι(w)q, and let A(p, z) = U(q, U(w)). Then KA(x, y) is
bounded, to within an additive constant, by the right-hand side of the above
inequality.
Corollary 1.4.2. For any p.r. function U p over pairs of strings, we have
+
K(U p (x, y)) < J(K(x)) + K( y | x) + J(l(p))
+
< J(K(x)) + K( y) + J(l(p)),

18
 1.4. Simple properties of information

and in particular,
+
K(x, y) < J(K(x)) + K( y).
This corollary implies the following continuity property of the function
K(n): for any natural numbers n, h, we have
+
|K(n + h) − K(n)| < J(log h). (1.4.3)
Indeed, n + h is a recursive function of n and h. The term 2 log K(x) making
up the difference between K(x) and J(K(x)) in Theorem 1.4.2 is attributable
to the fact that minimal descriptions cannot be concatenated without loosing
an “endmarker”. It cannot be eliminated, since there is a constant c such that
for all n, there are binary strings x, y of length ¶ n with
K(x) + K( y) + log n < K(x, y) + c.
Indeed, there are n2n pairs of binary strings whose sum of lengths is n. Hence
by Theorem 1.3.2 b, there will be a pair (x, y) of binary strings whose sum of
lengths is n, with K(x, y) > n + log n − 1. For these strings, inequality (1.3.5)
+ +
implies K(x) + K( y) < l(x) + l( y). Hence K(x) + K( y) + log n < n + log n <
K(x, y) + 1.
Regularities in a string will, in general, decrease its complexity radically.
If the whole string x of length n is given by some rule, that is we have
x(k) = U p (k), for some recursive function U p , then
+
K(x) < K(n) + J(l(p)).
Indeed, let us define the p.r. function V (q, k) = Uq (1) . . . Uq (k). Then x =
V (p, n) and the above estimate follows from Corollary 1.4.1.
For another example, let x = y1 y1 y2 y2 · · · yn yn , and y = y1 y2 · · · yn . Then
+
K(x) = K( y) even though the string x is twice longer. This follows from
Corollary 1.4.1 since x and y can be obtained from each other by a simple
recursive operation.
Not all “regularities” decrease the complexity of a string, only those which
distinguish it from the mass of all strings, putting it into some class which
is both small and algorithmically definable. For a binary string of length n,
p
it is nothing unusual to have its number of 0-s between n/2 − n and n/2.
Therefore such strings can have maximal or almost maximal complexity. If x
has k zeroes then the inequality
n
 
+
K(x) < log + J(log n) + J(log k)
k

19
1. COMPLEXITY

follows from Theorems 1.3.2 and 1.4.3.

Theorem 1.4.3. Let E = Wp be an enumerable set of pairs of strings defined

enumerated with the help of the program p for example as

Wp = { U(p, x) : x ∈ N }. (1.4.4)

We define the section

E a = { x : 〈a, x〉 ∈ E }. (1.4.5)
Then for all a and x ∈ E a , we have
+
K(x | a) < log |E a | + J(l(p)).

Proof. We define an interpreter A(q, b) as follows. We decompose q into

q = ι(p)β(t). From a standard enumeration (a(k), x(k)) (k = 1, 2, . . .) of the
set Wp , we produce an enumeration x(k, b) (k = 1, 2, . . .), without repetition,
of the set Wpb for each b, and define A(q, b) = x(t, b). For this interpreter A,
+
we get kA(x | a) < J(l(p)) + log |E a |.

It follows from Theorem 1.3.2 that whenever the set E a is finite, the esti-
mate of Theorem 1.4.3 is sharp for most of its elements x.
The shortest descriptions of a string x seem to carry some extra informa-
tion in them, above the description of x. This is suggested by the following
strange identities.
+
Theorem 1.4.4. We have K(x, K(x)) = K(x).
+ +
Proof. The inequality > follows from (1.4.1). To prove <, let A(p) =
〈U(p, l(p))〉. Then KA(x, K(x)) ¶ K(x).

More generally, we have the following inequality.

Theorem 1.4.5.
+
K( y | x, i − K( y | x, i)) < K( y | x, i).

The proof is exercise.

The above inequalities are in some sense “pathological”, and do not nec-
essarily hold for all “reasonable” definitions of descriptional complexity.

20
 1.5. Algorithmic properties of complexity

1.5 ALGORITHMIC PROPERTIES OF COMPLEXITY

The function K(x) is not computable, as it will be shown in this section.

However, it has a property closely related to computability. Let Q be the set
of rational numbers.

Definition 1.5.1 (Computability). Let f : S → R be a function. It is com-

putable if there is a recursive function g(x, n) with rational values, and
| f (x) − g(x, n)| < 1/n. ù

Definition 1.5.2 (Semicomputability). A function f : S → (−∞, ∞] is

(lower) semicomputable if the set

{ (x, r) : x ∈ S, r ∈ Q, r < f (x) }

is recursively enumerable. A function f is called upper semicomputable if − f

is lower semicomputable. ù

It is easy to show the following:

• A function f : S → R is lower semicomputable iff there exists a recursive
function with rational values, (or, equivalently, a computable real function)
g(x, n) nondecreasing in n, with f (x) = limn→∞ g(x, n).
• A function f : S → R is computable if it is both lower and upper semicom-
putable.
The notion of semicomputibility is naturally extendable over other dis-
crete domains, like S × S. It is also useful to extend it to functions R → R.
Let I denote the set of open rational intervals of R, that is

β = { (p, q) : p, q ∈ Q, p < q }.

Definition 1.5.3 (Computability for real functions). Let f : R → R be a

function. It is computable if there is a recursively enumerable set F ⊆ β 2
such that denoting FJ = { I : (I, J) ∈ F } we have
[
f −1 (J) = I.
I∈FJ

21
1. COMPLEXITY

This says that if f (x) ∈ I then sooner or later we will find an interval
I ∈ FJ with the property that f (z) ∈ J for all z ∈ I. Note that computability
implies continuity.

Definition 1.5.4. We say that f : R → R is lower semicomputable if there is a

recursively enumerable set G ⊆ Q × β such that denoting Gq = { I : (I, q) ∈
G } we have
[
f −1 ((q, ∞)) = I.
I∈FJ

This says that if f (x) > r then sooner or later we will find an interval
I ∈ FJ with the property that f (z) > q for all z ∈ I.
The following facts are useful and simple to verify:

Proposition 1.5.1.
(a) The function d·e : R → R is lower semicomputable.
(b) If f , g : R → R are computable then their composition f (g(·)) is, too.
(c) If f : R → R is lower semicomputable and g : R → R (or S → R) is
computable then f (g(·)) is lower semicomputable.
(d) If f : R → R is lower semicomputable and monotonic, and g : R → R (or
S → R) is lower semicomputable then f (g(·)) is lower semicomputable.

Definition 1.5.5. We introduce a universal lower semicomputable function.

For every binary string p, and x ∈ S, let

S p (x) = sup{ y/z : y, z ∈ Z, z 6= 0, 〈〈x〉, y, z〉 ∈ Wp }

where Wp was defined in (1.4.4). ù

The function S p (x) is lower semicomputable as a function of the pair

(p, x), and for different values of p, it enumerates all semicomputable func-
tions of x.

Definition 1.5.6. Let S p (x 1 , . . . , x k ) = S p (〈x 1 , . . . , x k 〉). For any lower semi-

computable function f , we call any binary string p with S p = f a Gödel
number of f . There is no universal computable function, but for any com-
putable function g, we call any number 〈〈p〉, 〈q〉〉 a Gödel number of g if
g = S p = −Sq . ù

22
 1.5. Algorithmic properties of complexity

With this notion, we claim the following.

Theorem 1.5.1. The function K(x | y) is upper semicomputable.

Proof. By Theorem 1.3.2 there is a constant c such that for any strings x, y,
we have K(x | y) < log〈x〉 + c. Let some Turing machine compute our
optimal binary p.r. interpreter. Let U t (p, y) be defined as U(p, y) if this
machine, when started on input (p, y), gives an output in t steps, undefined
otherwise. Let K t (x | y) be the smaller of log〈x〉 + c and

min{ l(p) ¶ t : U t (p, y) = x }.

Then the function K t (x | y) is computable, monotonic in t and lim t→∞ K t (x |

y) = K(x | y).

The function K(n) is not computable. Moreover, it has no nontrivial par-

tial recursive lower bound.

Theorem 1.5.2. Let U p (n) be a partial recursive function whose values are
numbers smaller than K(n) whenever defined. Then
+
U p (n) < J(l(p)).

Proof. The proof of this theorem resembles “Berry’s paradox ”, which says:
“The least number that cannot be defined in less than 100 words” is a def-
inition of that number in 12 words. The paradox can be used to prove,
whenever we agree on a formal notion of “definition”, that the sentence in
quotes is not a definition.
Let x(p, k) for k = 1, 2, . . . be an enumeration of the domain of U p . For
any natural number m in the range of U p , let f (p, m) be equal to the first
x(p, k) with U p (x(p, k)) = m. Then by definition, m < K( f (p, m)). On
the other hand, f (p, m) is a p.r. function, hence applying Corollary 1.4.2
and (1.3.5) we get
+
m < K( f (p, m)) < K(p) + J(K(m))
+
< l(p) + 1.5 log m.

23
1. COMPLEXITY

Church’s first example of an undecidable recursively enumerable set used

universal partial recursive functions and the diagonal technique of Gödel
and Cantor, which is in close connection to Russel’s paradox. The set he
constructed is complete in the sense of “many-one reducibility”.
The first example of sets not complete in this sense were Post’s simple
sets.

Definition 1.5.7. A computably enumerable set is simple if its complement is

infinite but does not contain any infinite computably enumerable subsets. ù

The paradox used in the above proof is not equivalent in any trivial way to
Russell’s paradox, since the undecidable computably enumerable sets we get
from Theorem 1.5.2 are simple. Let f (n) ¶ log n be any recursive function
with limn→∞ f (n) = ∞. We show that the set E = { n : K(n) ¶ f (n) } is
simple. It follows from Theorem 1.5.1 that E is recursively enumerable, and
from Theorem 1.3.2 that its complement is infinite. Let A be any computably
enumerable set disjoint from E. The restriction fA(n) of the function f (n) to
A is a p.r. lower bound for K(n). It follows from Theorem 1.5.2 that f (A) is
bounded. Since f (n) → ∞, this is possible only if A is finite.
Gödel used the diagonal technique to prove the incompleteness of any
sufficiently rich logical theory with a recursively enumerable axiom system.
His technique provides for any sufficiently rich computably enumerable the-
ory a concrete example of an undecidable sentence. Theorem 1.5.2 leads to
a new proof of this result, with essentially different examples of undecidable
propositions. Let us consider any first-order language L containing the lan-
guage of standard first-order arithmetic. Let 〈·〉 be a standard encoding of
the formulas of this theory into natural numbers. Let the computably enu-
merable theory Tp be the theory for which the set of codes 〈Φ〉 of its axioms
Φ is the computably enumerable set Wp of natural numbers.

Corollary 1.5.2. There is a constant c < ∞ such that for any p, if any sentences
with the meaning “m < K(n)” for m > J(l(p)) + c are provable in theory Tp
then some of these sentences are false.

Proof. For some p, suppose that all sentences “m < K(n)” provable in Tp are
true. For a sentence Φ, let Tp ` Φ denote that Φ is a theorem of the theory
Tp . Let us define the function

A(p, n) = max{ m : Tp ` “m < K(n)” }.

24
 1.5. Algorithmic properties of complexity

This function is semicomputable since the set { (p, 〈Φ〉) : Tp ` Φ } is recur-

sively enumerable. Therefore there is a binary string a such that A(p, n) =
S(a, 〈p, n〉). It is easy to see that we have a constant string b such that

S(a, 〈p, n〉) = S(q, n)

where q = bap. (Formally, this statement follows for example from the so-
called Smn -theorem.) The function Sq (n) is a lower bound on the complexity
function K(n). By an immediate generalization of Theorem 1.5.2 for semi-
+ +
computable functions), we have S(q, n) < J(l(q)) < J(l(p)).

We have seen that the complexity function K(x | y) is not computable,

and does not have any nontrivial recursive lower bounds. It has many in-
teresting upper bounds, however, for example the ones in Theorems 1.3.2
and 1.4.3. The following theorem characterizes all upper semicomputable
upper bounds (majorants) of K(x | y).

Theorem 1.5.3 (Levin). Let F (x, y) be a function of strings semicomputable

+
from above. The relation K(x | y) < F (x, y) holds for all x, y if and only if we
have
+
log |{ x : F (x, y) < m }| < m (1.5.1)

for all strings y and natural numbers m.

+
Proof. Suppose that K(x | y) < F (x, y) holds. Then (1.5.1) follows from b
of Theorem 1.3.2.
Now suppose that (1.5.1) holds for all y, m. Then let E be the computably
enumerable set of triples (x, y, m) with F (x, y) < m. It follows from (1.5.1)
+
and Theorem 1.4.3 that for all (x, y, m) ∈ E we have K(x | y, m) < m. By
+
Theorem 1.4.5 then K(x | y) < m.

The minimum of any finite number of majorants is again a majorant.

Thus, we can combine different heuristics for recognizing patterns of sequen-
ces.

25
1. COMPLEXITY

1.6 THE CODING THEOREM

1.6.1 SELF-DELIMITING COMPLEXITY

Some theorems on the addition of complexity do not have as simple a form

as desirable. For example, Theorem 1.4.2 implies
+
K(x, y) < J(K(x)) + K( y).

We would like to see just K(x) on the right-hand side, but the inequality
+
K(x, y) < K(x) + K( y) does not always hold (see Exercise 5). The problem
is that if we compose a program for (x, y) from those of x and y then we have
to separate them from each other somehow. In this section, we introduce a
variant of Kolmogorov’s complexity discovered by Levin and independently
by Chaitin and Schnorr, which has the property that “programs” are self-
delimiting: this will free us of the necessity to separate them from each other.

Definition 1.6.1. A set of strings is called prefix-free if for any pair x, y of

elements in it, x is not a prefix of y.
A one-to-one function into a set of strings is a prefix code, or instantaneous
code if its domain of definition is prefix free.
An interpreter f (p, x) is self-delimiting (s.d.) if for each x, the set D x of
strings p for which f (p, x) is defined is a prefix-free set. ù

Example 1.6.1. The mapping p → p o is a prefix code. ù

A self-delimiting p.r. function f (p) can be imagined as a function com-
putable on a special self-delimiting Turing machine.

Definition 1.6.2. A Turing machine T is self-delimiting if it has no input tape,

but can ask any time for a new input symbol. After some computation and
a few such requests (if ever) the machine decides to write the output and
stop. ù

The essential difference between a self-delimiting machine T and an or-

dinary Turing machine is that T does not know in advance how many input
symbols suffice; she must compute this information from the input symbols
themselves, without the help of an “endmarker”.
Requiring an interpreter A(p, x) to be self-delimiting in p has many ad-
vantages. First of all, when we concatenate descriptions, the interpreter will

26
 1.6. The coding theorem

be able to separate them without endmarkers. Lost endmarkers were respon-

sible for the additional logarithmic terms and the use of the function J(n) in
Theorem 1.4.2 and several other formulas for K.

Definition 1.6.3. A self-delimiting partial recursive interpreter T is called

optimal if for any other s.d. p.r. interpreter F , we have
+
KT < KF . (1.6.1)

Theorem 1.6.1. Let us prove that there is an optimal s.d. interpreter.

Proof. The proof of this theorem is similar to the proof of Theorem 1.3.1.
We take the universal partial recursive function V (a, p, x). We transform
each function Va (p, x) = V (a, p, x) into a self-delimiting function Wa (p, x) =
W (a, p, x), but so as not to change the functions Va which are already self-
delimiting. Then we form T from W just as we formed U from V . It is easy to
check that the function T thus defined has the desired properties. Therefore
we are done if we construct W .
Let some Turing machine M compute y = Va (p, x) in t steps. We define
Wa (p, x) = y if l(p) ¶ t and if M does not compute in t or fewer steps any
output Va (q, x) from any extension or prefix q of length ¶ t of p. If Va is self-
delimiting then Wa = Va . But Wa is always self-delimiting. Indeed, suppose
that p0 v p1 and that M computes Va (pi , x) in t i steps. The value Wa (pi , x)
can be defined only if l(pi ) ¶ t i . The definition of Wa guarantees that if
t 0 ¶ t 1 then Wa (p1 , x) is undefined, otherwise Wa (p0 , x) is undefined.

Definition 1.6.4. We fix an optimal self-delimiting p.r. interpreter T (p, x)

and write
H(x | y) = K T (x | y).
We call H( y | x) the (self-delimiting) conditional complexity of x with respect
to y. ù

We will use the expression “self-delimiting” only if confusion may arise.

Otherwise, under complexity, we generally understand the self-delimiting
complexity. Let T (p) = T (p, Λ). All definitions for unconditional complexity,
joint complexity etc. are automatically in force for H = K T .

27
1. COMPLEXITY

Let us remark that the s.d. interpreter defined in Theorem 1.6.1 has the
following stronger property. For any other s.d. interpreter G there is a string
g such that we have
G(x) = T (g x) (1.6.2)
for all x. We could say that the interpreter T is universal.
Let us show that the functions K and H are asymptotically equal.

Theorem 1.6.2.
+ +
K < H < J(K). (1.6.3)
+
Proof. Obviously, K < H. We define a self-delimiting p.r. function F (p, x).
The machine computing F tries to decompose p into the form p = uo v such
that u is the number l(v) in binary notation. If it succeeds, it outputs U(v, x).
We have
+ +
H( y | x) < K F ( y | x) < K( y | x) + 2 log K( y | x).

Let us review some of the relations proved in Sections 1.3 and 1.4. In-
+
stead of the simple estimate K(x) < n for a binary string x, of length n, only
the obvious consequence of Theorem 1.6.2 holds that is
+
H(x) < J(n).

We must similarly change Theorem 1.4.3. We will use the universal p.r. funct-
ion Tp (x) = T (p, x). The r.e. set Vp is the range of the function Tp . Let
E = Vp be an enumerable set of pairs of strings. The section E a is defined as
in (1.4.5). Then for all x ∈ E a , we have
+
H(x | a) < J(log |E a |) + l(p). (1.6.4)

We do not need J(l(p)) since the function T is self-delimiting. However, the

real analogon of Theorem 1.4.3 is the Coding Theorem, to be proved later in
this section.
Part b of Theorem 1.3.2 holds for H, but is not the strongest what can
be said. The counterpart of this theorem is the inequality (1.6.10) below.
Theorem 1.4.1 and its corollary hold without change for H. The additivity
relations will be proved in a sharper form for H in the next section.

28
 1.6. The coding theorem

1.6.2 UNIVERSAL SEMIMEASURE

Self-delimiting complexity has an interesting characterization that is very

useful in applications.

Definition 1.6.5. A function w : S → [0, 1] is called a semimeasure over the

space S if X
w(x) ¶ 1. (1.6.5)
x

It is called a measure (a probability distribution) if equality holds here.

A semimeasure is called constructive if it is lower semicomputable. A
constructive semimeasure which multiplicatively dominates every other con-
structive semimeasure is called universal. ù

Remark 1.6.2. Just as in an earlier remark, we warn that semimeasures will

later be defined over the space NN . They will be characterized by a nonneg-
ative function w over S = N∗ but with a condition different from (1.6.5). ù
The following remark is useful.

Proposition 1.6.3. If a constructive semimeasure w is also a measure then it

is computable.

Proof. If we compute an approximation w t of the function w from below for

which 1 − " < x w t (x) then we have |w(x) − w t (x)| < " for all x.
P

Theorem 1.6.3. There is a universal constructive semimeasure.

Proof. Every computable family P w i (i = 1, 2, . . . ) of semimeasures is domi-

nated by the semimeasure i 2−i w i . Since there are only countably many
constructive semimeasures there exists a semimeasure dominating them all.
The only point to prove is that the dominating semimeasure can be made
constructive.
Below, we construct a function µ p (x) lower semicomputable in p, x such
that µ p as a function of x is a constructive semimeasure for all p and all con-
structive semimeasures occur among the w p . Once we have Pµ p we are done
because for any positive computable function δ(p) with p δ(p) ¶ 1 the
semimeasure p δ(p)µ p is obviously lower semicomputable and dominates
P

all the µ p .

29
1. COMPLEXITY

Let S p (x) be the lower semicomputable function with Gödel number p

and let S pt (x) be a function recursive in p, t, x with rational values, donde-
creasing in t, such that lim t S pt (x) = max{0, S p (x)} and for each t, p, the
function S pt (x) is different from 0 only for x ¶ t. Let us define µ pt recur-
sively in t as follows. Let µ0p (x) = 0. Suppose that µ pt is already defined. If
S pt+1 is a semimeasure then we define µ pt+1 = S pt+1 , otherwise µ pt+1 = µ pt . Let
µ p = lim t µ pt . The function µ p (x) is, by its definition, lower semicomputable
in p, x and a semimeasure for each fixed p. It is equal to S p whenever S p is a
semimeasure, hence it enumerates all constructive semimeasures.

The above theorem justifies the following notation.

Definition 1.6.6. We choose a fixed universal constructive semimeasure and

call it m(x). ù

With this notation, we can make it a little more precise in which sense
this measure dominates all other constructive semimeasures.

Definition 1.6.7. For an arbitrary constructive semimeasure ν let us define

X
m(ν) = { m(p) : the (self-delimiting) program p computes ν }. (1.6.6)

Similar notation will be used for other objects: for example now m( f ) make
sense for a recursive function f . ù

Let us apply the above concept to some interesting cases.

Theorem 1.6.4. For all constructive semimeasures ν and for all strings x, we
have
∗
m(ν)ν(x) < m(x). (1.6.7)

Proof. Let us repeat the proof of Theorem 1.6.3, using m(p) in place of
δ(p). We obtain a new constructive semimeasure m0 (x) with the property
m(ν)ν(x) ¶ m0 (x) for all x. Noting m0 (x) = m(x) finishes the proof.
∗

Further, let us use the notation

H(x) = − log m(x)

for all kinds of argument x.

30
 1.6. The coding theorem

1.6.3 PREFIX CODES

+
The main theorem of this section says H(x) = − log m(x). Before proving it,
we relate descriptional complexity to the classical coding problem of infor-
mation theory.
Definition 1.6.8. A (binary) code is any mapping from a set E of binary
sequences to a set of objects. This mapping is the decoding function of the
code. If the mapping is one-to-one, then sometimes the set E itself is also
called a code.
A code is a prefix code if E is a prefix-free set. ù
The interpreter U(p) is a decoding function.
Definition 1.6.9. We call string p the first shortest description of x if it is the
lexicographically first binary word q with |q| = K(x) and U(q) = x. ù
The set of first shortest descriptions is a code. This imposes the implicit
lower bound stated in (1.5.1) on K(x):
|{ x : K(x) = n }| ¶ 2n . (1.6.8)
The least shortest descriptions in the definition of H(x) form moreover a
prefix code.
We recall a classical result of information theory.
Lemma 1.6.4 (Kraft’s Inequality, see [13]). For any sequence l1 , l2 , . . . of na-
tural numbers, there is a prefix code with exactly these numbers as codeword
lengths, if and only if X
2−li ¶ 1. (1.6.9)
i

Proof. Recall the standard correspondence x ↔ [x] between binary strings

and binary subintervals of the interval [0, 1]. A prefix code corresponds to
a set of disjoint intervals, and the length of the interval [x] is 2−l(x) . This
proves that (1.6.9) holds for the lengths of codewords of a prefix code.
Suppose that l i is given and (1.6.9) holds. We can assume that the se-
quence l i is nondecreasing. Chop disjoint, adjacent intervals I1 , I2 , . . . of
length 2−l1 , 2−l2 , . . . from the left end of the interval [0, 1]. The right end
Pk
of I k is j=1
2−l j . Since the sequence l j is nondecreasing, all intervals I j
are binary intervals. Take the binary string corresponding to I j as the j-th
codeword.

31
1. COMPLEXITY

+
Corollary 1.6.5. We have − log m(x) < H(x).

Proof. The lemma implies

X
2−H( y|x) ¶ 1. (1.6.10)
y

Since H(x) is an upper semicomputable function, the function 2−H(x) is

lower semicomputable. Hence it is a constructive semimeasure, and we have
+
− log m(x) < H(x).
The construction in the second part of the proof of Lemma 1.6.4 has
some disadvantages. We do not always want to rearrange the numbers l j , for
example because we want that the order of the codewords reflect the order
of our original objects. Without rearrangement, we can still achieve a code
with only slightly longer codewords.

Lemma 1.6.6 P (Shannon-Fano code). (see [13]) Let w1 , w2 , . . . be positive

numbers with j w j ¶ 1. There is a binary prefix code p1 , p2 , . . . where the
codewords are in lexicographical order, such that

|p j | ¶ − log w j + 2. (1.6.11)

Proof. We follow the construction above and cut off disjoint, adjacent (not
necessarily binary) intervals I j of length w j from the left end of [0, 1]. Let
v j be the length of the longest binary intervals contained in I j . Let p j be the
binary word corresponding to the first one of these. Four or fewer intervals
of length v j cover I j . Therefore (1.6.11) holds.

1.6.4 THE CODING THEOREM FOR H(x)

The above results suggest H(x) ¶ − log m(x). But in the proof, we have to
deal with the problem that m is not computable.

Theorem 1.6.5 (Coding Theorem, see [30, 17, 10]).

+
H(x) = − log m(x) (1.6.12)

Proof. We construct a self-delimiting p.r. function F (p) with the property that
K F (x) ¶ − log m(x) + 4. The function F to be constructed is the decoding

32
 1.6. The coding theorem

function of a prefix code hence the code-construction of Lemma 1.6.4 pro-

poses itself. But since the function m(x) is only lower semicomputable, it is
given only by a sequence converging to it from below.
Let { (z t , k t ) : t = 1, 2, . . . } be a recursive enumeration of the set { (x, k) :
k < m(x) } without repetition. Then
X XX X
2−kt = 2−kt ¶ 2m(x) < 2.
t x z t =1 x

Let us cut off consecutive adjacent, disjoint intervals I t of length 2−kt −1 from
the left side of the interval [0, 1]. We define F as follows. If [p] is a largest
binary subinterval of some I t then F (p) = z t . Otherwise F (p) is undefined.
The function F is obviously self-delimiting and partial recursive. It fol-
lows from the construction that for every x there is a t with z t = x and
0.5m(x) < 2−kt . Therefore, for every x there is a p such that F (p) = x and
|p| ¶ − log m(x) + 4.

The Coding Theorem can be straightforwardly generalized as follows. Let

f (x, y) be a lower semicomputable nonnegative function. Then we have
+
H( y | x) < − log f (x, y) (1.6.13)

f (x, y) ¶ 1. The proof is the same.

P
for all x with y

1.6.5 ALGORITHMIC PROBABILITY

We can interpret the Coding Theorem as follows. It is known in classical

information theory that for any probability distribution w(x) (x ∈ S) a binary
prefix code f w (x) can be constructed such that l( f w (x)) ¶ − log w(x)+1. We
learned that for computable, moreover, even for contstructive distributions
w there is a universal code with a self-delimiting partial-recursive decoding
function T independent of w such that for the codeword length H(x) we
have
H(x) ¶ − log w(x) + cw .
Here, only the additive constant cw depends on the distribution w.
Let us imagine the self-delimiting Turing machine T computing our in-
terpreter T (p). Every time the machine asks for a new bit of the description,
we could toss a coin to decide whether to give 0 or 1.

33
1. COMPLEXITY

Definition 1.6.10. Let PT (x) be the probability that the self-delimiting ma-
chine T gives out result x after receiving random bits as inputs. ù

We can write PT (x) = WT (x) where WT (x) is the set { 2−l(p) : T (p) =
P

x }. Since 2−H(x) = max WT (x), we have − log PT (x) ¶ H(x). The semimea-
∗
sure PT is constructive, hence P < m, hence
+ +
H < − log m < − log PT ¶ H,

hence
+ +
− log PT = − log m = H.
Hence the sum PT (x) of the set WT (x) is at most a constant times larger than
its maximal element 2−H(x) . This relation was not obvious in advance. The
outcome x might have high probability because it has many long descrip-
tions. But we found that then it must have a short description too. In what
follows it will be convenient to fix the definition of m(x) as follows.

Definition 1.6.11. From now on let us define

m(x) = PT (x). (1.6.14)

1.7 THE STATISTICS OF DESCRIPTION LENGTH

+
We can informally summarize the relation H = − log PT saying that if an
object has many long descriptions then it has a short one. But how many
descriptions does an object really have? With the Coding Theorem, we can
answer several questions of this kind. The reader can view this section as a
series of exercises in the technique acquired up to now.

Theorem 1.7.1. Let f (x, n) be the number of binary strings p of length n with
T (p) = x for the universal s.d. interpreter T defined in Theorem 1.6.1. Then
for every n ¾ H(x), we have
+
log f (x, n) = n − H(x, n). (1.7.1)
+
Using H(x) < H(x, n), which holds just as (1.4.1), and substituting
+ +
n = H(x) we obtain log f (x, H(x)) = H(x) − H(x, H(x)) < 0, implying the
following.

34
 1.7. The statistics of description length

Corollary 1.7.1. The number of shortest descriptions of any object is bounded

by a universal constant.

Since the number log f (x, H(x)) is nonnegative, we also derived the iden-
tity
+
H(x, H(x)) = H(x) (1.7.2)
which could have been proven also in the same way as Theorem 1.4.4.
Proof of Theorem 1.7.1. It is more convenient to prove equation (1.7.1) in
the form
2−n f (x, n) = m(x, n)
∗
(1.7.3)
∗
where m(x, y) = m(〈x, y〉). First we prove <. Let us define a p.r. self-
delimiting function F by F (p) = 〈T (p), l(p)〉. Applying F to a coin-tossing ar-
gument, 2−n f (x, n) is the probability that the pair 〈x, n〉 is obtained. There-
fore the left-hand side of (1.7.3), as a function of 〈x, n〉, is a constructive
semimeasure, dominated by m(x, n).
∗
Now we prove >. We define a self-delimiting p.r. function G as follows.
The machine computing G(p) tries to decompose p into three segments p =
β(c)o vw in such a way that T (v) is a pair 〈x, l(p) + c〉. If it succeeds then
it outputs x. By the universality of T , there is a binary string g such that
T (g p) = G(p) for all p. Let r = l(g). For an arbitrary pair x, n, let q be a
shortest binary string with T (q) = 〈x, n〉, and w an arbitrary string of length

l = n − l(q) − r − l(β(r)o ).

Then G(β(r)o qw) = T (gβ(r)o qw) = x. Since w is arbitrary here, there are
2l binary strings p of length n with T (p) = x.
How many objects are there with a given complexity n? We can answer
this question with a good approximation.

Definition 1.7.1. Let g T (n) be the number of objects x ∈ S with H(x) = n,

and Dn the set of binary strings p of length n for which T (p) is defined. Let
us define the moving average
c
1 X
h T (n, c) = g T (n + i).
2c + 1 i=−c

35
1. COMPLEXITY

Here is an estimation of these numbers.

Theorem 1.7.2. ([46]) There is a natural number c such that

+
log |Dn | = n − H(n), (1.7.4)
+
log h T (n, c) = n − H(n). (1.7.5)

Since we are interested in general only in accuracy up to additive con-

stants, we could omit the normalizing factor 1/(2c + 1) from the moving
average h T . We do not know whether this average can be replaced by g T (n).
This might depend on the special universal partial recursive function we use
to construct the optimal s.d. interpreter T (p). But equation (1.7.5) is true
for any optimal s.d. interpreter T (such that the inequality (1.6.1) holds for
all F ) while there is an optimal s.d. interpreter F for which g F (n) = 0 for
every odd n. Indeed, let F (00p) = T (p) if l(p) is even, F (1p) = T (p) if l(p)
is odd and let F be undefined in all other cases. Then F is defined only for
inputs of even length, while K F ¶ K T + 2.

Lemma 1.7.2. X
m(x, y) = m(x).
∗

Proof. The left-hand side is a constructive semimeasure therefore it is dom-

∗
inated by the right side. To show >, note that by equation (1.4.2) as
+
to H, we have H(x, x) = H(x). Therefore m(x) = m(x, x) <
∗
applied
P
y m(x, y).

Proof of Theorem 1.7.2: Let dn = |Dn |. Using Lemma 1.7.2 and Theo-
rem 1.7.1 we have
X X
g T (n) ¶ dn = f (x, n) = 2n m(x, n) = 2n m(n).
∗ ∗

x x

Since the complexity H has the same continuity property (1.4.3) as K, we

∗
can derive from here h T (n, c) ¶ 2n m(n)O(2c c 2 ). To prove h T (n, c) > dn for
an appropriate c, we will prove that the complexity of at least half of the
elements of Dn is near n.
For some constant c0 , we have H(p) ¶ n + c0 for all p ∈ Dn . Indeed, if the
s.d. p.r. interpreter F (p) is equal to p where T (p) is defined and is undefined

36
 1.7. The statistics of description length

+
otherwise then K F (p) ¶ n for p ∈ Dn . By H(p, l(p)) = H(p), and a variant of
Lemma 1.7.2, we have
X X
m(p) = m(p, n) = m(n).
∗ ∗

p∈Dn p∈Dn

Using the expression (1.7.4) for dn , we see that there is a constant c1 such
that X
dn−1 2−H(p) ¶ 2−n+c1 .
p∈Dn

Using Markov’s Inequality, hence the number of elements p of Dn with

H(p) ¾ n − c1 − 1 is at least dn /2. We finish the proof making c to be the
maximium of c0 and c1 + 1.
What can be said about the complexity of a binary string of length n? We
+
mentioned earlier the estimate H(x) < n + 2 log n. The same argument gives
the stronger estimate
+
H(x) < l(x) + H(l(x)). (1.7.6)
By a calculation similar to the proof of Theorem 1.7.2, we can show that the
estimate (1.7.6) is exact for most binary strings. First, it follows just as in
Lemma 1.7.2 that there is a constant c such that
X
2−n 2−H(p) ¶ c2−n m(n).
l(p)=n

From this, Markov’s Inequality gives that the number of strings p ∈ Bn with
H(p) < n − k is at most c2n−k .
There is a more polished way to express this result.

Definition 1.7.2. Let us introduce the following function of natural numbers:

H + (n) = max H(k).

k¶n

This is the smallest monotonic function above H(n). It is upper semi-

+
computable just as H is, hence 2−H is universal among the monotonically
decreasing constructive semimeasures. The following theorem expresses H +
more directly in terms of H.

37
1. COMPLEXITY

+
Theorem 1.7.3. We have H + (n) = log n + H(blog nc).
+
Proof. To prove <, we construct a s.d. interpreter as follows. The machine
computing F (p) finds a decomposition uv of p such that T (u) = l(v), then
outputs the number whose binary representation (with possible leading ze-
ros) is the binary string v. With this F , we have
+
H(k) < K F (k) ¶ log n + H(blog nc) (1.7.7)

for all k ¶ n. The numbers between n/2 and n are the ones whose binary
representation has length blog nc + 1. Therefore if the bound (1.7.6) is sharp
for most x then the bound (1.7.7) is sharp for most k.

The sharp monotonic estimate log n + H(blog nc) for H(n) is less satisfac-
tory than the sharp estimate log n for Kolmogorov’s complexity K(n), because
it is not a computable function. We can derive several computable upper
bounds from it, for example log n+2 log log n, log n+log log n+2 log log log n,
etc. but none of these is sharp for large n. We can still hope to find comput-
able upper bounds of H which are sharp infinitely often. The next theorem
provides one.

Theorem 1.7.4 (see [46]). There is a computable upper bound G(n) of the
function H(n) with the property that G(n) = H(n) holds for infinitely many n.

For the proof, we make use of a s.d. Turing machine T computing the
function T . Similarly to the proof of Theorem 1.5.1, we introduce a time-
restricted complexity.

Definition 1.7.3. We know that for some constant c, the function H(n) is
bounded by 2 log n + c. Let us fix such a c.
For any number n and s.d. Turing machine M , let HM (n; t) be the min-
imum of 2 log n + c and the lengths of descriptions from which M computes
n in t or fewer steps. ù

At the time we proved the Invariance Theorem we were not interested in

exactly how the universal p.r. function V was to be computed. Now we need
to be more specific.

Definition 1.7.4. Let us agree now that the universal p.r. function is com-
puted by a universal Turing machine V with the property that for any Turing

38
 1.7. The statistics of description length

machine M there is a constant m such that the number of steps needed to

simulate t steps of M takes no more than mt steps on V . Let T be the
Turing machine constructed from V computing the optimal s.d. interpreter
T , and let us write

H(n, t) = HT (n, t).

With these definitions, for each s.d. Turing machine M there are con-
stants m0 , m1 with
H(n, m0 t) ¶ HM (n, t) + m1 . (1.7.8)
The function H(n; t) is nonincreasing in t.
Proof of Theorem 1.7.4. First we prove that there are constants c0 , c1 such
that H(n; t) < H(n; t − 1) implies

H(〈n, t〉; c0 t) ¶ H(n; t) + c1 . (1.7.9)

Indeed, we can construct a Turing machine M simulating the work of T

such that if T outputs a number n in t steps then M outputs the pair 〈n, t〉
in 2t steps:
HM (〈n, t〉; 2t) ¶ H(n, t).
Combining this with the inequality (1.7.8) gives the inequality (1.7.9). We
define now a recursive function F as follows. To compute F (x), the s.d. Tur-
ing machine first tries to find n, t such that x = 〈n, t〉. (It stops even if it did
not find them.) Then it outputs H(x; c0 t). Since F (x) is the length of some
description of x, it is an upper bound for H(x). On the other hand, suppose
that x is one of the infinitely many integers of the form 〈n, t〉 with

H(n) = H(n; t) < H(n; t − 1).

+
Then the inequality (1.7.9) implies F (x) ¶ H(n) + c1 while H(n) < H(x)
is known (it is the equivalent of 1.4.4 for H), so F (x) ¶ H(n) + c for some
constant c. Now if F would not be equal to H at infinitely many places, only
close to it (closer than c) then we can decrease it by some additive constant
less than c and modify it at finitely many places to obtain the desired function
G.

39
2 RANDOMNESS

2.1 UNIFORM DISTRIBUTION

Speaking of a “random binary string”, one generally understands random-

ness with respect to the coin-toss distribution (when the probability P(x) of
a binary sequence x of length n is 2−n ). This is the only distribution consid-
ered in the present section.
One can hope to distinguish sharply between random and nonrandom
infinite sequences. Indeed, an infinite binary sequence whose elements are 0
with only finitely many exceptions, can be considered nonrandom without
qualification. This section defines randomness for finite strings. For a se-
quence x of length n, it would be unnatural to fix some number k, declare x
nonrandom when its first k elements are 0, but permit it to be random if only
the first k − 1 elements are 0. So, we will just declare some finite strings less
random than others. For this, we introduce a certain real-valued function
d(x) ¾ 0 measuring the deficiency of randomness in the string x. The occur-
rence of a nonrandom sequence is considered an exceptional event, therefore
the function d(x) can assume large values only with small probability. What
is “large” and “small” depends only on our “unit of measurement”. We re-
quire for all n, k
X
{ P(x) : x ∈ Bn , d(x) > k } < 2−k , (2.1.1)

saying that there be at most 2n−k binary sequences x of length n with d(x) >
k. Under this condition, we even allow d(x) to take the value ∞.
To avoid arbitrariness in the distinction between random and nonran-
dom, the function d(x) must be simple. We assume therefore that the set
{ (n, k, x) : l(x) = n, d(x) > k } is recursively enumerable, or, which is the

41
2. RANDOMNESS

same, that the function

d : Ω → (−∞, ∞]

is lower semicomputable.
Remark 2.1.1. We do not assume that the set of strings x ∈ Bn with small
deficiency of randomness is enumerable, because then it would be easy to
construct a “random” sequence. ù

Definition 2.1.1. A lower semicomputable function d : S2 → R (where R

is the set of real numbers) is called a Martin-Löf test (ML-test) if it satis-
fies (2.1.1).
A ML-test d0 (x) is universal if it additively dominates all other ML-tests:
for any other ML-test d(x) there is a constant c < ∞ such that for all x we
have d(x) < d0 (x) + c. ù

If a test d0 is universal then any other test d of randomness can discover at

most by a constant amount more deficiency of randomness in any sequence
x than d0 (x). Obviously, the difference of any two universal ML-tests is
bounded by some constant.
The following theorem reveals a simple connnection between descrip-
tional complexity and a certain randomness property.

Definition 2.1.2. Let us define the following function for binary strings x:

d0 (x) = l(x) − K(x | l(x)). (2.1.2)

Theorem 2.1.1 (Martin-Löf). The function d0 (x) is a universal Martin-Löf

test.

Proof. Since K(x | y) is semicomputable from above, d0 is semicomputable

from below. The property of semicomputability holds for d0 as a straightfor-
ward consequence of Theorem 1.5.1 Therefore d0 is a ML-test. We must show
that it is larger (to within an additive constant) than any other ML-test. Let d
be a ML-test. Let us define the function F (x, y) to be y − d(x) for y = l(x),
and ∞ otherwise. Then F is upper semicomputable, and satisfies (1.5.1).
+
Therefore by Theorem 1.5.3, we have K(x | l(x)) < l(x) − d(x).

42
 2.1. Uniform distribution

Theorem 2.1.1 says that under very general assumptions about random-
ness, those strings x are random whose descriptional complexity is close to
its maximum, l(x). The more “regularities” are discoverable in a sequence,
the less random it is. However, this is true only of regularities which decrease
the descriptional complexity. “Laws of randomness” are regularities whose
probability is high, for example the law of large numbers, the law of iterated
logarithm, the arcsine law, etc. A random sequence will satisfy all such laws.
Example 2.1.2. The law of large numbers says that in most binary sequences
of length n, the number of 0’s is close to the numer of 1’s. We prove this law
of probability theory by constructing a ML-test d(x) taking large values on
sequences in which the number of 0’s is far from the number of 1’s.
Instead of requiring (2.1.1) we require a somewhat stronger property of
d(x): for all n, X
P(x)2d(x) ¶ 1. (2.1.3)
x∈Bn

From this, the inequality (2.1.1) follows by the following well-known

inequality called the Markov Inequality which says the following. Let P be
any probability distribution, f any nonnegative function, with expected value
E P ( f ) = x P(x) f (x). For all λ ¾ 0 we have
P

X
{ P(x) : f (x) > λE P ( f ) } ¶ λ. (2.1.4)

For any string x ∈ S, and natural number i, let N (i | x) denote the number
of occurrences of i in x. For a binary string x of length n define p x = N (1 |
x)/n, and

Px ( y) = p Nx (1| y) (1 − p x )N (0| y)
d(x) = log Px (x) + n − log(n + 1).

We show that d(x) is a ML-test. It is obviously computable. We prove (2.1.3).

We have
X X 1 1 X
P(x)2d(x) = 2−n 2n Px (x) = Px (x)
x x
n+1 n+1 x
n   n
X 1 n k k k n−k 1 X
= ( ) (1 − ) < 1 = 1.
n + 1 k=0 k n n n + 1 k=0

43
2. RANDOMNESS

The test d(x) expresses the (weak) law of large numbers in a rather strong
version. We rewrite d(x) as
d(x) = n(1 − h(p x )) − log(n + 1)
where h(p) = −p log p−(1−p) log(1−p). The entropy function h(p) achieves
its maximum 1 at p = 1/2. Therefore the test d(x) tells us that the probabil-
ity of sequences x with p x < p < 1/2 for some constant p is bounded by the
exponentially decreasing quantitiy (n + 1)2−n(1−h(p)) . We also see that since
for some constant c we have
1 − h(p) > c(p − 1/2)2 ,
therefore if the difference |p x − 1/2| is much larger than
r
log n
cn
then the sequence x is “not random”, and hence, by Theorem 2.1.1 its com-
plexity is significantly smaller than n. ù

2.2 COMPUTABLE DISTRIBUTIONS

Let us generalize the notion of randomness to arbitrary discrete computable

probability distributions.

2.2.1 TWO KINDS OF TEST

Let P(x) be a probability distribution over some discrete countable space Ω,

which we will identify for simplicity with the set S of finite strings of natural
numbers. Thus, P is a nonnegative function with
X
P(x) = 1.
x

Later we will consider probability distributions P over the space NN , and they
will also be characterized by a nonnegative function P(x) over S. However,
the above condition will be replaced by a different one.
We assume P to be computable, with some Gödel number e. We want
to define a test d(x) of randomness with respect to P. It will measure how
justified is the assumption that x is the outcome of an experiment with dis-
tribution P.

44
 2.2. Computable distributions

Definition 2.2.1. A function d : S → R is an integrable test of randomness

with respect to P if it is lower semicomputable and satisfies the condition
X
P(x)2d(x) ¶ 1. (2.2.1)
x

It is universal if it dominates all other integrable tests to within an additive

constant. ù
(A similar terminology is used in [35] in order to distinguish tests satis-
fying condition (2.2.1) from Martin-Löf tests.)
Remark 2.2.1. We must allow d(x) to take negative values, even if only, say
values ¾ −1, since otherwise condition (2.2.1) could only be satisfied with
d(x) = 0 for all x with P(x) > 0. ù
Proposition 2.2.2. Let c = log(2π2 /6). If a function d(x) satisfies (2.2.1)
then it satisfies (2.1.1). If d(x) satisfies (2.1.1) then d − 2 log d − c satis-
fies (2.2.1).
Proof. The first statement follows by Markov’s Inequality (2.1.4). The second
one can be checked by immediate computation.
Thus, condition 2.2.1 is nearly equivalent to condition (2.1.1): to each
test d according to one of them, there is a test d 0 asymptotically equal to d
satisfying the other. But condition (2.2.1) has the advantage of being just
one inequality instead of the infinitely many ones.

2.2.2 RANDOMNESS VIA COMPLEXITY

We want to find a universal integrable test. Let us introduce the funct-

ion w(x) = P(x)2d(x) . The above conditions
P on d(x) imply that w(x) is
semicomputable from below and satisfies x w(x) ¶ 1: so, it is a construc-
tive semimeasure. With the universal constructive semimeasure of Theo-
rem 1.6.3, we also obtain a universal test for any computable probability
distribution P.
Definition 2.2.2. For an arbitrary measure P over a discrete space Ω, let us
denote
m(x)
d P (x) = log = − log P(x) − H(x). (2.2.2)
P(x)
ù

45
2. RANDOMNESS

The following theorem shows that randomness can be tested by checking

how close is the complexity H(x) to its upper bound − log P(x).

Theorem 2.2.1. The function d P (x) = − log P(x) − H(x) is a universal inte-
grable test for any fixed computable probability distribution P. More exactly, it
is lower semicomputable, satisfies (2.2.1) and for all integrable tests d(x) for
P, we have
+
d(x) < d P (x) + H(d) + H(P). (2.2.3)

Proof. Let d(x) be an integrable test for P. Then ν(x) = 2d(x) P(x) is a
+
constructive semimeasure, and it has a self-delimiting program of length <
∗
H(d) + H(P). It follows (using the definition (1.6.6)) that m(ν) < 2H(d)+H(P) ;
hence inequality (1.6.7) gives
∗
ν(x)2H(d)+H(P) < m(x).

Taking logarithms finishes the proof.

The simple form of our universal test suggests remarkable interpretations.

It says that the outcome x is random with respect to the distribution P, if
the latter assigns to x a large enough probability—however, not in absolute
terms, only relatively to the universal semimeasure m(x). The relativization
is essential, since otherwise we could not distinguish between random and
nonrandom outcomes for the uniform distribution Pn defined in Section 2.1.
For any (not necessarily computable) distribution P, the P-expected value
of the function m(x)/P(x) is at most 1, therefore the relation

m(x) ¶ kP(x)

holds with probability not smaller than 1 − 1/k. If P is computable then the
inequality
∗
m(P)P(x) < m(x)
holds for all x. Therefore the following applications are obtained.
• If we assume x to be the outcome of an experiment with some simple
computable probability distribution P then m(x) is a good estimate of
P(x). The goodness depends on how simple P is to define and how
random x is with respect to P: how justified is the assumption. Of
course, we cannot compute m(x) but it is nevertheless well defined.

46
 2.2. Computable distributions

• If we trust that a given object x is random with respect to a given

distribution P then we can use P(x) as an estimate of m(x). The degree
of approximation depends on the same two factors.
Let us illustrate the behavior of the universal semimeasure m(x) when x
runs over the set of natural numbers. We define the semimeasures v and w
as follows. Let v(n) = c/n2 for an appropriate constant c, let w(n) = v(x)
k
if n = 22 , and 0 otherwise. Then m(n) dominates both v(n) and w(n).
The function m(n) dominates 1/n log2 n, but jumps at many places higher
than that. Indeed, it is easy to see that m(n) converges to 0 slower than
any positive computable function converging to 0: P in particular, it is not
computable. Hence it cannot be a measure, that is x m(x) < 1. We feel
that we can make m(x) “large” whenever x is “simple”.
We cannot compare the universal test defined in the present section di-
rectly with the one defined in Section 2.1. There, we investigated tests for a
whole family Pn of distributions, where Pn is the uniform distribution on the
set Bn . That the domain of these distributions is a different one for each n is
not essential since we can identify the set Bn with the set {2n , . . . , 2n+1 − 1}.
Let us rewrite the function d0 (x) defined in (2.1.2) as

d0 (x) = n − K(x | n)

for x ∈ Bn , and set it ∞ for x outside Bn . Now the similarity to the expression

d P (x) = − log P(x) − log m(x)

is striking because n = log Pn (x). Together with the remark made in the
previous paragraph, this observation suggests that the value of − log m(x) is
close to the complexity K(x).

2.2.3 CONSERVATION OF RANDOMNESS

The idea of conservation of randomness has been expressed first by Levin,

who has published some papers culminating in [32] developing a general
theory. In this section, we address the issue in its simplest form only. Suppose
that we want to talk about the randomness of integers. One and the same
integer x can be represented in decimal, binary, or in some other notation:
this way, every time a different string will represent the same number. Still,
the form of the representation should not affect the question of randomness
of x significantly. How to express this formally?

47
2. RANDOMNESS

Let P be a computable measure and let f : S → S be a computable funct-

ion. We expect that under certain conditions, the randomness of x should
imply the randomness of f (x)—but, for which distribution? For example,
if x is a binary string and f (x) is the decimal notation for the number ex-
pressed by 1x then we expect f (x) to be random not with respect to P, but
with respect to the measure f ∗ P that is the image of P under f . This measure
is defined by the formula
X
( f ∗ P)( y) = P( f −1 ( y)) = P(x). (2.2.4)
f (x)= y

Proposition 2.2.3. If f is a computable function and P is a computable mea-

sure then f ∗ P is a computable measure.
Proof. It is sufficient to show that f ∗ P is lower semicomputable, since we
have seen that a lower semicomputable measure is computable. But, the
semicomputability can be seen immediately from the definition.
Let us see that randomness is indeed preserved for the image. Similarly
to (1.6.6), let us define for an arbitrary computable function f :
X
m( f ) = { m(p) : program p computes f }. (2.2.5)
Theorem 2.2.2. Let f be a computable function. Between the randomness test
for P and the test for the image of P, the following relation holds, for all x:
+
d f ∗ P ( f (x)) < d P (x) + H( f ) + H(P). (2.2.6)
Proof. Let us denote the function on the left-hand side of (2.2.6) by
m( f (x))
d P (x) = d f ∗ P ( f (x)) = log .
( f ∗ P)( f (x))
It is lower semicomputable, with the help of a program of length H( f ) +
H(P), by its very definition. Let us check that it is an integrable test, so it
satisfies the inequality (2.2.1). We have
X X X
P(x)2dP (x) = ( f ∗ P)( y)2d f ∗ P ( y) = m( y) ¶ 1.
x y y

+
Hence d P (x) is a test, and hence it is < d P (x) + H( f ) + H(P), by the uni-
versality of the test d P (x). (Strictly speaking, we must modify the proof of
Theorem 2.2.1 and see that a program of length H( f )+H(P) is also sufficient
to define the semimeasure P(x)d(x | P).)

48
 2.2. Computable distributions

The appearance of the term H( f ) on the right-hand side is understand-

able: using some very complex function f , we can certainly turn any string
into any other one, also a random string into a much less random one. On
the other hand, the term H(P) appears only due to the imperfection of our
concepts. It will disappear in the more advanced theory presented in later
sections, where we develop uniform tests.
Example 2.2.4. For each string x of length n ¾ 1, let

2−n
P(x) = .
n(n + 1)

This distribution is uniform on strings of equal length. Let f (x) be the funct-
ion that erases the first k bits. Then for n ¾ 1 and for any string x of length
n we have
2−n
( f ∗ P)(x) = .
(n + k)(n + k + 1)
(The empty string has the rest of the weight of f ∗ P.) The new distribution
is still uniform on strings of equal length, though the total probability of
strings of length n has changed somewhat. We certainly expect randomnes
conservation here: after erasing the first k bits of a random string of length
n + k, we should still get a random string of length n. ù
The computable function f applied to a string x can be viewed as some
kind of transformation x 7→ f (x). As we have seen, it does not make x less
random. Suppose now that we introduce randomization into the transforma-
tion process itself: for example, the machine computing f (x) can also toss
some coins. We want to say that randomness is also conserved under such
more general transformations, but first of all, how to express such a trans-
formation mathematically? We will describe it by a “matrix”: a computable
probability transition function T (x, y) ¾ 0 with the property that
X
T (x, y) = 1.
y

Now, the image of the distribution P under this transformation can be written
as T ∗ P, and defined as follows:
X
(T ∗ P)( y) = P(x)T (x, y).
x

49
2. RANDOMNESS

How to express now randomness conservation? It is certainly not true that

every possible outcome y is as random with respect to T ∗ P as x is with
respect to P. We can only expect that for each x, the conditional probability
(in terms of the transition T (x, y)) of those y whose non-randomness is
larger than that of x, is small. This will indeed be expressed by the corollary
below. To get there, we upperbound the T (x, ·)-expected value of 2dT ∗ P ( y) .
Let us go over to exponential notation:

Definition 2.2.3. Denote

m(x)
t P (x) = 2dP (x) = . (2.2.7)
P(x)

Theorem 2.2.3. We have

+
X
log T (x, y)2dT ∗ P ( y) < d P (x) + H(T ) + H(P). (2.2.8)
y

Proof. Let us denote the function on the left-hand side of (2.2.8) by t P (x). It
is lower semicomputable by its very construction, using a program of length
+
< H(T ) + H(P). Let us check that it satisfies x P(x)t P (x) ¶ 1 which, in this
P

notation, corresponds to inequality (2.2.1). We have

X X X
P(x)t P (x) = P(x) T (x, y)t T ∗ P ( y)
x x y
X X m( y) X
= P(x) T (x, y) = m( y) ¶ 1.
x y
(T ∗ P)( y) y

+
It follows that d P (x) = log t P (x) is an integrable test and hence d P (x) <
d P (x) + H(T ) + H(P). (See the remark at the end of the proof of Theo-
rem 2.2.2.)

Corollary 2.2.5. There is a constant c0 such that for every integer k ¾ 0, for
all x we have
X
{ T (x, y) : d T ∗ P ( y) − d P (x) > k + H(T ) + H(P) } ¶ 2−k+c0 .

50
 2.3. Infinite sequences

Proof. The theorem says

X
T (x, y)t T ∗ P ( y) < t P (x)2H(T )+H(P) .
∗

Thus, it upperbounds the expected value of the function 2dT ∗ P) ( y) according to

the distribution T (x, ·). Applying Markov’s inequality (2.1.4) to this function
yields the desired result.

2.3 INFINITE SEQUENCES

These lecture notes treat the theory of randomness over continuous spaces
mostly separately, starting in Section 4.1. But the case of computable mea-
sures over infinite sequences is particularly simple and appealing, so we give
some of its results here, even if most follow from the more general results
given later.
In this section, we will fix a finite or countable alphabet Σ = {s1 , s2 , . . .},
and consider probability distributions over the set

X = ΣN

of infinite sequences with members in Σ. An alternative way of speaking of

this is to consider a sequence of random variables X 1 , X 2 , . . . , where X i ∈ Σ,
with a joint distribution. The two interesting extreme special cases are Σ =
B = {0, 1}, giving the set of infinite 0-1 sequences, and Σ = N, giving the set
sequences of natural numbers.

2.3.1 NULL SETS

Our goal is here to illustrate the measure theory developed in Section A.2,
through Subsection A.2.3, on the concrete example of the set of infinite se-
quences, and then to develop the theory of randomness in it.
We will distinguish a few simple kinds of subsets of the set S of sequences,
those for which probability can be defined especially easily.

Definition 2.3.1.
• For string z ∈ Σ∗ , we will denote by zX the set of elements of X with prefix
z. Such sets will be called cylinder sets.

51
2. RANDOMNESS

• A subset of X is open if it is the union of any number (not necessarily finite)

of cylinder sets. It is called closed if its complement is open.
• An open set is called constructive if it is the union of a recursively enumer-
able set of cylinder sets.
• A set is called Gδ if it is the intersection of a sequence of open sets. It is
called Fσ if it is the union of a sequence of closed sets.
• We say that a set E ⊆ X is finitely determined if there is an n and an E ⊆ Σn
such that
[
E = EX = sX .
s∈E

Let F be the class of all finitely determined subsets of X .

• A class of subsets of X is called an algebra if it is closed with respect to
finite intersections and complements (and then of course, also with respect
to finite unions). It is called a σ-algebra (sigma-algebra) when it is also
closed with respect to countable intersections (and then of course, also
with respect to countable unions).
ù

Example 2.3.1. An example open set that is not finitely determined, is the set
E of all sequences that contain a substring 11. ù
The following observations are easy to prove.

Proposition 2.3.2.
(a) Every open set G can be represented as the union of a sequence of disjoint
cylinder sets.
(b) The set of open sets is closed with respect to finite intersection and arbitrar-
ily large union.
(c) Each finitely determined set is both open and closed.
(d) The class F of finitely determined sets forms an algebra.

Probability is generally defined as a nonnegative, monotonic function (a

“measure”, see later) on some subsets of the event space (in our case the set
X ) that has a certain additivity property. We do not need immediately the
complete definition of measures, but let us say what we mean by an additive
set function.

52
 2.3. Infinite sequences

Definition 2.3.2. Consider a nonnegative, function µ is defined over some

subsets of X . that is also monotonic, that is A ⊆ B implies µ(A) ⊆ µ(B).
We say that itSis additive if, whenever it is defined on the sets E1 , . . . , En ,
n
and on E = i=1 Ei and the sets Ei are mutually disjoint, then we have
µ(E) = µ(E1 ) + · · · + µ(En ).
We say that µ is countably additive, or σ-additive (sigma-additive), S∞if in
addition whenever it is defined on the sets E1 , E2 , . . . , andPon E = i=1 Ei
∞
and the sets Ei are mutually disjoint, then we have µ(E) = i=1 µ(Ei ). ù

First we consider only measures defined on cylinder sets.

Definition 2.3.3 (Measure over ΣN ). Let µ : Σ∗ → R+ be a function assigning

a nonnegative number to each string in Σ∗ . We will call µ a measure if it
satisfies the condition
X
µ(x) = µ(xs). (2.3.1)
s∈Σ

We will take the liberty to also write

µ(sX ) = µ(s)

for all s ∈ Σ∗ , and µ(;) = 0. We call µ a probability measure if µ(Λ) = 1 and

correspondingly, µ(X ) = 1.
A measure is called computable if it is computable as a function µ : Σ∗ →
R+ according to Definition 1.5.2. ù

The following observation lets us extend measures.

Lemma 2.3.3. Let µ be a measure defined as above. If a cylinder set sX is

P disjoint union x X = x 1 X ∪ x 2 X ∪ . . . of cylinder sets, then we have µ(z) =
the
i µ(x i ).

Proof. Let us call an arbitrary y ∈ Σ∗ good if

X
µ(z y) = µ(z y X ∩ x i X )
i

holds. We have to show that the empty string Λ is good.

It is easy to see that y is good if z y X ⊆ x i X for some i. Also, if ys is good
for all s ∈ Σ then y is good. Now assume that Λ is not good. Then there is

53
2. RANDOMNESS

also an s1 ∈ Σ that is not good. But then there is also an s2 ∈ Σ such that
s1 s2 is not good. And so on, we obtain an infinite sequence s1 s2 · · · such that
s1 ·S· · sn is not good for any n. But then the sequence zs1 s2 · · · is not contained
in i x i X , contrary to the assumption .
Corollary 2.3.4. Two disjoint union representations of the same open set
[ [
xi X = yi X
i i

µ(x i ) = i µ( yi ).
P P
imply i
S
Proof. The set i x i X can also be written as
[ [
xi X = xi X ∩ yj X .
i i, j

An element x i X ∩ y j X is nonempty only if one of the strings x i , y j is a con-

tinuation of the other. Calling this string zi j weShave x i X ∩ y j X = zi j X . Now
Lemma 2.3.3 is applicable to the union x i X = j (x i X ∩ y j X ), giving
X
µ(x i ) = µ(x i X ∩ y j X ),
j
X X
µ(x i ) = µ(x i X ∩ y j X ).
i i, j

µ( y j ).
P
The right-hand side is also equal similarly to j

The above corollary allows the following definition:

Definition 2.3.4.PFor an open set G given as a disjoint union of cylinder sets
x i X let µ(G) = i µ(x i ). For a closed set F = X \ G where G is open, let
µ(F ) = µ(X ) − µ(G). ù
The following is easy to check.
Proposition 2.3.5. The measure as defined on open sets is monotonic and
countably additive. In particular, it is countably additive on the algebra of
finitely determined sets.
Before extending measures to a wider range of sets, consider an impor-
tant special case. With infinite sequences, there are certain events that are
not impossible, but still have probability 0.

54
 2.3. Infinite sequences

Definition 2.3.5. Given a measure µ, a set N ⊆ X is called a null set with

T to µ if there is a sequence
respect G1 , G2 , . . . of open sets with the property
N ⊆ m Gm , and µ(Gm ) ¶ 2 . We will say that the sequence Gm witnesses
−m

the fact that N is a null set. ù

Examples 2.3.6. Let Σ = {0, 1}, let us define the measure λ by λ(x) = 2−n
for all n and for all x ∈ Σn .
1. For every infinite sequence ξ ∈ X , the one-element set {ξ} is a null set
with respect to λ. Indeed, for each natural number n, let H n = { ξ ∈ X :
ξ(0) = Tξ(0), ξ(1) = ξ(1), . . . , ξ(n) = ξ(n) }. Then λ(H n ) = 2
−n−1
, and
{s} = n H n .
2. For a less trivial example, consider the set E of those elements t ∈ X that
are 1 in each positive even position, that is

E = { t ∈ X : t(2) = 1, t(4) = 1, t(6) = 1 . . . }.

Then E is a null set. Indeed, for each natural number n, let Gn = { T t∈X :
t(2) = 1, t(4) = 1, . . . , t(2n) = 1 }. This helps expressing E as E = n Gn ,
where λ(Gn ) = 2−n .
ù

Proposition 2.3.7. Let N1 , NS

2 , . . . be a sequence of null sets with respect to a
measure µ. Their union N = i Ni is also a null set.

Proof. Let Gi,1 , Gi,2 ,S

. . . be the infinite sequence witnessing the fact that Ni is a
∞
null set. Let H m = i=1 Gi,m+i . Then the sequence H m of open sets witnesses
the fact that N is a null set.
We would like to extend our measure to null sets N and say µ(N ) = 0.
The proposition we have just proved shows that such an extension would be
countably additive on the null sets. But we do not need this extension for
the moment, so we postpone it. Still, given a probability measure P over X ,
it becomes meaningful to say that a certain property holds with probability
1. What this means is that the set of those sequences that do not have this
property is a null set.
Following the idea of Martin-Löf, we would like to call a sequence non-
random, if it is contained in some simple null set; that is it has some easily
definable property with probability 0. As we have seen in part 1 of Exam-
ple 2.3.6, it is important to insist on simplicity, otherwise (say with respect to

55
2. RANDOMNESS

the measure λ) every sequence might be contained in a null set, namely the
one-element set consisting of itself. But most of these sets are not defined
simply at all. An example of a simply defined null set is given is part 2 of
Example 2.3.6. These reflections justify the following definition, in which
“simple” is specified as “constructive”.

Definition 2.3.6. Let µ be a computable measure. A set N ⊆ X is called a

constructive null set if there is recursively enumerable set ΓS⊆ N × Σ∗ with the
property that denoting Γm = { x : (m, x) ∈ Γ } and Gm = x∈Γm x X we have
N ⊆ m Gm , and µ(Gm ) ¶ 2−m .
T
ù

In words, the difference betweenSthe definition of null sets and construc-

tive null sets is that the sets Gm = x∈Γm x X here are required to be con-
structive open, moreover, in such a way that from m one can compute the
program generating Gm . In even looser words, a set is a constructive null
set if for any " > 0 one can construct effectively a union of cylinder sets
containing it, with total measure ¶ ".
Now we are in a position to define random infinite sequences.

Definition 2.3.7 (Random sequence). An infinite sequence ξ is random with

respect to a probability measure P if and only if ξ is not contained in any
constructive null set with respect to P. ù

It is easy to see that the set of nonrandom sequences is a null set. Indeed,
there is only a countable number or constructive null sets, so even their union
is a null set. The following theorem strengthens this observation significantly.

Theorem 2.3.1. Let us fix a computable probability measure P. The set of all
nonrandom sequences is a constructive null set.

Thus, there is a universal constructive null set, containing all other con-
structive null sets. A sequence is random when it does not belong to this
set.

Proof of Theorem 2.3.1. The proof uses the projection technique that has ap-
peared several times in this book, for example in proving the existence of a
universal lower semicomputable semimeasure.
We know that it is possible to list all recursively enumerable subsets of
the set N × Σ∗ into a sequence, namely that there is a recursively enumerable

56
 2.3. Infinite sequences

set ∆ ⊆ N2 × Σ∗ with the property that for every recursively enumerable set
Γ ⊆ N × Σ∗ there is an e with Γ = { (m, x) : (e, m, x) ∈ ∆ }. We will write

∆e,m = { x : (e, m, x) ∈ ∆ },
[
De,m = xX.
x∈∆e,m

We transform the set ∆ into another set ∆0 with the following property.
• For each e, m we have (e,m,x)∈∆0 µ(x) ¶ 2−m+1 .
P

• If for some e we have (e,m,x)∈∆ µ(x) ¶ 2−m for all m then for all m we
P

have { x : (e, m, x) ∈ ∆0 } = { x : (e, m, x) ∈ ∆ }.

This transformation is routine, so we leave it to the reader. By the construc-
tion of ∆0 , for every constructive null set N there is an e with
\
0
N⊆ De,m .
m

Define the recursively enumerable set

Γ̂ = { (m, x) : ∃ e (e, m + e + 2, x) ∈ ∆0 }.

Then Ĝm =
S 0
e De,m+e+2 . For all m we have
X X X X
µ(x) = µ(x) ¶ 2−m−e−1 = 2−m .
x∈Γ̂m e (e,m+e+2,x)∈∆0 e

This shows that Γ̂ defines a constructive null set. Let Γ be any other recur-
sively enumerable subset of N × Σ∗ that defines a constructive null set. Then
there is an e such that for all m we have Gm = De,m
0
. The universality follows
now from
\ \ \ \
Gm = 0
De,m ⊆ 0
De,m+e+2 ⊆ Ĝm .
m m m m

2.3.2 PROBABILITY SPACE

Now we are ready to extend measure to a much wider range of subsets of X .

57
2. RANDOMNESS

Definition 2.3.8. Elements of the smallest σ-algebra A containing the cylin-

der sets of X are called the Borel sets of X . The pair (X , A ) is an example of
a measureable space, where the Borel sets are called the measureable sets.
A nonnegative sigma-additive function µ over A is called a measure. It is
called a probability measure if µ(X ) = 1. If µ is fixed then the triple (X , A , µ)
is called a measure space. If it is a probability measure then the space is called
a probability space. ù

In the Appendix we cited a central theorem of measure theory,

Caratheodory’s extension theorem. It implies that if a measure is defined
on an algebra L in a sigma-additive way then it can be extended uniquely
to the σ-algebra generated by L , that is the smallest σ-algebra containing
L . We defined a measure µ over X as a nonnegative function µ : Σ∗ → R+
satisfying the equality (2.3.1). Then we defined µ(x X ) = µ(x), and fur-
ther extended µ in a σ-additive way to all elements of the algebra F . Now
Caratheodory’s theorem allows us to extend it uniquely to all Borel sets, and
thus to define a measureable space (X , A , µ).
Of course, all null sets in A get measure 0.
Open, closed and measureable sets can also be defined in the set of real
numbers.

Definition 2.3.9. A subset G ⊆ R is open if it is the union of a set of open

intervals (ai , bi ). It is closed if its complement is open.
The set B of Borel sets of B is defined as the smallest σ-algebra containing
all open sets. ù

The pair (R, B) is another example of a measureable space.

Definition 2.3.10 (Lebesgue measure). Consider the set left-closed intervals

of the line (including intervals of the form (−∞, a). Let L be the set of finite
disjoint unions of such intervals. This isP
an algebra. We define the function
λ over L as follows: λ( i [ai , bi )) = i bi − ai . It is easy to check that
S

this is a σ-additive measure and therefore by Caratheodory’s theorem can be

extended to the set B of all Borel sets. This function is called the Lebesgue
measure over R, giving us the measureable space (R, B, λ). ù

Finally, we can define the notion of a measureable function over X .

Definition 2.3.11 (Measureable functions). A function f : X → R is called

measureable if and only if f −1 (E) ∈ A for all E ∈ B. ù

58
 2.3. Infinite sequences

The following is easy to prove.

Proposition 2.3.8. Function f : X → R is measureable if and only if all sets of

the form f −1 ((r, ∞)) = { x : f (x) > r } are measureable, where r is a rational
number.

2.3.3 COMPUTABILITY
Measureable functions are quite general; it is worth introducing some more
resticted kinds of function over the set of sequences.

Definition 2.3.12. A function f : X → R is continuous if and only if for every

x ∈ X , for every " > 0 there is a cylinder set C 3 x such that | f ( y)− f (x)| < "
for all y ∈ C.
A function f : X → R is lower semicontinuous if for every r ∈ R the
set { x ∈ X : f (x) > r } is open. It is upper semicontinuous if − f is lower
semicontinuous. ù

The following is easy to verify.

Proposition 2.3.9. A function f : X → R is continuous if and only if it is both

upper and lower semicontinuous.

Definition 2.3.13. A function f : X → R is computable if for every open

rational interval (u, v) the set f −1 ((u, v)) is a constructive open set of X ,
uniformly in u, v. ù

Informally this means that if for the infinite sequence ξ ∈ X we have

u < f (ξ) < v then from u, v sooner or later we will find a prefix x of ξ with
the property u < f (x X ) < v.

Definition 2.3.14. A function f : X → R is lower semicomputable if for every

rational r the set { s ∈ X : f (s) > r } is a constructive open set of X , uniformly
in r. It is upper semicomputable if − f is lower semicomputable. ù

The following is easy to verify.

Proposition 2.3.10. A function X → R is computable if and only if it is both

lower and upper semicomputable.

59
2. RANDOMNESS

2.3.4 INTEGRAL
The definition of integral over a measure space is given in the Appendix, in
Subsection A.2.3. Here, we give an exposition specialized to infinite sequen-
ces.

Definition 2.3.15. A measurable function f : X → R is called a step function

if its range is finite. The set of step functions will be called E .
Given a step function f which takes values x i on sets Ai , and a finite
measure µ, we define
Z Z
X
µ( f ) = µ f = f dµ = f (x)µ(d x) = x i µ(Ai ).
i

Proposition A.2.7, when specified to our situation here, says the follow-
ing.

Proposition 2.3.11. The functional µ defined above on step functions can be

extended to the set E+ of monotonic limits of nonnegative elements of E , by
continuity. The set E+ is the set of all nonnegative measurable functions.

Now we extend the notion of integral to a wider class of functions.

Definition 2.3.16. A measurable function f is called integrable with respect

to a finite measure µ if µ| f |+ < ∞ and µ| f |− < ∞. In this case, we define
µ f = µ| f |+ − µ| f |− . ù

It is easy to see that the mapping f 7→ µ f is linear when f runs through

the set of measureable functions with µ| f | < ∞.
The following is also easy to check.

Proposition 2.3.12. Let µ be a computable measure.

(a) If f is computable then a program to compute µ f can be found from the
program to compute f .
(b) If f is lower semicomputable then a program to lower semicompute µ f can
be found from a program to lower semicompute f .

60
 2.3. Infinite sequences

2.3.5 RANDOMNESS TESTS

We can now define randomness tests similarly to Section 2.2.

Definition 2.3.17. A function d : X → R is an integrable test of randomness

with respect to the probability measure P if it is lower semicomputable and
satisfies the condition Z
2d(x) P(d x) ¶ 1. (2.3.2)

It is called a Martin-Löf test, if instead of the latter condition only the weaker
one is satisfied saying P(d(x) > m) < 2−m for each positive integer m.
It is universal if it dominates all other integrable tests to within an additive
constant. Universal Martin-Löf tests are defined in the same way. ù

Randomness tests and constructive null sets are closely related.

Theorem 2.3.2. Let P be a computable probability measure over the set S =

Σ∞ . There is a correspondence between constructive null sets and randomness
tests:
(a) For every randomness tests d, the set N = { ξ : d(ξ) = ∞ } is a constructive
null set.
(b) For every constructive null set N there is a randomness test d with N = { ξ :
d(ξ) = ∞ }.

Proof. Let d be a randomness test: then for each k the set Gk = { ξ : d(ξ) >
k } is a constructive open set, and by Markov’s inequality (wich is proved in
the continuous case just as in the discrete case) we have P(Gk ) ¶ 2−k . The
sets Gk witness that N is a constructive null set. T∞
Let N be a constructive null set with N ⊆ G , where Gk is a
k=1 k
uniform sequence of constructive open sets with P(Gk ) = 2−k . Without
loss of generality assume that the sequence Gk is decreasing. Then the
function d(ξ) = sup{ k : ξ ∈ Gk } is lower semicomputable and satisfies
P{ ξ : d(ξ) ¾ k } ¶ 2−k , so it is a Martin-Löf test. Just as in Proposition 2.2.2,
it is easy to check that d(x) − 2 log d(x) − c is an integrable test for some
constant c.

Just as there is a universal constructive null set, there are universal ran-
domness tests.

61
2. RANDOMNESS

Theorem 2.3.3. For a computable measure P, there is a universal integrable

test d P (ξ) for any fixed computable probability distribution P. More exactly,
the function ξ 7→ d P (ξ) is lower semicomputable, satisfies (2.2.1) and for all
integrable tests d(ξ) for P, we have
+
d(ξ) < d P (ξ) + H(d) + H(P). (2.3.3)

The proof is similar to the proof Theorem 1.6.3 on the existence of a

universal constructive semimeasure. It uses the projection technique and a
weighted sum.

2.3.6 RANDOMNESS AND COMPLEXITY

We hope for a result connecting complexity with randomness similar to The-

orem 2.2.1. Somewhat surprisingly, there is indeed such a result. This is
a surprise since in an infinite sequence, arbitrarily large oscillations of any
quantity depending on the prefixes are actually to be expected.

Theorem 2.3.4. Let X = ΣN be the set of infinite sequences. For all computable
measures over X , we have
+
dµ (ξ) = sup − log µ(ξ1:n ) − H(ξ1:n ) .


(2.3.4)
n

+
Here, the constant in = depends on the computable measure µ.

Corollary 2.3.13. Let λ be the uniform distribution over the set of infinite
binary sequences. Then
+
dλ (ξ) = sup n − H(ξ1:n ) .


(2.3.5)
n

In other words, an infinite binary sequence is random (with respect to

the uniform distribution) if and only if the complexity H(ξ1:n ) of its initial
segments of length n never decreases below n by more than an additive
constant.
+
Proof. To prove <, define the function
X m(s | µ) X m(ξ
1:n | µ) m(ξ1:n | µ)
fµ (ξ) = 1sX (ξ) = ¾ sup .
s
µ(s) n
µ(ξ1:n ) n µ(ξ1:n )

62
 2.3. Infinite sequences

The function ξ 7→ fµ (ξ) is lower semicomputable with µξ fµ (ξ) ¶ 1, and

hence
+ +
dµ (ξ) > log f (ξ) > sup − log µ(ξ1:n ) − H(ξ1:n | µ) .


n
+
The proof of < reproduces the proof of Theorem 5.2 of [18]. Let us
abbreviate:
1 y (ξ) = 1 y X (ξ).
Since dd(ξ)e only takes integer values and is lower semicomputable, there
are computable sequences yi ∈ Σ∗ and ki ∈ N with
1X k
2dd(ξ)e = sup 2ki 1 yi (ξ) ¾ 2 i 1 yi (ξ)
i 2 i

with the property that if i < j and 1 yi (ξ) = 1 y j (ξ) = 1 then ki < k j . The
inequality follows from the fact that for P any finite sequence n1 < n2 < . . . ,
γ( =
P nj nj ki
j 2 ¶ 2 max j 2 . The function y) yi = y 2 is lower semicomputable.
With it, we have
X X
2ki 1 yi (ξ) = 1 y (ξ)γ( y).
i y∈Σ∗

∗
Since µ2dde ¶ 2, we have µ( y)γ( y) ¶ 2, hence µ( y)γ( y) < m( y), that is
P
y
∗
γ( y) < m( y)/µ( y). It follows that
∗ m( y) m(ξ1:n )
2d(ξ) < sup 1 y (ξ) = sup .
y∈Σ∗ µ( y) n µ(ξ1:n )
+
Taking logarithms, we obtain the < part of the theorem.

2.3.7 UNIVERSAL SEMIMEASURE , ALGORITHMIC PROBABILITY

Universal semimeasures exist over infinite sequences just as in the discrete

space.
Definition 2.3.18. A function µ : Σ∗ → R+ is a semimeasure if it satisfies
X
µ(x) ¾ µ(xs),
s∈Σ

µ(Λ) ¶ 1.
If it is lower semicomputable we will call the semimeasure constructive. ù

63
2. RANDOMNESS

These requirements imply, just as for measures, the following generaliza-

tion to an arbitrary prefix-free set A ⊆ Σ∗ :
X
µ(x) ¶ 1.
x∈A

Standard technique gives the following:

Theorem 2.3.5 (Universal semimeasure). There is a universal constructive

semimeasure µ, that is a constructive semimeasure with the property that for
every other constructive semimeasure ν there is a constant cν > 0 with µ(x) ¾
cν ν(x).

Definition 2.3.19. Let us fix a universal constructive semimeasure over X

and denote it by M (x). ù

There is a graphical way to represent a universal semimeasure, via

monotonic Turing machines, which can be viewed a generalization of self-
delimiting machines.

Definition 2.3.20. A Turing machine T is monotonic if it has no input tape

or output tape, but can ask repeatedly for input symbols and can emit repeat-
edly output symbols. The input is assumed to be an infinite binary string, but
it is not assumed that T will ask for all of its symbols, even in infinite time. If
the finite or infinite input string is p = p1 p2 · · · , the (finite or infinite) output
string is written as T (p) ∈ Σ∗ ∪ ΣN . ù

Just as we can generalize self-delimiting machines to obtain the notion

of monotonic machines, we can generalize algorithmic probability, defined
for a discrete space, to a version defined for infinite sequences. Imagine
the self-delimiting Turing machine T computing the function T (p) as in the
definition, and assume that its input symbols come from coin-tossing.

Definition 2.3.21 (Algorithmic probability over infinite sequences). For a

string x ∈ Σ∗ , let PT (x) be the probability of T (π) w x, when π = π1 π2 · · ·
is the infinite coin-tossing sequence. ù

We can compute
X
PT (x) = 2−l(p) ,
T (p)wx,p minimal

64
 2.3. Infinite sequences

where “minimal” means that no prefix p0 of p gives T (p0 ) w x. This expres-

sion shows that PT (x) is lower semicomputable. It is also easy to check that
it is a semimeasure, so we have a constructive semimeasure. The following
theorem is obtained using a standard construction:

Theorem 2.3.6. Every constructive semimeasure µ(x) can be represented as

µ(x) = PT (x) with the help of an appropriate a monotonic machine T .

This theorem justifies the following.

Definition 2.3.22. From now on we will call the universal semimeasure

M (x) also the algorithmic probability. A monotonic Turing machine T giving
PT (x) = M (x) will be called an optimal machine. ù

2.3.8 RANDOMNESS VIA ALGORITHMIC PROBABILITY

Fix a computable measure P(x) over the space X . Just as in the discrete case,
the universal semimeasure gives rise to a universal Martin-Löf test.

Definition 2.3.23. Denote

M (ξ1:n )
d0P (ξ) = log sup .
n P(ξ1:n )
ù

Theorem 2.3.7. The function d0P (ξ) is a universal Martin-Löf test.

Proof. Lower semicomputability follows from the form of definition. By stan-

dard methods, produce a list of finite strings y1 , y2 , . . . with:
(a) M ( yi )/P( yi ) > 2m
(b) The cylinder sets yi X cover of Gm , and are disjoint (thus the set
{ y1 , y2 , . . . } is prefix-free).
We have
X X
P(Gm ) = P( yi ) < 2−m M ( yi ) ¶ 2−m .
i i

The above theorem suggests that − log M (x) should also be considered a
kind of description complexity:

65
2. RANDOMNESS

Definition 2.3.24. Denote

K M (x) = − log M (x).

How does K M (x) compare to H(x)?

Proposition 2.3.14. We have the bounds

+ +
K M (x) < H(x) < K M (x) + H(l(x)).
+
Proof. To prove K M (x) < H(x) define the lower semicomputable function
X
µ(x) = sup m(x y),
S y∈S

By its definition this is a constructive semimeasure and it is ¾ m(x). This

∗ +
implies M (x) > m(x), and thus K M (x) < H(x).
For the other direction, define the following lower semicomputable funct-
ion ν over Σ∗ :

ν(x) = m(l(x)) · M (x).

To show that it is a semimeasure compute:

X X X X
ν(x) = m(n) M (x) ¶ m(n) ¶ 1.
x∈Σ∗ n x∈Σn n

∗ +
It follows that m(x) > ν(x) and hence H(x) < K M (x) + H(l(x)).
Just as in the discrete case, the universality of M (x) implies
+
K M (x) < − log P(x) + H(P).

On the other hand,

sup − log P(ξ1:n ) − K M (ξ1:n )

n

is a randomness test. So for random ξ, the value K M (ξ1:n ) remains within a

constant of − log P(x): it does not oscillate much.

66
3 INFORMATION

3.1 INFORMATION-THEORETIC RELATIONS

In this subsection, we use some material from [23].

3.1.1 THE INFORMATION -THEORETIC IDENTITY

Information-theoretic entropy is related to complexity in both a formal and

a more substantial way.

CLASSICAL ENTROPY AND CLASSICAL CODING

Entropy is used in information theory in order to characterize the compress-

ibility of a statistical source.

Definition 3.1.1. The entropy of a discrete probability distribution P is de-

fined as X
H (P) = − P(x) log P(x).
x

A simple argument shows that this quantity is non-negative. It is instruc-

tive to recall a more general inequality, taken from [13], implying it:

Theorem 3.1.1. Let ai , bi > 0, a = i ai , b = i bi . Then we have

P P

X ai a
ai log ¾ a log , (3.1.1)
i
bi b

only if ai /bi is constant. In particular, if ai = 1 and

P P
with equality P i i bi ¶ 1
a
then we have i ai log bi ¾ 0.
i

67
3. INFORMATION

Proof. Exercise, an application of Jensen’s inequality to the concave function

log x.
P
If P is a probability distribution and Q(x) ¾ 0, x Q(x) ¶ 1 then this
inequality implies X
H(P) ¶ P(x) log Q(x).
x

We can interpret this inequality as follows. We have seen in Subsection 1.6.3

that if x 7→ z x is a prefix code, mapping the
P objects x into binary strings z x
−l(z x )
that are not prefixes of each other then x 2 ¶ 1. Thus, substituting
Q(x) = 2−l(z x ) above we obtain Shannon’s result:

Theorem 3.1.2 (Shannon). If x 7→ z x is a prefix code then for its expected

codeword length we have the lower bound:
X
P(x)l(z x ) ¾ H(P).
x

ENTROPY AS EXPECTED COMPLEXITY

Applying Shannon’s theorem 3.1.2 to the code obtained by taking z x as the

shortest self-delimiting description of x, we obtain the inequality
X
P(x)H(x) ¾ H (P)
x

∗
On the other hand, since m(x) is a universal semimeasure, we have m(x) >
+
P(x): more precisely, H(x) < − log P(x) + H(P), leading to the following
theorem.

Theorem 3.1.3. For a computable distribution P we have

+
X
H (P) ¶ P(x)H(x) < H (P) + H(P). (3.1.2)
x

Note that H (P) is the entropy of the distribution P while H(P) is just
the description complexity of the function x 7→ P(x), it has nothing to do
directly with the magnitudes of the values P(x) for each x as real numbers.
These two relations give
X
+
H(P) = P(x)H(x) − H(P),
x

68
 3.1. Information-theoretic relations

the entropy is within an additive constant equal to the expected complexity.

Our intended interpretation of H(x) as information content of the individual
object x is thus supported by a tight quantitative relationship to Shannon’s
statistical concept.

IDENTITIES, INEQUALITIES

The statistical entropy obeys meaningful identities which immediately follow

from the definition of conditional entropy.

Definition 3.1.2. Let X and Y be two discrete random variables with a joint
distribution. The conditional entropy of Y with respect to X is defined as
X X
H (Y | X ) = − P[X = x] P[Y = y | X = x] log P[Y = y | X = x].
x y

The joint entropy of X and Y is defined as the entropy of the pair (X , Y ). ù

The following additivity property is then verifiable by simple calculation:

H (X , Y ) = H (X ) + H (Y | X ). (3.1.3)

Its meaning is that the amount of information in the pair (X , Y ) is equal to

the amount of information in X plus the amount of our residual uncertainty
about Y once the value of X is known.
Though intuitively expected, the following identity needs proof:

H (Y | X ) ¶ H (Y ).

We will prove it after we define the difference below.

Definition 3.1.3. The information in X about Y is defined by

I (X : Y ) = H (Y ) − H (Y | X ).

This is the amount by which our knowledge of X decreases our uncer-

tainty about Y .
The identity below comes from equation (3.1.3).

I (X : Y ) = I (Y : X ) = H (X ) + H (Y ) − H (X , Y )

69
3. INFORMATION

The meaning of the symmetry relation is that the quantity of information in

Y about X is equal to the quantity of information in X about Y . Despite
its simple derivation, this fact is not very intuitive; especially since only the
quantities are equal, the actual contents are generally not (see [22]). We
also call I (X : Y ) the mutual information of X and Y . We can also write
X P[X = x, Y = y]
I (X : Y ) = P[X = x, Y = y] log . (3.1.4)
x, y
P[X = x]P[Y = y]
Proposition 3.1.1. The information I (X : Y ) is nonnegative and is 0 only if
X , Y are independent.
Proof. Apply Theorem 3.1.1 to (3.1.4). This is 0 only if P[X = x, Y = y] =
P[X = x]P[Y = y] for all x, y.

THE ALGORITHMIC ADDITION THEOREM

For the notion of algorithmic entropy H(x) defined in Section 1.6, it is a

serious test of fitness to ask whether it has an additivity property analogous to
the identity (3.1.3). Such an identity would express some deeper relationship
than the trivial identity (3.1.3) since the conditional complexity is defined
using the notion of conditional computation, and not by an algebraic formula
involving the unconditional definition.
A literal translation of the identity (3.1.3) turns out to be true for the
function H(x) as well as for Kolmogorov’s complexity K(x) with good ap-
proximation. But the exact additivity property of algorithmic entropy is sub-
tler.
Theorem 3.1.4 (Addition). We have
+
H(x, y) = H(x) + H( y | x, H(x)). (3.1.5)
+
Proof. We first prove <. Let F be the following s.d. interpreter. The machine
computing F (p) tries to parse p into p = uv so that T (u) is defined, then
outputs the pair (T (u), T (v, 〈T (u), l(u)〉)). If u is a shortest description of x
and v a shortest description of y given the pair (x, H(x)) then the output of
F is the pair (x, y).
+
Now we prove >. Let c be Pa constant (existing in force of Lemma 1.7.2)
−H(x, y)
such that for all x we have y 2 ¶ 2−H(x)+c . Let us define the semi-
computable function f (z, y) by
f ((x, k), y) = 2k−H(x, y)−c .

70
 3.1. Information-theoretic relations

Then for k = H(x) we have f (z, y) ¶ 1, hence the generaliza-

P
y
+
tion (1.6.13) of the coding theorem implies H( y | x, H(x)) < H(x, y) −
H(x).

Recall the definition of x ∗ as the first shortest description of x.

Corollary 3.1.2. We have

+
H(x ∗ , y) = H(x, y). (3.1.6)

The Addition Theorem implies

+ +
H(x, y) < H(x) + H( y | x) < H(x) + H( y)

while these relations do not hold for Kolmogorov’s complexity K(x), as

pointed out in the discussion of Theorem 1.4.2. For an ordinary interpreter,
extra information is needed to separate the descriptions of x and y. The self-
delimiting interpreter accepts only a description that provides the neccesary
information to determine its end.
The term H(x) cannot be omitted from the condition in H( y | x, H(x))
in the identity (3.1.5). Indeed, it follows from equation (1.7.2) that with
+
y = H(x), we have H(x, y) − H(x) = 0. But H(H(x) | x) can be quite large,
as shown in the following theorem given here without proof.

Theorem 3.1.5 (see [17]). There is a constant c such that for all n there is a
binary string x of length n with

H(H(x) | x) ¾ log n − log log n − c.

We will prove this theorem in Section 3.3. For the study of information
relations, let us introduce yet another notation.

Definition 3.1.4. For any functions f , g, h with values in S, let

f  g mod h

mean that there is a constant c such that H( f (x) | g(x), h(x)) ¶ c for all
x. We will omit modh if h is not in the condition. The sign  will mean
inequality in both directions. ù

71
3. INFORMATION

The meaning of x  y is that the objects x and y hold essentially the

same information. Theorem 1.4.1 implies that if f  g then we can replace
f by g wherever it occurs as an argument of the functions H or K. As an
illustration of this definition, let us prove the relation
x ∗  〈x, H(x)〉
+
which implies H( y | x, H(x)) = H( y | x ∗ ), and hence
+
H(x, y) − H(x) = H( y | x ∗ ). (3.1.7)
The direction  holds because we can compute x = T (x ∗ ) and H(x) = l(x ∗ )
from x ∗ . On the other hand, if we have x and H(x) then we can enumer-
ate the set E(x) of all shortest descriptions of x. By the corollary of Theo-
rem 1.7.1, the number of elements of this set is bounded by a constant. The
+
estimate (1.6.4) implies H(x ∗ | x, H(x)) = 0.
Though the string x ∗ and the pair (x, H(x)) contain the same informa-
tion, they are not equivalent under some stronger criteria on the decoder. To
obtain x from the shortest description may take extremely long time. But
even if this time t is acceptable, to find any description of length H(x) for x
might require about t2H(x) steps of search.
The Addition Theorem can be made formally analogous to its
information-theoretic couterpart using Chaitin’s notational trick.
Definition 3.1.5. Let H ∗ ( y | x) = H( y | x ∗ ). ù
Of course, H ∗ (x) = H(x). Now we can formulate the relation (3.1.7) as
+
H ∗ (x, y) = H ∗ (x) + H ∗ ( y | x).
However, we prefer to use the function H( y | x) since the function H ∗ ( y |
x) is not semicomputable. Indeed, if there was a function h(x, y) upper
+
semicomputable in 〈x, y〉 such that H(x, y) − H(x) = h(x, y) then by the
+
argument of the proof of the Addition Theorem, we would have H( y | x) <
h(x, y) which is not true, according to remark after the proof of the Addition
Theorem.
Definition 3.1.6. We define the algorithmic mutual information with the
same formula as classical information:
I(x : y) = H( y) − H( y | x).
ù

72
 3.1. Information-theoretic relations

Since the quantity I(x : y) is not quite equal to H(x) − H(x | y) (in fact,
Levin showed (see [17]) that they are not even asymptotically equal), we
prefer to define information also by a symmetrical formula.

Definition 3.1.7. The mutual information of two objects x and y is

I ∗ (x : y) = H(x) + H( y) − H(x, y).

The Addition Theorem implies the identity

+ +
I ∗ (x : y) = I(x ∗ : y) = I( y ∗ : x).

Let us show that classical information is indeed the expected value of algo-
rithmic information, for a computable probability distribution p:

Lemma 3.1.3. Given a computable joint probability mass distribution p(x, y)

over (x, y) we have
+
XX
I (X : Y ) − H(p) < p(x, y)I ∗ (x : y) (3.1.8)
x y
+
< I (X : Y ) + 2H(p),

where H(p) is the length of the shortest prefix-free program that computes
p(x, y) from input (x, y).

Proof. We have
X X
+
p(x, y)I ∗ (x : y) = p(x, y)[H(x) + H( y) − H(x, y)].
x, y x, y

p(x, y) = p1 (x) and p(x, y) = p2 ( y) to obtain

P P
Define y x
X X X X
+
p(x, y)I(x : y) = p1 (x)H(x) + p2 ( y)H( y) − p(x, y)H(x, y).
x, y x y x, y

The distributions pi (i = 1, 2) are computable. We have seen in (3.1.2) that

+ P +
H (q) < x q(x)H(x) < H (q) + H(q).
+ + +
Hence, H (pi ) < x pi (x)H(x) < H (pi ) + H(pi ). (i = 1, 2), and H (p) <
P
+

x, y p(x, y)H(x, y) < H (p) + H(p). On the other hand, the probabilistic
P

73
3. INFORMATION

mutual information is expressed in the entropies by I (X : Y ) = H (p1 ) +

+
H (p2 ) − H (p). By construction of the qi ’s above, we have H(p1 ), H(p2 ) <
H(p). Since the complexities are positive, substitution establishes the lemma.

Remark 3.1.4. The information I ∗ (x : y) can be written as

+ m(x, y)
I ∗ (x : y) = log .
m(x)m( y)

Formally, I ∗ (x : y) looks like dm×m ((x, y) | m × m) with the function d· (·)

introduced in (2.2.2). Thus, it looks like I ∗ (x : y) measures the deficiency
of randomness with respect to the distribution m × m. The latter distribu-
tion expresses our “hypothesis” that x, y are “independent” from each other.
There is a serious technical problem with this interpretation: the function
d P (x) was only defined for computable measures P. Though the definition
can be extended, it is not clear at all that the expression d P (x) will also play
the role of universal test for arbitrary non-computable distributions. Levin’s
theory culminating in [32] develops this idea, and we return to it in later
parts of these notes. ù
Let us examine the size of the defects of naive additivity and information
symmetry.

Corollary 3.1.5. For the defect of additivity, we have

+ +
H(x) + H( y | x) − H(x, y) = I(H(x) : y | x) < H(H(x) | x). (3.1.9)

For information asymmetry, we have

+
I(x : y) − I( y : x) = I(H( y) : x | y) − I(H(x) : y | x). (3.1.10)

Proof. Immediate from the addition theorem.

Theorem 3.1.6 (Levin). The information I(x : y) is not even asymptotically

symmetric.

Proof. For some constant c, assume |x| = n and H(H(x) | x) ¾ log n −

log log n − c. Consider x, x ∗ and H(x). Then we can check immediately
the relations
+ +
I(x : H(x)) < 3 log log n < log n − log log n − c < I(x ∗ : H(x)).

74
 3.1. Information-theoretic relations

and
+
I(H(x) : x) = I(H(x) : x ∗ ).
It follows that symmetry breaks down in an exponentially great measure
either on the pair (x, H(x)) or the pair (x ∗ , H(x)).

DATA PROCESSING

The following useful identity is also classical, and is called the data processing
identity:
I (Z : (X , Y )) = I (Z : X ) + I (Z : Y | X ). (3.1.11)
Let us see what corresponds to this for algorithmic information.
Here is the correct conditional version of the addition theorem:

Theorem 3.1.7. We have

+
H(x, y | z) = H(x | z) + H( y | x, H(x | z), z). (3.1.12)

Proof. The same as for the unconditional version.

Remark 3.1.6. The reader may have guessed the inequality
+
H(x, y | z) = H(x | z) + H( y | x, H(x), z),

but it is incorrect: taking z = x, y = H(x), the left-hand side equals H(x ∗ |

+
x), and the right-hand side equals H(x | x) + H(H(x) | x ∗ , x) = 0. ù
The following inequality is a useful tool. It shows that conditional com-
plexity is a kind of one-sided distance, and it is the algorithmic analogue of
the well-known (and not difficult classical inequality)

H (X | Y ) ¶ H (Z | Y ) + H (X | Z).

Theorem 3.1.8 (Simple triangle inequality). We have

+ +
H(z | x) < H( y, z | x) < H( y | x) + H(z | y). (3.1.13)

Proof. According to (3.1.12), we have

+
H( y, z | x) = H( y | x) + H(z | y, H( y | x), x).
+
The left-hand side is > H(z | x), the second term of the right-hand side is
+
< H(z | y).

75
3. INFORMATION

Equation (3.1.12) justifies the following.

Definition 3.1.8.

I ∗ (x : y | z) = H(x | z) + H( y | z) − H(x, y | z). (3.1.14)

Then we have
+
I ∗ (x : y | z) = H( y | z) − H( y | x, H(x | z), z)
+
= H(x | z) − H(x | y, H( y | z), z).

Theorem 3.1.9 (Algorithmic data processing identity). We have

+
I ∗ (z : (x, y)) = I ∗ (z : x) + I ∗ (z : y | x ∗ ). (3.1.15)

Proof. We have
+
I ∗ (z : (x, y)) = H(x, y) + H(z) − H(x, y, z)
+
I ∗ (z : x) = H(x) + H(z) − H(x, z)
+
I ∗ (z : (x, y)) − I ∗ (z : x) = H(x, y) + H(x, z) − H(x, y, z) − H(x)
+
= H( y | x ∗ ) + H(z | x ∗ ) − H( y, z | x ∗ )
+
= I ∗ (z : y | x ∗ ),

where we used additivity and the definition (3.1.14) repeatedly.

Corollary 3.1.7. The information I ∗ is monotonic:

+
I ∗ (x : ( y, z)) > I ∗ (x : y). (3.1.16)

The following theorem is also sometimes useful.

Theorem 3.1.10 (Starry triangle inequality). For all x, y, z, we have

+ +
H(x | y ∗ ) < H(x, z | y ∗ ) < H(z | y ∗ ) + H(x | z ∗ ). (3.1.17)

76
 3.1. Information-theoretic relations

Proof. Using the algorithmic data-processing identity (3.1.15) and the defi-
nitions, we have

H(z) − H(z | y ∗ ) + H(x | z ∗ ) − H(x | y, H( y | z ∗ ), z ∗ )

+ +
= I ∗ ( y : z) + I ∗ ( y : x | z ∗ ) = I ∗ ( y : (x, z))
+
= H(x, z) − H(x, z | y ∗ ),

which gives, after cancelling H(z) + H(x | z ∗ ) on the left-hand side with
H(x, z) on the right-hand side, to which it is equal according to additivity,
and changing sign:
+
H(x, z | y ∗ ) = H(z | y ∗ ) + H(x | y, H( y | z ∗ ), z ∗ ).

From here, we proceed as in the proof of the simple triangle inequal-

ity (3.1.13)

3.1.2 INFORMATION NON - INCREASE

In this subsection, we somewhat follow the exposition of [23].

The classical data processing identity has a simple but important appli-
cation. Suppose that the random variables Z, X , Y form a Markov chain in
this order. Then I(Z : Y | X ) = 0 and hence I(Z : (X , Y )) = I(Z : X ). In
words: all information in Z about Y is coming through X : a Markov transi-
tion from X to Y cannot increase the information about Z. Let us try to find
the algorithmic counterpart of this theorem.
Here, we rigorously show that this is the case in the algorithmic statistics
setting: the information in one object about another cannot be increased by
any deterministic algorithmic method by more than a constant. With added
randomization this holds with overwhelming probability. For more elaborate
versions see [30, 32].
Suppose we want to obtain information about a certain object x. It does
not seem to be a good policy to guess blindly. This is confirmed by the
following inequality. X ∗
m( y)2 I(x: y) < 1 (3.1.18)
y

which says that for each x, the expected value of 2 I(x: y) is small, even with
respect to the universal constructive semimeasure m( y). The proof is imme-

77
3. INFORMATION

diate if we recognize that by the Coding Theorem,

2−H( y|x)
2 I(x: y) =
∗

m( y)
and use the estimate (1.6.10).
We prove a strong version of the information non-increase law under
deterministic processing (later we need the attached corollary):

Theorem 3.1.11. Given x and z, let q be a program computing z from x ∗ .

Then we have
+
I ∗ ( y : z) < I ∗ ( y : x) + H(q).

Proof. By monotonicity (3.1.16) and the data processing identity (3.1.15):

+ +
I ∗ ( y : z) < I ∗ ( y : (x, z)) = I ∗ ( y : x) + I ∗ ( y : z | x ∗ ).

By definition of information and the definition of q:

+ +
I ∗ ( y : z | x ∗ ) < H(z | x ∗ ) < H(q).

Randomized computation can increase information only with negligible

probability. Suppose that z is obtained from x by some randomized compu-
tation. The probability p(z | x) of obtaining z from x is a semicomputable
distribution over the z’s. The information increase I ∗ (z : y) − I ∗ (x : y) satis-
fies the theorem below.

Theorem 3.1.12. For all x, y, z we have

∗
(z: y)−I ∗ (x: y) ∗
m(z | x ∗ )2 I < m(z | x ∗ , y, H( y | x ∗ )).
∗
Therefore it is upperbounded by m(z | x) < m(z | x ∗ ).

Remark 3.1.8. The theorem implies

X ∗ ∗
m(z | x ∗ )2 I (z: y)−I (x: y) < 1.
∗
(3.1.19)
z

This says that the m(· | x ∗ )-expectation of the exponential of the increase
is bounded by a constant. It follows that for example, the probability of an
∗
increase of mutual information by the amount d is < 2−d . ù

78
 3.2. The complexity of decidable and enumerable sets

Proof. We have, by the data processing inequality:

+ +
I ∗ ( y : (x, z)) − I ∗ ( y : x) = I ∗ ( y : z | x ∗ ) = H(z | x ∗ ) − H(z | y, H( y | x ∗ ), x ∗ ).
+
Hence, using also I ∗ ( y : (x, z)) > I ∗ ( y : z) by monotonicity:
+
I ∗ ( y : z) − I ∗ ( y : x) − H(z | x ∗ ) < −H(z | y, H( y | x ∗ ), x ∗ ).

Putting both sides into the exponent gives the statement of the theorem.
Remark 3.1.9. The theorem on information non-increase and its proof look
similar to the theorems on randomness conservation. There is a formal con-
nection, via the observation made in Remark 3.1.4. Due to the difficulties
mentioned there, we will explore the connection only later in our notes. ù

3.2 THE COMPLEXITY OF DECIDABLE AND ENUMERABLE

SETS

If ω = ω(1)ω(2) · · · is an infinite binary sequence, then we may be interested

in how the complexity of the initial segments

ω(1 : n) = ω(1) · · · ω(n)

grows with n. We would guess that if ω is “random” then K(ω(1 : n)) is

approximately n; this question will be studied satisfactorily in later sections.
Here, we want to look at some other questions.
Can we determine whether the sequence ω is computable, by just looking
at the complexity of its initial segments? It is easy to see that if ω is comput-
+ +
able then K(ω(1 : n)) < log n. But of course, if K(ω(1 : n)) < log n then it is
not necessarily true yet that ω is computable. Maybe, we should put n into
+
the condition. If ω is computable then K(ω(1 : n) | n) < 0. Is it true that
+
K(ω(1 : n) | n) < 0 for all n then indeed ω is computable? Yes, but the proof
is quite difficult (see either its original, attributed to Albert Meyer in [36], or
in [59]). The suspicion arises that when measuring the complexity of start-
ing segments of infinite sequences, neither Kolmogorov complexity nor its
prefix version are the most natural choice. Other examples will support the
suspicion, so let us introduce, following Loveland’s paper [36], the following
variant.

79
3. INFORMATION

Definition 3.2.1. Let us say that a program p decides a string x =

(x(1), x(2), . . . ) on a Turing machine T up to n if for all i ∈ {1, . . . , n} we
have
T (p, i) = x(i).
Let us further define the decision complexity

K T (x; n) = min{ l(p) : p decides x on T up to n }.

As for the Kolmogorov complexity, an invariance theorem holds, and

there is an optimal machine T0 for this complexity. Again, we omit the index
T , assuming that such an optimal machine has been fixed.
The differences between K(x), K(x | n) and K(x; n) are interesting to
explore; intuitively, the important difference is that the program achieving
the decision complexity of x does not have to offer precise information about
the length of x. We clearly have
+ +
K(x | n) < K(x; n) < K(x).

Examples that each of these inequalities can be strict, are left to exercises.
Remark 3.2.1. If ω(1 : n) is a segment of an infinite sequence ω then we can
write
K(ω; n)
instead of K(ω(1 : n); n) without confusion, since there is only one way to
understand this expression. ù
Decision complexity offers an easy characterization of decidable infinite
sequences.

Theorem 3.2.1. Let ω = (ω(1), ω(2), . . . ) be an infinite sequence. Then ω is

decidable if and only if
+
K(ω; n) < 0. (3.2.1)

Proof. If ω is decidable then (3.2.1) is obviously true. Suppose now

that (3.2.1) holds: there is a constant c such that for all n we have K(ω(1 :
n); n) < c. Then there is a program p of length ¶ c with the property that for
infinitely many n, decides x on the optimal machine T0 up to n. Then for all
i, we have T0 (p, i) = ω(i), showing that ω is decidable.

80
 3.2. The complexity of decidable and enumerable sets

Let us use now the new tool to prove a somewhat more surprising result.
Let ω be a 0-1 sequence that is the indicator function of a recursively enu-
merable set. In other words, the set { i : ω(i) = 1 } is recursively enumerable.
As we know such sequences are not always decidable, so K(ω(1 : n); n) is
not going to be bounded. How fast can it grow? The following theorem gives
exact answer, showing that it grows only logarithmically.

Theorem 3.2.2 (Barzdin, see [2]).

(a) Let ω be the indicator function of a recursively enumerable set E. Then we
have
+
K(ω; n) < log n.

(b) The set E can be chosen such that for all n we have K(ω; n) ¾ log n.
0
Proof. Let usP prove (a) first. Let n be the first power of 2 larger than n.
Let k(n) = i¶n0 ω(i). For a constant d, let p = p(n, d) be a program that
contains an initial segment q of size d followed by a the number k(n) in
binary notation, and padded to dlog ne + d with 0’s. The program q is self-
delimiting, so it sees where k(n) begins.
The machine T0 (p, i) works as follows, under the control of q. From the
length of the whole p, it determines n0 . It begins to enumerate ω(1 : n0 ) until
it found k(n) 1’s. Then it knows the whole ω(1 : n0 ), so it outputs ω(i).
Let us prove now (b). Let us list all possible programs q for the machine
T0 as q = 1, 2, . . . . Let ω(q) = 1 if and only if T0 (q, q) = 0. The sequence ω
is obviously the indicator function of a recursively enumerable set. To show
that ω has the desired property, assume that for some n there is a p with
T0 (p, i) = ω(i) for all i ¶ n. Then p > n since ω(p) is defined to be different
from T0 (p, p). It follows that l(p) ¾ log n.

Decision complexity has been especially convenient for the above theo-
rem. Neither K(ω(1 : n)) nor K(ω(1 : n) | n) would have been suitable
to formulate such a sharp result. To analyze the phenomenon further, we
introduce some more concepts.

Definition 3.2.2. Let us denote, for this section, by E the set of those binary
strings p on which the optimal prefix machine T halts:

E t = { p : T (p) halts in < t steps },

(3.2.2)
E = E ∞.

81
3. INFORMATION

Let
χ = χE (3.2.3)
be the infinite sequence that is the indicator function of the set E, when the
latter is viewed as a set of numbers. ù

It is easy to see that the set E is complete among recursively enumerable

sets with respect to many-one reduction. The above theorems show that
though it contains an infinite amount of information, this information is not
stored in the sequence χ densely at all: there are at most log n bits of it in the
segment χ(1 : n). There is an infinite sequence, though, in which the same
information is stored much more densely: as we will see later, maximally
densely.

Definition 3.2.3 (Chaitin’s Omega). Let

X
Ωt = 2−l(p) ,
p∈E t (3.2.4)
Ω=Ω . ∞

Let Ω(1 : n) be the sequence of the first n binary digits in the expansion
of Ω, and let it also denote the binary number 0.Ω(1) · · · Ω(n). Then we have

Ω(1 : n) < Ω < Ω(1 : n) + 2−n .

Theorem 3.2.3. The sequences Ω and χ are Turing-equivalent.

Proof. Let us show first that given Ω as an oracle, a Turing machine can
compute χ. Suppose that we want to know for a given string p of length k
whether χ(p) = 1 that is whether T (p) halts. Let t(n) be the first t for which
Ω t > Ω(1 : n). If a program p of length n is not in E t(n) then it is not in E
at all, since 2−l(p) = 2n < Ω − Ω t . It follows that χ(1 : 2n ) can be computed
from Ω(1 : n).
To show that Ω can be computed from E, let us define the recursively
enumerable set
E 0 = { r : r rational, Ω > r }.
The set E 0 is reducible to E since the latter is complete among recursively
enumerable sets with respect to many-one reduction. On the other hand, Ω
is obviously computable from E 0 .

82
 3.3. The complexity of complexity

The following theorem shows that Ω stores the information more densely.
+
Theorem 3.2.4. We have H(Ω(1 : n)) > n.

Proof. Let p1 be a self-delimiting program outputting Ω(1 : n). Recall the

definition of the sets E t in (3.2.2) and the numbers Ω t in (3.2.4). Let Ω1
denote the real number whose binary digits after 0 are given by Ω(1 : n),
and let t 1 be the first t with Ω t > Ω1 . Let x 1 be the first string x such that
T (p) 6= x for any p ∈ E t 1 .
We have H(x 1 ) ¾ n. On the other hand, we just computed x 1 from p1 , so
+ + +
H(x 1 | p1 ) < 0. We found n ¶ H(x 1 ) < H(p1 ) + H(x 1 | p1 ) < H(p1 ).

3.3 THE COMPLEXITY OF COMPLEXITY

3.3.1 COMPLEXITY IS SOMETIMES COMPLEX

This section is devoted to a quantitative estimation of the uncomputabil-

ity of the complexity function K(x). Actually, we will work with the pre-
fix complexity H(x), but the results would be very similar for K(x). The
first result shows that the value H(x) is not only not computable from
x, but its conditional complexity H(H(x) | x) given x is sometimes quite
large. How large can it be expected to be? Certainly not much larger than
log H(x) + 2 log log H(x), since we can describe any value H(x) using this
many bits. But it can come close to this, as shown by Theorem 3.1.5. This
theorem says that for all n, there exists x of length n with
+
H(H(x) | x) > log n − log log n. (3.3.1)

Proof of Theorem 3.1.5. Let U(p, x) be the optimal self-delimiting machine

for which H( y | x) = minU(p,x)= y l(p). Let s ¶ log n be such that if l(x) = n
then a p of length x can be found for which U(p, x) = H(x). We will show
+
s > log n − log log n. (3.3.2)

Let us say that p ∈ {0, 1}s is suitable for x ∈ {0, 1}n if there exists a
k = U(p, x) and a q ∈ {0, 1}k with U(p, Λ) = x. Thus, p is suitable for x
if it produces the length of some program of x, not necessarily a shortest
program.

83
3. INFORMATION

Let Mi denote the set of those x of length n for which there exist at least
i different suitable p of length s. We will examine the sequence

{0, 1}n = M0 ⊇ M1 ⊇ . . . ⊇ M j ⊇ M j+1 = ;,

where M j 6= ;. It is clear that 2s ¾ j. To lowerbound j, we will show that the

sets Mi decrease only slowly:
+
log |Mi | < log |Mi+1 | + 4 log n. (3.3.3)

We can assume
log |Mi \ Mi+1 | ¾ log |Mi | − 1, (3.3.4)
otherwise (3.3.3) is trivial. We will write a program that finds an element x
of Mi \ Mi+1 with the property H(x) ¾ log |Mi | − 1. The program works as
follows.
• It finds i, n with the help of descriptions of length log n + 2 log log n,
and s with the help of a description of length 2 log log n.
• It finds |Mi+1 | with the help of a description of length log |Mi+1 |+log n+
2 log log n.
• From these data, it determines the set Mi+1 , and then begins to enu-
merate the set Mi \ Mi+1 as x 1 , x 2 , . . . . For each of these elements x r , it
knows that there are exactly i programs suitable for x r , find all those,
and find the shortest program for x produced by these. Therefore it
can compute H(x r ).
• According to the assumption (3.3.4), there is an x r with H(x r ) ¾
log |Mi | − 1. The program outputs the first such x r .
+
The construction of the program shows that its length is < log |Mi+1 |+4 log n,
hence for the x we found
+
log |Mi | − 1 ¶ H(x) < log |Mi+1 | + 4 log n,

which proves (3.3.3). This implies j ¾ n/(4 log n), and hence (3.3.2).

3.3.2 COMPLEXITY IS RARELY COMPLEX

Let
f (x) = H(H(x) | x).

84
 3.3. The complexity of complexity

We have seen that f (x) is sometimes large. Here, we will show that the se-
quences x for which this happens are rare. Recall that we defined χ in (3.2.3)
as the indicator sequence of the halting problem.

Theorem 3.3.1. We have

+
I(χ : x) > f (x) − 2.4 log f (x).

In view of the inequality (3.1.18), this shows that such sequences are
rare, even in terms of the universal probability, so they are certainly rare if
we measure them by any computable distribution. So, we may even call such
sequences “exotic”.
In the proof, we start by showing that the sequences in question are rare.
Then the theorem will follow when we make use of the fact that f (x) is
computable from χ.
We need a lemma about the approximation of one measure by another
from below.

Lemma 3.3.1. For any semimeasure ν P and measure µ ¶ ν, let Sm = { x :

2m µ(x) ¶ ν(x) }. Then µ(Sm ) ¶ 2−m . If x (ν(x) − µ(x)) < 2−n then ν(S1 ) <
2−n+1 .

The proof is straightforward.

Let
X (n) = { x : f (x) = n }.

Lemma 3.3.2. We have

∗
m(X (n)) < n1.2 2−n .

Proof. Using the definition of E t in (3.2.2), let

X
m t (x) = { p : T (p) = x in < t steps },
H t (x) = − log m t (x).

Then according to the definition (1.6.14) we have m(x) = m∞ (x), and

+
H(x) = H ∞ (x). Let

t(k) = min{ t : Ω(1 : k) < Ω t },

µ = m t(k) .

85
3. INFORMATION

+
Clearly, H t(k) (x) > H(x). Let us show that H t(k) (x) is a good approximation
for H(x) for most x. Let
Y (k) = { x : H(x) ¶ H t(k) (x) − 1 }.
/ Y (k) we have
By definition, for x ∈
+
|H t(k) (x) − H(x)| = 0.
On the other hand, applying Lemma 3.3.1 with µ = m t(k) , ν = m, we obtain
m(Y (k)) < 2−k+1 . (3.3.5)
Note that
+
H(H t(k) (x) | x, Ω(1 : k)) = 0,
/ Y (k) we have
therefore for x ∈
+
H(H(x) | x, Ω(1 : k)) = 0,
+
H(H(x) | x) < k + 1.2 log k.
+
If n = k + 1.2 log k then k = n − 1.2 log n, and hence, if x ∈
/ Y (n − 1.2 log n)
+
then H(H(x) | x) < n. Thus, there is a constant c such that
X (n) ⊆ Y (n − 1.2 log n − c).
Using (3.3.5) this gives the statement of the lemma.
Proof of Theorem 3.3.1. Since f (x) is computable from Ω, the function
ν(x) = m(x)2 f (x) ( f (x))−2.4
is computable from Ω. Let us show that it is a semimeasure (within a multi-
plicative constant). Indeed, using the above lemma:
X X X X X ∗
ν(x) = ν(x) = 2k k−2.4 m(X (k)) = k−1.2 < 1.
x k x∈X (k) k k

∗
Since m(· | Ω) is the universal semimeasure relative to Ω we find m(x | Ω) >
ν(x), hence
+
H(x | Ω) < − log ν(x) = H(x) − f (x) + 2.4 log f (x),
+
I(Ω : x) > f (x) − 2.4 log f (x).
Since Ω is equivalent to χ, the proof is complete.

86
4 GENERALIZATIONS

4.1 CONTINUOUS SPACES , NONCOMPUTABLE MEASURES

This section starts the consideration of randomness in continuous spaces and

randomness with respect to noncomputable measures.

4.1.1 INTRODUCTION
The algorithmic theory of randomness is well developed when the underly-
ing space is the set of finite or infinite sequences and the underlying prob-
ability distribution is the uniform distribution or a computable distribution.
These restrictions seem artificial. Some progress has been made to extend
the theory to arbitrary Bernoulli distributions by Martin-Löf in [37], and to
arbitrary distributions, by Levin in [29, 31, 32]. The paper [25] by Hertling
and Weihrauch also works in general spaces, but it is restricted to comput-
able measures. Similarly, Asarin’s thesis [1] defines randomness for sample
paths of the Brownian motion: a fixed random process with computable dis-
tribution.
The exposition here has been inspired mainly by Levin’s early paper [31]
(and the much more elaborate [32] that uses different definitions): let us
summarize part of the content of [31]. The notion of a constructive topolog-
ical space X and the space of measures over X is introduced. Then the paper
defines the notion of a uniform test.R Each test is a lower semicomputable
function (µ, x) 7→ fµ (x), satisfying fµ (x)µ(d x) ¶ 1 for each measure µ.
There are also some additional conditions. The main claims are the follow-
ing.
(a) There is a universal test tµ (x), a test such that for each other test f there
is a constant c > 0 with fµ (x) ¶ c · tµ (x). The deficiency of randomness is
defined as dµ (x) = log tµ (x).

87
4. GENERALIZATIONS

(b) The universal test has some strong properties of “randomness conserva-
tion”: these say, essentially, that a computable mapping or a computable
randomized transition does not decrease randomness.
(c) There is a measure M with the property that for every outcome x we
have t M (x) ¶ 1. In the present work, we will call such measures neutral.
(d) Semimeasures (semi-additive measures) are introduced and it is shown
that there is a lower semicomputable semi-measure that is neutral (let
us assume that the M introduced above is lower semicomputable).
(e) Mutual information I(x : y) is defined with the help of (an appropriate
version of) Kolmogorov complexity, between outcomes x and y. It is
shown that I(x : y) is essentially equal to d M ×M (x, y). This interprets
mutual information as a kind of “deficiency of independence”.
This impressive theory leaves a number of issues unresolved:
1. The space of outcomes is restricted to be a compact topological space,
moreover, a particular compact space: the set of sequences over a finite
alphabet (or, implicitly in [32], a compactified infinite alphabet). How-
ever, a good deal of modern probability theory happens over spaces that
are not even locally compact: for example, in case of the Brownian mo-
tion, over the space of continuous functions.
2. The definition of a uniform randomness test includes some conditions
(different ones in [31] and in [32]) that seem somewhat arbitrary.
3. No simple expression is known for the general universal test in terms
of description complexity. Such expressions are nice to have if they are
available.
Here, we intend to carry out as much of Levin’s program as seems possible
after removing the restrictions. A number of questions remain open, but we
feel that they are worth to be at least formulated. A fairly large part of
the exposition is devoted to the necessary conceptual machinery. This will
also allow to carry further some other initiatives started in the works [37]
and [29]: the study of tests that test nonrandomness with respect to a whole
class of measures (like the Bernoulli measures).
Constructive analysis has been developed by several authors, converging
approximately on the same concepts, We will make use of a simplified version
of the theory introduced in [55]. As we have not found a constructive mea-
sure theory in the literature fitting our purposes, we will develop this theory
here, over (constructive) complete separable metric spaces. This generality

88
 4.1. Continuous spaces, noncomputable measures

is well supported by standard results in measure theoretical probability, and

is sufficient for constructivizing a large part of current probability theory.
The appendix recalls some of the needed topology, measure theory, con-
structive analysis and constructive measure theory. We also make use of the
notation introduced there.

4.1.2 UNIFORM TESTS

We first define tests of randomness with respect to an arbitrary measure.

Recall the definition of lower semicomputable real functions on a computable
metric space X.

Definition 4.1.1. Let us be given a computable complete metric space X =

(X , d, D, α). For an arbitrary measure µ ∈ M (X), a µ-test of randomness is a
µ-lower semicomputable function f :⊆ X → R+ with the property µ f ¶ 1.
We call an element x random with respect to µ if f (x) < ∞ for all µ-tests
f . But even among random elements, the size of the tests quantifies (non-
)randomness.
A uniform test of randomness is a lower semicomputable function f :
X × M (X) → R+ , written as (x, µ) 7→ fµ (x) such that fµ (·) is a µ-test for each
µ. ù

The condition µ f ¶ 1 guarantees that the probability of those outcomes

whose randomness is ¾ m is at most 1/m. The definition of tests is in
the spirit of Martin-Löf’s tests. The important difference is in the semicom-
putability condition: instead of restricting the measure µ to be computable,
we require the test to be lower semicomputable also in its argument µ.
The following result implies that every µ-test can be extended to a uni-
form test.

Theorem 4.1.1. Let φe , e = 1, 2, . . . be an enumeration of all lower semicom-

putable functions X × Y × M (X) → R+ , where Y is also a computable metric
space, and s : Y × M (X) → R a lower semicomputable function. There is a
recursive function e 7→ e0 with the property that
(a) For each e, the function φe0 (x, y, µ) is everywhere defined with
µ x φe0 (x, y, µ) ¶ s( y, µ).
(b) For each e, y, µ, if µ x φe (x, yµ) < s( y, µ) then φe0 (·, y, µ) = φe (·, y, µ).

89
4. GENERALIZATIONS

This theorem generalizes a theorem of Hoyrup and Rojas (in allowing a

lower semicomputable upper bound s( y, µ)).
Proof. By Proposition B.1.20, we can represent H e (x, y, µ) as a supremum
φe = supi he,i where he,i (x, y, µ) is a computable function of e, i monoton-
ically increasing in i. Similarly, we can represent s( y, µ) as a supremum
sup j s j ( y, µ) where s j ( y, µ) is a computable function monotonically increas-
ing in j. The integral µ x he,i (x, y, µ) is computable as a function of ( y, µ), in
particular it is upper semicomputable.
Define h0e,i, j (x, y, µ) as he,i (x, y, µ) for all j, y, µ with µ x he,i (x, y, µ) <
s j ( y, µ), and 0 otherwise. Since s j ( y, µ) is computable this definition makes
the function h0e,i, j (x, y, µ) lower semicomputable. The function h00e,i (x, y, µ) =
sup j h0e,i, j (x, y, µ) is then also lower semicomputable, with h00e,i (x, y, µ) ¶
he,i (x, y, µ), and µ x h0e,i (x, y, µ) ¶ s( y, µ). Also, h00e,i (x, y, µ) is monotonically
increasing in i. The function φe0 (x, y, µ) = supi h00e,i (x, y, µ) is then also lower
semicomputable, and by Fubini’s theorem we have µ x φe0 (x, y, µ) ¶ s( y, µ).
Define φe0 (x, y, µ) = φe0 (x, y, µ). Consider any e, y, µ such that
µ φe (x, y, µ) < s( y, µ) holds.
x
Then for every i there is a j with
µ he,i (x, y, µ) < s j ( y, µ), and hence h0e,i, j (x, y, µ) = he,i (x, y, µ). It fol-
x

lows that h00e,i (x, y, µ) = he,i (x, y, µ) for all i and hence φe0 (x, y, µ) =
φe (x, y, µ).

Corollary 4.1.1 (Uniform extension). There is an operation H e (x, µ) 7→

H e0 (x, µ) with the property that H e0 (x, µ) is a uniform test and if 2H e (·, µ)
is a µ-test then H e0 (x, µ) = H e (x, µ).

Proof. In Theorem 4.1.1 set φe (x, y, µ) = 12 H e (x, µ) with s( y, µ) = 1.

Corollary 4.1.2 (Universal generalized test). Let s : Y × M (X) → R+ a lower

semicomputable function. Let E be the set of lower semicomputable functions
φ(x, y, µ) ¾ 0 with µ x φ(x, y, µ) ¶ s( y, µ). There is a function ψ ∈ E that is
optimal in the sense that for all φ ∈ E there is a constant cφ with φ ¶ 2cφ ψ.

Proof. Apply the operation e 7→ e0 of theorem 4.1.1 to the sequence

φe (x, y, µ) (e = 1, 2, . . . ) of all lower semicomputable functions of x, y, µ.
The elements of the sequence φe0 (x, y, µ), e = 1, 2, . . . are in E and and
the sequence 2φe0 (x, y, µ),Pe = 1, 2, . . . contains all elements of E. Hence
∞
the function ψ(x, y, µ) = e=1 2−e φe0 (x, y, µ) is in E and has the optimality
property.

90
 4.1. Continuous spaces, noncomputable measures

Definition 4.1.2. A uniform test u is called universal if for every other test t
there is a constant c t > 0 such that for all x, µ we have t µ (x) ¶ cuµ (x). ù

Theorem 4.1.2 (Universal test, by Hoyrup-Rojas). There is a universal uni-

form test.

Proof. This is a special case of Corollary 4.1.2 with s( y, µ) = 1.

Definition 4.1.3. Let us fix a universal uniform test, called tµ (x). An element
x ∈ X is called random with respect to measure µ ∈ M (X) if tµ (x) < ∞.
The deficiency of randomness is defined as dµ (x) = log tµ (x). ù

If the space is discrete then typically all elements are random with respect
to µ, but they will still be distinguished according to their different values of
dµ (x).

4.1.3 SEQUENCES
Let our computable metric space X = (X , d, D, α) be the Cantor space of Ex-
ample B.1.19.2: the set of sequences over a (finite or countable) alphabet
ΣN . We may want to measure the non-randomness of finite sequences, view-
ing them as initial segments of infinite sequences. Take the universal test
tµ (x). For this, it is helpful to apply the representation of Proposition B.1.16,
taking into account that adding the extra parameter µ does not change the
validity of the theorem:

Proposition 4.1.3. There is a function g : M (X) × Σ∗ → R+ with tµ (ξ) =

supn gµ (ξ¶n ), and with the following properties:
(a) g is lower semicomputable.
(b) v v w implies gµ (v) ¶ gµ (w).
(c) For all integer n ¾ 0 we have w∈Σn µ(w)gµ (w) ¶ 1.
P

The properties of the function gµ (w) clearly imply that supn gµ (ξ¶n ) is a
uniform test.
The existence of a universal function among the functions g can be
proved by the usual methods:

Proposition 4.1.4. Among the functions gµ (w) satisfying the properties listed
in Proposition 4.1.3, there is one that dominates to within a multiplicative
constant.

91
4. GENERALIZATIONS

These facts motivate the following definition.

Definition 4.1.4 (Extended test). Over the Cantor space, we extend the defi-
nition of a universal test tµ (x) to finite sequences as follows. We fix a function
tµ (w) with w ∈ Σ∗ whose existence is assured by Proposition 4.1.4. For infi-
nite sequences ξ we define tµ (ξ) = supn tµ (ξ¶n ). The test with values defined
also on finite sequences will be called an extended test. ù

We could try to define extended tests also over arbitrary constructive met-
ric spaces, extending them to the canonical balls, with the monotonicity prop-
erty that tµ (v) ¶ tµ (w) if ball w is manifestly included in ball v. But there is
nothing simple and obvious corresponding to the integral requirement (c).
Over the space ΣN for a finite alphabet Σ, an extended test could also be
extracted directly from a test, using the following formula (as observed by
Vyugin and Shen).

Definition 4.1.5.

u(z) = inf{ u(ω) : ω is an infinite extension of z }.

This function is lower semicomputable, by Proposition B.1.9.

4.1.4 CONSERVATION OF RANDOMNESS

For i = 1, 0, let Xi = (X i , di , Di , αi ) be computable metric spaces, and let

Mi = (M (Xi ), σi , νi ) be the effective topological space of probability mea-
sures over Xi . Let Λ be a computable probability kernel from X1 to X0 as
defined in Subsection B.2.3. In the following theorem, the same notation
dµ (x) will refer to the deficiency of randomness with respect to two different
spaces, X1 and X0 , but this should not cause confusion. Let us first spell out
the conservation theorem before interpreting it.

Theorem 4.1.3. For a computable probability kernel Λ from X1 to X0 , we have

∗
λ xy tΛ∗ µ ( y) < tµ (x). (4.1.1)

Proof. Let tν (x) be the universal test over X0 . The left-hand side of (4.1.1)
can be written as
uµ = ΛtΛ∗ µ .

92
 4.1. Continuous spaces, noncomputable measures

According to (A.2.4), we have µuµ = (Λ∗ µ)tΛ∗ µ which is ¶ 1 since t is a test.

If we show that (µ, x) 7→ uµ (x) is lower semicomputable then the universality
∗
of tµ will imply that uµ < tµ .
According to Proposition B.1.20, as a lower semicomputable function,
tν ( y) can be written as supn g n (ν, y), where (g n (ν, y)) is a computable se-
quence of computable functions. We pointed out in Subsection B.2.3 that
the function µ 7→ Λ∗ µ is computable. Therefore the function (n, µ, x) 7→
g n (Λ∗ µ, f (x)) is also a computable. So, uµ (x) is the supremum of a com-
putable sequence of computable functions and as such, lower semicom-
putable.
It is easier to interpret the theorem first in the special case when Λ = Λh
for a computable function h : X 1 → X 0 , as in Example B.2.11. Then the
theorem simplifies to the following.
+
Corollary 4.1.5. For a computable function h : X 1 → X 0 , we have dh∗ µ (h(x)) <
dµ (x).
Informally, this says that if x is random with respect to µ in X1 then h(x)
is essentially at least as random with respect to the output distribution h∗ µ
in X0 . Decrease in randomness can only be caused by complexity in the
definition of the function h.
Let us specialize the theorem even more:
Corollary 4.1.6. For a probability distribution µ over the space X × Y let µX
be its marginal on the space X . Then we have
+
dµX (x) < dµ ((x, y)).
This says, informally, that if a pair is random then each of its elements is
random (with respect to the corresponding marginal distribution).
In the general case of the theorem, concerning random transitions, we
cannot bound the randomness of each outcome uniformly. The theorem as-
serts that the average nonrandomness, as measured by the universal test with
respect to the output distribution, does R not increase. In logarithmic notation:
y dΛ∗ µ ( y) + dΛ∗ µ ( y) +
λx 2 < dµ (x), or equivalently, 2 λ x (d y) < dµ (x).
Corollary 4.1.7. Let Λ be a computable probability kernel from X1 to X0 . There
is a constant c such that for every x ∈ X1 , and integer m > 0 we have
λ x { y : dΛ∗ µ ( y) > dµ (x) + m + c } ¶ 2−m .

93
4. GENERALIZATIONS

Thus, in a computable random transition, the probability of an increase

of randomness deficiency by m units (plus a constant c) is less than 2−m . The
constant c comes from the description complexity of the transition Λ.
A randomness conservation result related to Corollary 4.1.5 was proved
in [25]. There, the measure over the space X0 is not the output measure of
the transformation, but is assumed to obey certain inequalities related to the
transformation.

4.2 TEST FOR A CLASS OF MEASURES

4.2.1 FROM A UNIFORM TEST

A Bernoulli measure is what we get by tossing a (possibly biased) coin re-

peatedly.

Definition 4.2.1. Let X = BN be the set of infinite binary sequences, with

the usual sequence topology. Let B p be the measure on X that corresponds
to tossing a coin independently with probability p of success: it is called the
Bernoulli measure with parameter p. Let B denote the set of all Bernoulli
measures. ù

Given a sequence x ∈ X we may ask the question whether x is random

with respect to at least some Bernoulli measure. (It can clearly not be random
with respect to two different Bernoulli measures since if x is random with
respect to B p then its relative frequencies converge to p.) This idea suggests
two possible definitions for a test of the property of “Bernoulliness”:
1. We could define tB (x) = infµ∈B (x).
2. We could define the notion of a Bernoulli test as a lower semicomputable
function f (x) with the property B p f ¶ 1 for all p.
We will see that in case of this class of measures the two definitions lead to
essentially the same test.
Let us first extend the definition to more general sets of measures, still
having a convenient property.

Definition 4.2.2 (Class tests). Consider a class C of measures that is effec-

tively compact in the sense of Definition B.1.10 or (equivalently for metric
spaces) in the sense of Theorem B.1.1. A lower semicomputable function

94
 4.2. Test for a class of measures

f (x) is called a C -test if for all µ ∈ C we have µ f ¶ 1. It is a universal C -

test if it dominates all other C -tests to within a multiplicative constant. ù

Example 4.2.1. It is easy to show that the class B is effectively compact. One
way is to appeal to the general theorem in Proposition B.1.10 saying that
applying a computable function to an effectively compact set (in this case
the interval [0, 1]), the image is also an effectively compact set. ù
For the case of infinite sequences, we can also define extended tests.

Definition 4.2.3 (Extended class test). Let our space X be the Cantor space
of infinite sequences ΣN . Consider a class C of measures that is effectively
compact in the sense of Definition B.1.10 or (equivalently for metric spaces)
in the sense of Theorem B.1.1. A lower semicomputable function f : Σ∗ →
R+ is called an extended C -test if it is monotonic with respect to the prefix
relation and for all µ ∈ C and integer n ¾ 0 we have
X
µ(x) f (x) ¶ 1.
x∈Σn

It is universal if it dominates all other extended C -tests to within a multi-

plicative constant. ù

The following observation is immediate.

Proposition 4.2.2. A function f : ΣN → R+ is a class test if and only if it can

be represented as limn g(ξ¶n ) where g(x) is an extended class test.

The following theorem defines a universal C -test.

Theorem 4.2.1. Let tµ (x) be a universal uniform test. Then u(x) =

infµ∈C tµ (x) defines a universal C -test.

Proof. Let us show first that u(x) is a C -test. It is lower semicomputable

according to Proposition B.1.9. Also, for each µ we have µu ¶ µtµ ¶ 1,
showing that u(x) is a C -test.
Let us now show that it is universal. Let f (x) be an arbitrary C -test. By
Corollary 4.1.1 there is a uniform test gµ (x) such that for all µ ∈ C we have
gµ (x) = f (x)/2. It follows from the universality of the uniform test tµ (x)
∗ ∗ ∗
that f (x) < gµ (x) < tµ (x) for all µ ∈ C . But then f (x) < infµ∈C tµ (x) =
u(x).

95
4. GENERALIZATIONS

For the case of sequences, the same statement can be made for extended
tests. (This is not completely automatic since a test is obtained from an
extended test via a supremum, on the other hand a class test is obtained,
according the theorem above, via an infimum.)

4.2.2 TYPICALITY AND CLASS TESTS

The set of Bernoulli measures has an important property shared by many

classes considered in practice: namely that random sequences determine the
measure to which it belongs.
Definition 4.2.4. Consider a class C of measures over a computable metric
space X = (X , d, D, α). We will say that a lower semicomputable function
s : X × C → R+
is a separating test for C if
• sµ (·) is a test for each µ ∈ C .
• If µ 6= ν then sµ (x) ∨ sν (x) = ∞ for all x ∈ X .
Given a separating test sµ (x) we call an element x typical for µ ∈ C if sµ (x) <
∞. ù
A typical element determines uniquely the measure µ for which it is
typical. Note that if a separating tests exists for a class then any two dif-
ferent measures µ1 , µ2 in the class are orthogonal to each other, that is
there are disjoint measureable sets A1 , A2 with µ j (A j ) = 1. Indeed, let
A j = { x : sµ j (x) < ∞ }.
Let us show a nontrivial example: the class of B of Bernoulli measures.
Recall that by Chebyshev’s inequality we have
X
B p ({ x ∈ B : |
n
x(i) − np| ¾ λn1/2 (p(1 − p))1/2 }) ¶ λ−2 .
i

Since p(1 − p) ¶ 1/4, this implies

X
B p ({ x ∈ B : |
n
x(i) − np > λn1/2 /2 }) < λ−2 .
i

Setting λ = n0.1 and ignoring the factor 1/2 gives

X
B p ({ x ∈ Bn : | x(i) − np| > n0.6 }) < n−0.2 .
i

96
 4.2. Test for a class of measures

Setting n = 2k :
k
X
B p ({ x ∈ B2 : | x(i) − 2k p| > 20.6k }) < 2−0.2k . (4.2.1)
i

Now, for the example.

Example 4.2.3. For a sequence ξ in BN , and for p ∈ [0, 1] let
2k
X
g p (x) = g Bp (x) = sup{ k : | ξ(i) − 2k p| > 20.6k }.
i=1

Then we have
X
B pξ g p (ξ) ¶ k · 2−0.2k = c < ∞
k

for some constant c, so s p (ξ) = g p (x)/c is a test for each p. The property
P2 k
s p (ξ) < ∞ implies that 2−k i=1 ξ(i) converges to p. For a given ξ this is
impossible for both p and q for p 6= q, hence s p (ξ) ∨ sq (ξ) = ∞. ù
The following structure theorem gives considerable insight.

Theorem 4.2.2. Let C be an effectively compact class of measures, let tµ (x) be

the universal uniform test and let tC (x) be a universal class test for C . Assume
that a separating test sµ (x) exists for C . Then we have the representation

tµ (x) = tC (x) ∨ sµ (x)

∗

for all µ ∈ C , x ∈ X .
∗
Proof. First, we have tC (x) ∨ sµ (x) < tµ (x). Indeed as we know from the
Uniform Extension Corollary 4.1.1, we can extend sµ (x)/2 to a uniform test,
∗
hence sµ (x) < tµ (x). Also by definition tC (x) ¶ tµ (x).
On the other hand, let us show tC (x) ∨ sµ (x) ¾ tµ (x). Suppose first that
x is not random with respect to any ν ∈ C : then tC (x) = ∞. Suppose now
that x is random with respect to some ν ∈ C , ν 6= µ. Then sµ (x) = ∞.
Finally, suppose tµ (x) < ∞. Then tν (x) = ∞ for all ν ∈ C , ν 6= µ, hence
tC (x) = infν∈C tν (x) = tµ (x), so the inequality holds again.
The above theorem separates the randomness test into two parts. One
part tests randomness with respect to the class C , the other one tests typi-
cality with respect to the measure µ. In the Bernoulli example,

97
4. GENERALIZATIONS

• Part tB (ξ) checks “Bernoulliness”, that is independence. It encom-

passes all the irregularity criteria.
• Part s p (ξ) checks (crudely) for the law of large numbers: whether rel-
ative frequency converges (fast) to p.
If the independence of the sequence is taken for granted, we may assume
that the class test is satisfied. What remains is typicality testing, which is
similar to ordinary statistical parameter testing.
Remarks 4.2.4. 1. Separation is the only requirement of the test sµ (x), oth-
erwise, for example in the Bernoulli test case, no matter how crude the
convergence criterion expressed by sµ (x), the maximum tC (x) ∨ sµ (x) is
always (essentially) the same universal test.
2. Though the convergence criterion can be crude, but one still seems to
need some kind of constructive convergence of the relative frequencies if
the separation test is to be defined in terms of relative frequency conver-
gence.
ù
Example 4.2.5. For " > 0 let P(") be the Markov chain X 1 , X 2 , . . . with set of
states {0, 1}, with transition probabilities T (0, 1) = T (1, 0) = " and T (0, 0) =
T (1, 1) = 1 − ", and with P[ X 1 = 0 ] = P[ X 1 = 1 ] = 1/2. Let C (δ) be the
class of all P(") with " ¾ δ. For each δ > 0, this is an effectively compact
class, and a separating test is easy to construct since an effective law of large
numbers holds for these Markov chains.
We can generalize to the set of m-state stationary Markov chains whose
eigenvalue gap is ¾ ". ù
This example is in contrast to V’yugin’s example [54] showing that, in
the nonergodic case, in general no recursive speed of convergence can be
guaranteed in the Ergodic Theorem (which is the appropriate generalization
of the law of large numbers)1 .
We can show that if a computable Markov chain is ergodic then its law of
large numbers does have a constructive speed of convergence. Hopefully, this
observation can be extended to some interesting compact classes of ergodic
processes.
1
This paper of Vyugin also shows that though there is no recursive speed of convergence,
a certain constructive proof of the pointwise ergodic theorem still gives rise to a test of
randomness.

98
 4.2. Test for a class of measures

4.2.3 MARTIN-LÖF’S APPROACH

Martin-Löf also gave a definition of Bernoulli tests in [37]. For its definition
let us introduce the following notation.
Notation 4.2.5. The set of sequences with a given frequency of 1’s will be
denoted as follows:
X
B(n, k) = { x ∈ Bn : x(i) = k }.
i

ù
Martin-Löf’s definition translates to the integral constraint version as fol-
lows:
Definition 4.2.6. Let X = BN be the set of infinite binary sequences with the
usual metrics. A combinatorial Bernoulli test is a function f : B∗ → R+ with
the following constraints:
(a) It is lower semicomputable.
(b) It is monotonic with respect to the prefix relation.
(c) For all 0 ¶ k ¶ n we have
n
X  
f (x) ¶ . (4.2.2)
x∈B(n,k)
k

ù
The following observation is useful.
Proposition 4.2.6. If a combinatorial Bernoulli test f (x) is given on strings x
of length less than n, then extending it to longer strings using monotonicity we
get a function that is still a combinatorial Bernoulli test.
Proof. It is sufficient to check the relation (4.2.2). We have (even for k = 0,
when B(n − 1, k − 1) = ;):
X X X
f (x) ¶ f ( y) + f ( y)
x∈B(n,k) y∈B(n−1,k−1) y∈B(n−1,k)
n−1 n−1 n
     
¶ + = .
k−1 k k

99
4. GENERALIZATIONS

The following can be shown using standard methods:

Proposition 4.2.7 (Universal combinatorial Bernoulli test). There is a uni-
versal combinatorial Bernoulli test f (x), that is a combinatorial Bernoulli test
with the property that for every combinatorial Bernoulli test h(x) there is a
constant ch > 0 such that for all x we have h(x) ¶ ch g(x).
Definition 4.2.7. Let us fix a universal combinatorial Bernoulli test b(x) and
extend it to infinite sequences ξ by
b(ξ) = sup b(ξ¶n ).
n

Let tB (ξ) be a universal class test for Bernoulli measures, for ξ ∈ BN . ù

Let us show that the general class test for Bernoulli measures and Martin-
Löf’s Bernoulli test yield the same random sequences.
Theorem 4.2.3. With the above definitions, we have b(ξ) = tB (ξ). In words:
∗

a sequence is nonrandom with respect to all Bernoulli measures if and only if it

is rejected by a universal combinatorial Bernoulli test.
Proof. We first show b(ξ) ¶ tB (ξ). Moreover, we show that b(x) is an ex-
tended class test for the class of Bernoulli measures. We only need to check
P sum condition, namely that for all 0 ¶ p ¶ 1, and all n > 0 the inequality
the
x∈Bn B p (x)b(x) ¶ 1 holds. Indeed, we have

X n
X X
B p (x) f (x) = p k (1 − p)n−k f (x)
x∈Bn k=0 x∈B(n,k)
n
n
X  
¶ p (1 − p)
k n−k
= 1.
k=0
k

On the other hand, let f (x) = p tB (x), x ∈ B∗ be the extended test for tB (ξ).
For all integers N > 0 let n = b N /2c. Then as N runs through the integers,
n also runs through all integers. For x ∈ BN let F (x) = f (x ¶n ). Since f (x)
is lower semicomputable and monotonic with respect to the prefix relation,
this is also true of F (x). P
We need to estimate x∈B(N ,K) F (x). For this, note that for y ∈ B(n, k)
we have
N −n
 
|{ x ∈ B(N , K) : y v x }| = . (4.2.3)
K−k

100
 4.2. Test for a class of measures

Now for 0 ¶ K ¶ N we have

X X
F (x) = f ( y)|{ x ∈ B(N , K) : y v x }|
x∈B(N ,K) y∈Bn
n 
X N −n
 X (4.2.4)
= f ( y).
k=0
K−k y∈B(n,k)

N −n
 N

Let us estimate K−k K
. If K = k = 0 then this is 1. If k = n then it is
K···(K−n+1)
N ···(N −n+1)
. Otherwise, using p = K/N :

N −n

K−k (N − n)(N − n − 1) · · · (N − K − (n − k) + 1)/(K − k)!
=
N

N (N − 1) · · · (N − K + 1)/K!
K
K · · · (K − k + 1) · (N − K) · · · (N − K − (n − k) + 1)
= .
N · · · (N − n + 1)
K k (N − K)n−k
n
N

¶ = p (1 − p)
k n−k
. (4.2.5)
(N − n)n N −n

Thus in all cases the estimate

n
N −n N N
   
¶ p (1 − p)
k n−k
K−n K N −n

holds. We have
n ‹n
N n
  N
= 1+ ¶ e 2(N −n) ¶ e2 ,
N −n N −n

since we assumed 2n2 ¶ N . Substituting into (4.2.4) gives

X n
X X
F (x) ¶ e 2
p k (1 − p)n−k f ( y) ¶ e2 ,
x∈B(N ,K) k=0 y∈B(n,k)

since f (x) is an extended class test for Bernoulli measures. It follows

∗ ∗
that e−2 F (x) < b(x), hence also F (x) < b(x). But we have tC (ξ) =
∗
supn f (ξ¶n ) = supn F (ξ¶n ), hence tC (ξ) < b(ξ).

101
4. GENERALIZATIONS

4.3 NEUTRAL MEASURE

Let tµ (x) be our universal uniform randomness test. We call a measure M

neutral if t M (x) ¶ 1 for all x. If M is neutral then no experimental outcome x
could refute the theory (hypothesis, model) that M is the underlying measure
to our experiments. It can be used as “apriori probability”, in a Bayesian
approach to statistics. Levin’s theorem says the following:

Theorem 4.3.1. If the space X is compact then there is a neutral measure over
X.

The proof relies on a nontrivial combinatorial fact, Sperner’s Lemma,

which also underlies the proof of the Brouwer fixpoint theorem. Here is a
version of Sperner’s Lemma, spelled out in continuous form:

Proposition 4.3.1. Let p1 , . . . , pk be points of some finite-dimensional space

Rn . Suppose that there are closed sets F1 , . . . , Fk with the property that for every
subset 1 ¶ i1 < · · · < i j ¶ k of the indices, the simplex S(pi1 , . . . , pi j ) spanned
T
by pi1 , . . . , pi j is covered by the union Fi1 ∪ · · · ∪ Fi j . Then the intersection i Fi
of all these sets is not empty.

The following lemma will also be needed.

Lemma 4.3.2. For every closed set A ⊂ X and measure µ, if µ(A) = 1 then
there is a point x ∈ A with tµ (x) ¶ 1.

Proof. This follows easily from µ t µ = µ x 1A(x)t µ (x) ¶ 1.

Proof of Theorem 4.3.1. For every point x ∈ X, let F x be the set of measures
for which tµ (x) ¶ 1. If we show that for every finite set of points x 1 , . . . , x k ,
we have
F x 1 ∩ · · · ∩ F x k 6= ;, (4.3.1)
then we will be done. Indeed, according to Proposition A.2.28, the compact-
ness of X implies the compactness of the space M(X) of measures. Therefore
if every finite subset of the family { F x : x ∈ X } of closed sets has a nonempty
intersection, then the whole family has a nonempty intersection: this inter-
section consists of the neutral measures.
To show (4.3.1), let S(x 1 , . . . , x k ) be the set of probability measures con-
centrated on x 1 , . . . , x k . Lemma 4.3.2 implies that each such measure be-
longs to one of the sets F x i . Hence S(x 1 , . . . , x k ) ⊂ F x 1 ∪ · · · ∪ F x k , and the

102
 4.3. Neutral measure

same holds for every subset of the indices {1, . . . , k}. Sperner’s Lemma 4.3.1
implies F x 1 ∩ · · · ∩ F x k 6= ;.
When the space is not compact, there are generally no neutral probability
measures, as shown by the following example.

Proposition 4.3.3. Over the discrete space X = N of natural numbers, there is

no neutral measure.

Proof. It is sufficient to construct a randomness test t µ (x) with the property

that for every measure µ, we have sup x t µ (x) = ∞. Let
X
t µ (x) = sup{ k ∈ N : µ( y) > 1 − 2−k }. (4.3.2)
y<x

By its construction, this P is a lower semicomputable function with

sup x t µ (x) = ∞. It is a test if x µ(x)t µ (x) ¶ 1. We have
X X X X
µ(x)t µ (x) = µ(x) < 2−k ¶ 1.
x k>0 t µ (x)¾k k>0

Using a similar construction over the space NN of infinite sequences of

natural numbers, we could show that for every measure µ there is a sequence
x with tµ (x) = ∞.
Proposition 4.3.3 is a little misleading, since N can be compactified into
N = N ∪ {∞} (as in Part 1 of Example A.1.9). Theorem 4.3.1 implies that
there is a neutral probability measure M over the compactified space N. Its
restriction to N is, of course, not a probability measure, since it satisfies only
x<∞ M (x) ¶ 1. We called these functions semimeasures.
P

Remark 4.3.4.
1. It is easy to see that Theorem 4.6.1 characterizing randomness in terms
of complexity holds also for the space N.
2. The topological space of semimeasures over N is not compact, and there
is no neutral one among them. Its topology is not the same as what we get
when we restrict the topology of probability measures over N to N. The
difference is that over N, for example the set of measures { µ : µ(N) >
1/2 } is closed, since N (as the whole space) is a closed set. But over N,
this set is not necessarily closed, since N is not a closed subset of N.

103
4. GENERALIZATIONS

ù
Neutral measures are not too simple, even over N, as the following theo-
rem shows.

Theorem 4.3.2. There is no neutral measure over N that is upper semicom-

putable over N or lower semicomputable over N.

Proof. Let us assume that ν is a measure that is upper semicomputable over

N. Then the set
{ (x, r) : x ∈ N, r ∈ Q, ν(x) < r }
is recursively enumerable: let (x i , ri ) be a particular enumeration. For each
n, let i(n) be the first i with ri < 2−n , and let yn = x i(n) . Then ν( yn ) <
+
2−n , and at the same time H( yn ) < H(n). As mentioned, in Remark 4.3.4,
Theorem 4.6.1 characterizing randomness in terms of complexity holds also
for the space N. Thus,
+ +
dν ( yn ) = − log ν( yn ) − H( yn | ν) > n − H(n).

Suppose now that ν is lower semicomputable over N. The proof for this case
is longer. We know that ν is the monotonic limit of a recursive sequence
i 7→ νi (x) of recursive semimeasures with rational values νi (x). For every
k = 0, . . . , 2n − 2, let

Vn,k = { µ ∈ M (N) : k · 2−n < µ({0, . . . , 2n − 1}) < (k + 2) · 2−n },

J = { (n, k) : k · 2−n < ν({0, . . . , 2n − 1}) }.

The set J is recursively enumerable. Let us define the functions j : J → N and

x : J → {0, . . . , 2n − 1} as follows: j(n, k) is the smallest i with νi ({0, . . . , 2n −
1}) > k · 2−n , and

x n,k = min{ y < 2n : ν j(n,k) ( y) < 2−n+1 }.

Let us define the function fµ (x, n, k) as follows. We set fµ (x, n, k) = 2n−2 if

the following conditions hold:
(a) µ ∈ Vn,k ;
(b) µ(x) < 2−n+2 ;
(c) (n, k) ∈ J and x = x n,k .

104
 4.3. Neutral measure

Otherwise, fµ (x, n, k) = 0. Clearly, the function (µ, x, n, k) 7→ fµ (x, n, k) is

lower semicomputable. Condition (b) implies
X
µ( y) fµ ( y, n, k) ¶ µ(x n,k ) fµ (x n,k , n, k) < 2−n+2 · 2n−2 = 1. (4.3.3)
y

Let us show that ν ∈ Vn,k implies

fν (x n,k , n, k) = 2n−2 . (4.3.4)

Consider x = x n,k . Conditions (a) and (c) are satisfied by definition. Let us
show that condition (b) is also satisfied. Let j = j(n, k). By definition, we
have ν j (x) < 2−n+1 . Since by definition ν j ∈ Vn,k and ν j ¶ ν ∈ Vn,k , we have

ν(x) ¶ ν j (x) + 2−n+1 < 2−n+1 + 2−n+1 = 2−n+2 .

Since all three conditions (a), (b) and (c) are satisfied, we have
shown (4.3.4). Now we define
X 1 X
gµ (x) = fµ (x, n, k).
n¾2
n(n + 1) k

Let us prove that gµ (x) is a uniformPtest. It is lower semicomputable by

definition, so we only need to prove x µ(x) fµ (x) ¶ 1. For this, let I n,µ =
{ k : µ ∈ Vn,k }. Clearly by definition, |I n,µ | ¶ 2. We have, using this last fact
and the test property (4.3.3):
X X 1 XX X 1
µ(x)gµ (x) = µ(x) fµ (x, n, k) ¶ · 2 ¶ 1.
x n¾2
n(n + 1) k∈I x n¾2
n(n + 1)
n,µ

Thus, gµ (x) is a uniform test. If ν ∈ Vn,k then we have

∗ 1 2n−2
tν (x n,k ) > gν (x n,k ) ¾ fµ (x n,k , n, k) ¾ .
n(n + 1) n(n + 1)

Hence ν is not neutral.

Remark 4.3.5. In [31] and [32], Levin imposed extra conditions on tests
which allow to find a lower semicomputable neutral semimeasure. ù

105
4. GENERALIZATIONS

The universal lower semicomputable semimeasure m(x) has a certain

property similar to neutrality. According to Theorem 2.3.4 specialized to
+
one-element sequences, for every computable measure µ we have dµ (x) =
+
− log µ(x) − H(x) (where the constant in = depends on µ). So, for comput-
able measures, the expression

dµ (x) = − log µ(x) − H(x) (4.3.5)

can serve as a reasonable deficiency of randomness. (We will also use the
test t = 2d .) If we substitute m for µ in dµ (x), we get 0. This substitution
is not justified, of course. The fact that m is not a probability measure can
be helped, using compactification as above, and extending the notion of ran-
domness tests. But the test dµ can replace dµ only for computable µ, while
m is not computable. Anyway, this is the sense in which all outcomes might
be considered random with respect to m, and the heuristic sense in which m
may be considered “neutral”.

4.4 MONOTONICITY, QUASI - CONVEXITY / CONCAVITY

Some people find that µ-tests as defined in Definition 4.1.1 are too gen-
eral, in case µ is a non-computable measure. In particular, randomness
with respect to computable measures has a certain—intuitively meaningful—
monotonicity property: roughly, if ν is greater than µ then if x is random
with respect to µ, it should also be random with respect to ν.
Proposition 4.4.1. For computable measures µ, ν we have for all rational c >
0:
+
2−k µ ¶ ν ⇒ dν (x) < dµ (x) + k + H(k). (4.4.1)
+
Here the constant in < depends on µ, ν, but not on k.
Proof. We have 1 ¾ νtν ¾ 2−k µtν , hence 2−k tν is a µ-test. Using the method
of Theorem 4.1.1 in finding universal tests, one can show that the sum
X
2−k−H(k) tν
k:2−k µtν <1

∗
is a µ-test, and hence < tµ . Therefore this is true of each member of the sum,
which is just what the theorem claims.

106
 4.4. Monotonicity, quasi-convexity/concavity

There are other properties true for tests on computable measures that we
may want to require for all measures. For the following properties, let us
define quasi-convexity, which is a weakening of the notion of convexity.

Definition 4.4.1. A function f : V → R defined on a vector space V is called

quasi-convex if for every real number x the set { v : f (v) ¶ x } is convex. It is
quasi-concave if − f is quasi-convex. ù

It is easy to see that quasi-convexity is equivalent to the inequality

f (λu + (1 − λ)v) ¶ f (u) ∨ f (v)

for all u, v and 0 < λ < 1, while quasi-concavity is equivalent to

f (λu + (1 − λ)v) ¾ f (u) ∧ f (v)

The uniform test with respect to computable measures is approximately

both quasi-convex and quasi-concave. Let ν = λµ1 + (1 − λ)µ2 .
Quasi-convexity means, roughly, that if x is random with respect to both
µ1 and µ2 then it is also random with respect to ν. This property strengthens
monotonicity in the cases where it applies.

Proposition 4.4.2. Let µ1 , µ2 be computable measures and 0 < λ < 1 comput-

able, with ν = λµ1 + (1 − λ)µ2 . Then we have
+
dν (x) < dµ1 (x) ∨ dµ2 (x) + H(λ).

Proof. The relation 1 ¾ νtν = λµ1 tν + (1 − λ)µ2 tν implies 1 ¾ µi tν for some

+
i ∈ {1, 2}. Then dν < dµi + H(λ) (since λ was used to define ν and thus
tν ).
Quasi-concavity means, roughly, that if x is non-random with respect to
both µ1 and µ2 then it is also nonrandom with respect to ν:

Proposition 4.4.3. Let µ1 , µ2 be computable measures and 0 < λ < 1 arbitrary

(not even necessarily computable), with ν = λµ1 + (1 − λ)µ2 . Then we have
+
dν > dµ1 ∧ dµ2 .

Proof. The function tµ1 ∧ tµ2 is lower semicomputable, and is a µi -test for
∗
each i. Therefore it is also a ν-test, and as such is < tν . Here, the constant in
∗
the < depends only on (the programs for) µ1 , µ2 , and not on λ.

107
4. GENERALIZATIONS

These properties do not survive for arbitrary measures and arbitrary con-
stants.
Example 4.4.4. Let measure µ1 be uniform over the interval [0, 1/2], let µ2
be uniform over [1/2, 1]. For 0 ¶ x ¶ 1 let

ν x = (1 − x)µ1 + xµ2 .

Then ν1/2 is uniform over [0, 1]. Let φ x be the uniform distribution over
[x, x + 1/2], and ψ x the uniform distribution over [0, x] ∪ [x + 1/2, 1].
Let p < 1/2 be random with respect to the uniform distribution ν1/2 .
1. The relations

ν1/2 ¶ p−1 ν p , dν1/2 (p) < ∞, dνp (p) = ∞

show that any statement analogous to the monotonicity property of

Proposition 4.4.1 fails when the measures involved are not required to
be computable.
2. For rational r with 0 < r < p the relations

ν := (1 − p)ν r + pν1−r , dν (p) = ∞, dνr (p) < ∞, dν1−r (p) < ∞

provide a similar counterexample to Proposition 4.4.2.

3. The relations

ν1/2 = (ν p + ν1−p )/2, dν1/2 (p) < ∞, dνp (p) = dν1−p (p) = ∞

provide a similar counterexample to Proposition 4.4.3.

The following counterexample relies on a less subtle effect:

ν1/2 = (φ p + ψ p )/2, dν1/2 (p) < ∞, dφp (p) = dψp (p) = ∞,

since as a boundary point of the support, p is computable from both φ p

and ψ p in a uniform way.
For a complete proof, uniform tests must be provided for each of the cases:
this is left as exercise for the reader. ù
The non-monotonicity example could be used to argue that the we al-
lowed too many µ-tests, that the test tµ (x) should not be allowed to depend
on properties of µ that exploit the computational properties of µ so much
stronger than its quantitative properties.

108
 4.5. Algorithmic entropy

4.5 ALGORITHMIC ENTROPY

Some properties of description complexity make it a good expression of the

idea of individual information content.

4.5.1 ENTROPY
The entropy of a discrete probability distribution µ is defined as
X
H (µ) = − µ(x) log µ(x).
x

To generalize entropy to continuous distributions the relative entropy is de-

fined as follows. Let µ, ν be two measures, where µ is taken (typically, but
not always), to be a probability measure, and ν another measure, that can
also be a probability measure but is most frequently not. We define the rela-
tive entropy Hν (µ) as follows. If µ is not absolutely continuous with respect
to ν then Hν (µ) = −∞. Otherwise, writing

dµ µ(d x)
= =: f (x)
dν ν(d x)
for the (Radon-Nikodym) derivative (density) of µ with respect to ν, we
define
µ(d x)
Z
dµ
Hν (µ) = − log dµ = −µ x log = −ν x f (x) log f (x).
dν ν(d x)

Thus, H (µ) = H# (µ) is a special case.

Example 4.5.1. Let f (x) be a probability density function for the distribution
µ over the real line, and let λ be the Lebesgue measure there. Then
Z
Hλ (µ) = − f (x) log f (x)d x.

ù
In information theory and statistics, when both µ and ν are probability
measures, then −Hν (µ) is also denoted D(µ k ν), and called (after Kullback)
the information divergence of the two measures. It is frequently used in the
role of a distance between µ and ν. It is not symmetric, but can be shown to

109
4. GENERALIZATIONS

obey the triangle inequality, and to be nonnegative. Let us prove the latter
property: in our terms, it says that relative entropy is nonpositive when both
µ and ν are probability measures.
Proposition 4.5.2. Over a space X , we have
µ(X )
Hν (µ) ¶ −µ(X ) log . (4.5.1)
ν(X )
In particular, if µ(X ) ¾ ν(X ) then Hν (µ) ¶ 0.
Proof. The inequality −a ln a ¶ −a ln b + b − a expresses the concavity of the
logarithm function. Substituting a = f (x) and b = µ(X )/ν(X ) and integrat-
ing by ν:
µ(X ) µ(X )
(ln 2)Hν (µ) = −ν x f (x) ln f (x) ¶ −µ(X ) ln + ν(X ) − µ(X )
ν(X ) ν(X )
µ(X )
= −µ(X ) ln ,
ν(X )
giving (4.5.1).

4.5.2 ALGORITHMIC ENTROPY

Le us recall some facts on description complexity. Let us fix some (finite or

infinite) alphabet Σ and consider the discrete space Σ∗ .
The universal lower semicomputable semimeasure m(x) over Σ∗ was de-
fined in Definition 1.6.6. It is possible to turn m(x) into a measure, by com-
pactifying the discrete space Σ∗ into
Σ∗ = Σ∗ ∪ {∞}
(as in part 1 of Example A.1.9; this process makes
P sense also for a construc-
tive discrete space), and setting m(∞) = 1 − x∈Σ∗ m(x). The extended
measure m is not quite lower semicomputable since the number µ(Σ∗ \ {0})
is not necessarily lower semicomputable.
Remark 4.5.3. A measure µ is computable over Σ∗ if and only if the function
x 7→ µ(x) is computable for x ∈ Σ∗ . This property does not imply that the
number X
1 − µ(∞) = µ(Σ∗ ) = µ(x)
x∈Σ∗
is computable. ù

110
 4.5. Algorithmic entropy

Let us allow, for a moment, measures µ that are not probability measures:
they may not even be finite. Metric and computability can be extended to
this case, the universal test tµ (x) can also be generalized. The Coding Theo-
rem 1.6.5 and other considerations suggest the introduction of the following
notation, for an arbitrary measure µ:
Definition 4.5.1. We define the algorithmic entropy of a point x with respect
to measure µ as
Hµ (x) = −dµ (x) = − log tµ (x). (4.5.2)
ù
Then, with # defined as the counting measure over the discrete set Σ∗
(that is, #(S) = |S|), we have
+
H(x) = H# (x).

This allows viewing Hµ (x) as a generalization of description complexity.

The following theorem generalizes an earlier known theorem stating that
over a discrete space, for a computable measure, entropy is within an addi-
+
tive constant the same as “average complexity”: H (µ) = µ x H(x).
Theorem 4.5.1. Let µ be a probability measure. Then we have

Hν (µ) ¶ µ x Hν (x | µ). (4.5.3)

If X is a discrete space then the following estimate also holds:

+
Hν (µ) > µ x Hν (x | µ). (4.5.4)

Proof. Let δ be the measure with density tν (x | µ) with respect to ν: tν (x |

δ(d x)
µ) = ν(d x) . Then δ(X ) ¶ 1. It is easy to see from the maximality property
of tν (x | µ) that tν (x | µ) > 0, therefore according to Proposition A.2.12, we
 −1
ν(d x) δ(d x)
have δ(d x) = ν(d x) . Using Proposition A.2.12 and 4.5.2:

µ(d x)
Hν (µ) = −µ x log ,
ν(d x)
δ(d x) ν(d x)
−µ x Hν (x | µ) = µ x log = −µ x log ,
ν(d x) δ(d x)
µ(d x) µ(X )
Hν (µ) − µ x Hν (x | µ) = −µ x log ¶ −µ(X ) log ¶ 0.
δ(d x) δ(X )

111
4. GENERALIZATIONS

This proves (4.5.3).

µ(d x) µ(x)
Over a discrete space X , the function (x, µ, ν) 7→ ν(d x)
= ν(x)
is comput-
µ(d x) ∗
able, therefore by the maximality property of Hν (x | µ) we have ν(d x) < tν (x |
µ(d x) +
µ), hence Hν (µ) = −µ x log ν(d x) > µ x Hν (x | µ).

4.5.3 ADDITION THEOREM

The Addition Theorem (3.1.5) can be generalized to the algorithmic entropy

Hµ (x) introduced in (4.5.2) (a somewhat similar generalization appeared
in [52]). The generalization, defining Hµ,ν = Hµ×ν , is
+
Hµ,ν (x, y) = Hµ (x | ν) + Hν ( y | x, Hµ (x | ν), µ). (4.5.5)
Before proving the general addition theorem, we establish a few useful facts.
Proposition 4.5.4. We have
+
Hµ (x | ν) < − log ν y 2−Hµ,ν (x, y) .
Proof. The function f (x, µ, ν) that is the right-hand side, is upper semicom-
putable by definition, and obeys µ x 2− f (x,µ,ν) ¶ 1. Therefore the inequality
follows from the minimum property of Hµ (x).
Let z ∈ N, then the inequality
+
Hµ (x) < H(z) + Hµ (x | z) (4.5.6)
will be a simple consequence of the general addition theorem. The follow-
ing lemma, needed in the proof of the theorem, generalizes this inequality
somewhat:
Lemma 4.5.5. For a computable function ( y, z) 7→ f ( y, z) over N, we have
+
Hµ (x | y) < H(z) + Hµ (x | f ( y, z)).
Proof. The function
X
(x, y, µ) 7→ gµ (x, y) = 2−Hµ (x| f ( y,z))−H(z)
z
∗
is lower semicomputable, and µ x gµ (x, y) ¶ z 2−H(z) ¶ 1. Hence gµ (x, y) <
P

2−Hµ (x| y) . The left-hand side is a sum, hence the inequality holds for each
element of the sum: just what we had to prove.

112
 4.5. Algorithmic entropy

The following monotonicity property will be needed:

Lemma 4.5.6. For i < j we have
+
i + Hµ (x | i) < j + Hµ (x | j).

Proof. From Lemma 4.5.5, with f (i, n) = i + n we have

+ +
Hµ (x | i) − Hµ (x | j) < H( j − i) < j − i.

Let us generalize the minimum property of Hµ (x).

Proposition 4.5.7. Let ( y, ν) 7→ Fν ( y) be an upper semicomputable function
with values in Z = Z ∪ {−∞, ∞}. Then by Corollary 4.1.2 among the lower
semicomputable functions (x, y, ν) 7→ gν (x, y) with ν x gν (x, y) ¶ 2−Fν ( y) there
is one that is maximal to within a multiplicative constant. Choosing fν (x, y)
as such a function we have for all x with Fν ( y) > −∞:

fν (x, y) = 2−Fν ( y) tν (x | y, Fν ( y)),

∗

+
or in logarithmic notation − log fν (x, y) = Fν ( y) + Hν (x | y, Fν ( y)).
∗
Proof. To prove the inequality >, define

gν (x, y, m) = max 2−i tν (x | y, i).

i¾m

Function gν (x, y, m) is lower semicomputable and decreasing in m. There-

fore
gν (x, y) = gν (x, y, Fν ( y))
is also lower semicomputable since it is obtained by substituting an upper
semicomputable function for m in gν (x, y, m). The multiplicative form of
Lemma 4.5.6 implies

gν (x, y, m) = 2−m tν (x | y, m),

∗

gν (x, y) = 2−Fν ( y) tν (x | y, Fν ( y)).

∗

We have, since tν is a test:

ν x 2−m tν (x | y, m) ¶ 2−m ,
ν x gν (x, y) < 2−Fν ( y) ,
∗

113
4. GENERALIZATIONS

∗
implying gν (x, y) < fν (x, y) by the optimality of fν (x, y).
To prove the upper bound, consider all lower semicomputable func-
tions φe (x, y, m, ν) (e = 1, 2, . . . ). By Theorem 4.1.1, there is a recur-
sive mapping e 7→ e0 with the property that ν x φe0 (x, y, m, ν) ¶ 2−m+1 , and
for each y, m, ν if ν x φe (x, y, m, ν) < 2−m+1 then φe = φe0 . Let us ap-
ply this transformation to the function φe (x, y, m, ν) = fν (x, y). The re-
sult is a lower semicomputable function fν0 (x, y, m) = φe0 (x, y, m, ν) with
the property that ν x fν0 (x, y, m) ¶ 2−m+1 , further ν x fν (x, y) ¶ 2−m implies
fν0 (x, y, m) = fν (x, y). Now (x, y, m, ν) 7→ 2m−1 fν0 (x, y, m) is a uniform test
∗
of x conditional on y, m and hence it is < tν (x | y, m). Substituting Fν ( y) for
m the relation ν x fν (x, y) ¶ 2−m is satisfied and hence we have

fν (x, y) = fν0 (x, y, Fν ( y)) < 2−Fν ( y)+1 tν (x | y, Fν ( y)).

∗

As mentioned above, the theory generalizes to measures that are not

probability measures. Taking fµ (x, y) = 1 in Proposition 4.5.7 gives the
inequality
+
Hµ (x | blog µ(X )c) < log µ(X ),
with a physical meaning when µ is the phase space measure. Using (4.5.6),
this implies
+
Hµ (x) < log µ(X ) + H(blog µ(X )c). (4.5.7)

Theorem 4.5.2 (General addition). The following inequality holds:

+
Hµ,ν (x, y) = Hµ (x | ν) + Hν ( y | x, Hµ (x | ν), µ).
+
Proof. To prove the inequality < define Gµ,ν (x, y, m) = mini¾m i + Hν ( y |
x, i, µ). This function is upper semicomputable and increasing in m. There-
fore function
Gµ,ν (x, y) = Gµ,ν (x, y, Hµ (x | ν))
is also upper semicomputable since it is obtained by substituting an upper
semicomputable function for m in Gµ,ν (x, y, m). Lemma 4.5.6 implies
+
Gµ,ν (x, y, m) = m + Hν ( y | x, m, µ),
+
Gµ,ν (x, y) = Hµ (x | ν) + Hν ( y | x, Hµ (x | ν), µ).

114
 4.5. Algorithmic entropy

Now, we have

ν y 2−m−Hν ( y|x,m,µ) ¶ 2−m ,

ν y 2−Gµ,ν (x, y) < 2−Hµ (x|µ) .
∗

∗ +
Integrating over x by µ gives µ x ν y 2−G < 1, implying Hµ,ν (x, y) < Gµ,ν (x, y)
+
by the minimality property of Hµ,ν (x, y). This proves the < half of the theo-
rem.
+
To prove the inequality > let fν (x, y, µ) = 2−Hµ,ν (x, y) . Proposition 4.5.4
implies that there is a constant c with ν y fν (x, y, µ) ¶ 2−Hµ (x|ν)+c . Let

Fν (x, µ) = Hµ (x | ν).

Proposition 4.5.7 gives (substituting y for x and (x, µ) for y):

+
Hµ,ν (x, y) = − log fν (x, y, µ) > Fν (x, µ) + Hν ( y | x, Fν (x, µ), µ),

which is what needed to be proved.

The function Hµ (ω) behaves quite differently for different kinds of mea-
sures µ. Recall the following property of complexity:
+ +
H( f (x) | y) < H(x | g( y)) < H(x). (4.5.8)

for any computable functions f , g. This implies

+
H( y) < H(x, y).

In contrast, if µ is a probability measure then

+
Hν ( y) > Hµ,ν (ω, y).

This comes from the fact that 2−Hν ( y) is a test for µ × ν.

4.5.4 INFORMATION
Mutual information has been defined in Definition 3.1.7 as I ∗ (x : y) =
+
H(x) + H( y) − H(x, y). By the Addition theorem, we have I ∗ (x : y) =
+
H( y) − H( y | x, H(x)) = H(x) − H(x | y, H( y)). The two latter expressions
show that in some sense, I ∗ (x : y) is the information held in x about y as

115
4. GENERALIZATIONS

well as the information held in y about x. (The terms H(x), H( y) in the

conditions are logarithmic-sized corrections to this idea.) Using (4.3.5), it is
interesting to view mutual information I ∗ (x : y) as a deficiency of random-
ness of the pair (x, y) in terms of the expression dµ , with respect to m × m:

I ∗ (x : y) = H(x) + H( y) − H(x, y) = dm×m (x, y).

Taking m as a kind of “neutral” probability, even if it is not quite such, al-

lows us to view I ∗ (x : y) as a “deficiency of independence”. Is it also true
+
that I ∗ (x : y) = dm×m (x)? This would allow us to deduce, as Levin did,
“information conservation” laws from randomness conservation laws.2
Expression dm×m (x) must be understood again in the sense of compactifi-
cation, as in Section 4.3. There seem to be two reasonable ways to compact-
ify the space N×N: we either compactify it directly, by adding a symbol ∞, or
we form the product N × N. With either of them, preserving Theorem 4.6.1,
+
we would have to check whether H(x, y | m × m) = H(x, y). But, knowing
the function m(x) × m( y) we know the function x 7→ m(x) = m(x) × m(0),
∗

hence also the function (x, y) 7→ m(x, y) = m(〈x, y〉). Using this knowledge,
it is possible to develop an argument similar to the proof of Theorem 4.3.2,
+
showing that H(x, y | m × m) = H(x, y) does not hold.

Question 1. Is there a neutral measure M with the property that I ∗ (x : y) =

d M ×M (x, y)? Is this true maybe for all neutral measures M ? If not, how far
apart are the expressions d M ×M (x, y) and I ∗ (x : y) from each other?

4.6 RANDOMNESS AND COMPLEXITY

We have seen in the discrete case that complexity and randomness are closely
related. The connection is more delicate technically in the continuous case,
but its exploration led to some nice results.

4.6.1 DISCRETE SPACE

It is known that for computable µ, the test dµ (x) can be expressed in terms
of the description complexity of x (we will prove these expressions below).
2
We cannot use the test tµ for this, since it can be shown easily that it does not to obey
randomness conservation.

116
 4.6. Randomness and complexity

Assume that X is the (discrete) space of all binary strings. Then we have

dµ (x) = − log µ(x) − H(x) + O(H(µ)). (4.6.1)

The meaning of this equation is the following. The expression − log µ(x)
is an upper bound (within O(H(µ))) of the complexity H(x), and nonran-
domness of x is measured by the difference between the complexity and this
upper bound. Assume that X is the space of infinite binary sequences. Then
equation (4.6.1) must be replaced with

dµ (x) = sup − log µ(x ¶n ) − H(x ¶n ) + O(H(µ)) .

(4.6.2)
n

For noncomputable measures, we cannot replace O(H(µ)) in these relations

with anything finite, as shown in the following example. Therefore however
attractive and simple, exp(− log µ(x) − H(x)) is not a universal uniform test
of randomness.

Proposition 4.6.1. There is a measure µ over the discrete space X of bi-

nary strings such that for each n, there is an x with dµ (x) = n − H(n) and
+
− log µ(x) − H(x) < 0.

Proof. Let us treat the domain of our measure µ as a set of pairs (x, y). Let
x n = 0n , for n = 1, 2, . . . . For each n, let yn be some binary string of length n
with the property H(x n , yn ) > n. Let µ(x n , yn ) = 2−n . Then − log µ(x n , yn ) −
H(x n , yn ) ¶ n − n = 0. On the other hand, let t µ (x, y) be the test nonzero
only on strings x of the form x n :

m(n)
t µ (x n , y) = P .
z∈B n µ(x n , z)

The form of the definition ensures semicomputability and we also have

X X
µ(x, y)t µ (x, y) ¶ m(n) < 1,
x, y n

∗
therefore t µ is indeed a test. Hence tµ (x, y) > t µ (x, y). Taking logarithms,
+
dµ (x n , yn ) > n − H(n).
The same example shows that the test defined as exp(− log µ(x) − H(x))
over discrete sets, does not satisfy the randomness conservation property.

117
4. GENERALIZATIONS

Proposition 4.6.2. The test defined as fµ (x) = exp(− log µ(x) − H(x)) over
discrete spaces X does not obey the conservation of randomness.

Proof. Let us use the example of Proposition 4.6.1. Consider the function π :
(x,
P y) 7→ x. The image of the measure µ under−nthe projection is (πµ)(x) =
y µ(x, y). Thus, (πµ)(x n ) = µ(x n , yn ) = 2 . Then we have seen that
log fµ (x n , yn ) ¶ 0. On the other hand,
+
log fπµ (π(x n , yn )) = − log(πµ)(x n ) − H(x n ) = n − H(n).

Thus, the projection π takes a random pair (x n , yn ) into an object x n that is

very nonrandom (when randomness is measured using the tests fµ ).

In the example, we have the abnormal situation that a pair is random but
one of its elements is nonrandom. Therefore even if we would not insist on
universality, the test exp(− log µ(x) − H(x)) is unsatisfactory.
Looking into the reasons of the nonconservation in the example, we will
notice that it could only have happened because the test fµ is too special. The
fact that − log(πµ)(x n ) − H(x n ) is large should show that the pair (x n , yn )
can be enclosed into the “simple” set {x n } × Y of small probability; unfortu-
nately, this observation does not reflect on − log µ(x, y) − H(x, y) (it does
for computable µ).
It is a natural idea to modify equation (4.6.1) in such a way that the
complexity H(x) is replaced with H(x | µ). However, this expression must
be understood properly. We need to use the definition of H(x) as − log m(x)
directly, and not as prefix complexity.
Let us mention the following easy fact:
+
Proposition 4.6.3. If µ is a computable measure then H(x | µ) = H(x). The
+
constant in = depends on the description complexity of µ.

Theorem 4.6.1. If X is the discrete space Σ∗ then we have

+
dµ (x) = − log µ(x) − H(x | µ). (4.6.3)

Note that in terms of the algorithmic entropy notation introduced

in (4.5.2), this theorem can be expressed as
+
Hµ (x) = H(x | µ) + log µ(x).

118
 4.6. Randomness and complexity

Proof. In exponential notation, equation (4.6.3) can be written as tµ (x) =

∗

∗
m(x | µ)/µ(x). Let us prove > first. We will show that the right-hand side of
∗
this inequality is a test, and hence < tµ (x). However, the right-hand side is
clearly lower semicomputable in (x, µ) and when we “integrate” it (multiply
it by µ(x) and sum it), its sum is ¶ 1; thus, it is a test.
∗
Let us prove < now. The expression tµ (x)µ(x) is clearly lower semicom-
+
putable in (x, µ), and its sum is ¶ 1. Hence, it is < m(x | µ).
Remark 4.6.4. It important not to consider relative computation with respect
to µ as oracle computation in the ordinary sense. Theorem 4.3.1 below will
show the existence of a measure with respect to which every element is ran-
dom. If randomness is defined using µ as an oracle then we can always find
elements nonrandom with respect to µ.
For similar reasons, the proof of the Coding Theorem does not transfer to
the function H(x | µ) since an interpreter function should have the property
of intensionality, depending only on µ and not on the sequence representing
it. (It does transfer without problem to an oracle version of H µ (x).) The
Coding Theorem still may hold, at least in some cases: this is currently not
known. Until we know this, we cannot interpret H(x | µ) as description
complexity in terms of interpreters and codes.
(Thanks to Alexander Shen for this observation: this remark corrects an
error in the paper [21].) ù

4.6.2 NON-DISCRETE SPACES

For non-discrete spaces, unfortunately, we can only provide a less intuitive

expression.

Proposition 4.6.5. let X = (X , d, D, α) be a complete computable metric space,

and let E be the enumerated set of bounded Lipschitz functions introduced
in (A.1.3), but for the space M(X) × X. The uniform test of randomness tµ (x),
can be expressed as
X m( f | µ)
tµ (x) = f (µ, x) y
∗
. (4.6.4)
f ∈E
µ f (µ, y)
∗
Proof. For >, we will show that the right-hand side of the inequality is a test,
∗
and hence < tµ (x). For simplicity, we skip the notation about the enumera-
tion of E and treat each element f as its own name. Each term of the sum is

119
4. GENERALIZATIONS

clearly lower semicomputable in ( f , x, µ), hence the sum is lower semicom-

putable in (x, µ). It remains to show that the µ-integral of the sum is ¶ 1.
But, the µ-integral of the generic term is ¶ m( f | µ), and the sum of these
terms is ¶ 1 by the definition of the function m(· | ·). Thus, the sum is a test.
∗
For <, note that (µ, x) 7→ tµ (x), as a lower semicomputable function,
is the supremum of P x by f i (µ, x),
P functions in E . Denoting their differences
we have tµ (x) = i f i (µ, x). The test property implies i µ f i (µ, x) ¶ 1.
Since the function (µ, i) 7→ µ x f i (µ, x) is lower semicomputable, this implies
∗
µ x f i (µ, x) < m(i | µ), and hence
∗ m(i | µ)
f i (µ, x) < f i (µ, x) .
µ x f i (µ, x)
It is easy to see that for each f ∈ E we have
X
m(i | µ) ¶ µ( f | µ),
i: f i = f

which leads to (4.6.4).

∗
Remark 4.6.6. If we only want the > part of the result, then E can be re-
placed with any enumerated computable sequence of bounded computable
functions. ù

4.6.3 INFINITE SEQUENCES

In case of the space of infinite sequences and a computable measure, The-

orem 2.3.4 gives a characterization of randomness in terms of complexity.
This theorem does not seem to transfer to a more general situation, but un-
der some conditions, at least parts of it can be extended.
For arbitrary measures and spaces, we can say a little less:
Proposition 4.6.7. For all measures µ ∈ MR (X ), for the deficiency of random-
ness dµ (x), we have
+
dµ (x) > sup − log µ(x ¶n ) − H(x ¶n | µ) .


(4.6.5)
n

Proof. Consider the function

X m(s | µ) X m(x ¶n | µ) m(x ¶n | µ)
fµ (x) = 1Γs (x) = ¾ sup .
s
µ(Γs ) n
µ(x ¶n ) n µ(x ¶n )

120
 4.6. Randomness and complexity

The function (µ, x) 7→ fµ (x) is clearly lower semicomputable and satisfies

µ x fµ (x) ¶ 1, and hence
+ +
dµ (x) > log f (x) > sup − log µ(x ¶n ) − H(x ¶n | µ) .


n

Definition 4.6.1. Let MR (X ) be the set of measures µ with µ(X ) = R. ù

+
We will be able to prove the > part of the statement of Theorem 2.3.4
in a more general space, and without assuming computability. Assume that
a separating sequence b1 , b2 , . . . is given as defined in Subsection 4.7, along
with the set X 0 . For each x ∈ X 0 , the binary sequence x 1 , x 2 , . . . has been
defined. Let
X
µ(Γs ) = R − { µ(Γs0 ) : l(s) = l(s0 ), s0 6= s }.

Then (s, µ) 7→ µ(Γs ) is lower semicomputable, and (s, µ) 7→ µ(Γs ) is upper

semicomputable. And, every time that the functions bi (x) form a partition
with µ-continuity, we have µ(Γs ) = µ(Γs ) for all s.

Theorem 4.6.2. Suppose that the space X is effectively compact. Then for all
computable measures µ ∈ MR0 (X ), for the deficiency of randomness dµ (x), the
characterization (2.3.4) holds.
+
Proof. The proof of part > of the inequality follows directly from Proposi-
tion 4.6.7, just as in the proof of Theorem 2.3.4.
+
The proof of < is also similar to the proof of that theorem. The only
part that needs to be reproved is the statement that for every lower semi-
computable function f over X , there are computable sequences yi ∈ N∗ and
qi ∈ Q with f (x) = supi qi 1 yi (x). This follows now, since according to Propo-
sition 4.7.3, the cells Γ y form a basis of the space X .

4.6.4 BERNOULLI TESTS

In this part, we will give characterize Bernoulli sequences in terms of their

complexity growth.
Recall Definition 4.2.6: A function f : B∗ → R is a combinatorial Bernoulli
test if

121
4. GENERALIZATIONS

(a) It is lower semicomputable.

(b) It is monotonic with respect to the prefix relation.
n

(c) For all 0 ¶ k ¶ n we have x∈B(n,k) f (x) ¶ k .
P

According to Theorem 4.2.3, a sequence ξ is nonrandom with respect to all

Bernoulli measures if and only if supn b(ξ¶n ) = ∞, where b(x) is a combina-
torial Bernoulli test.
We need some definitions.
Pn
Definition 4.6.2. For a finite or infinite sequence x let Sn (x) = i=1
x(i).
n
k
For 0 ¶ p ¶ 1 and integers 0 ¶ k ¶ n, denote B p (n, k) = k p (1 − p)n−k .
An upper semicomputable function D : N2 → N, defined for n ¾ 1, 0 ¶
k ¶ n will be called a gap function if
n
XX
B p (n, k)2−D(n,k) ¶ 1 (4.6.6)
n¾1 k=0

holds for all 0 ¶ p ¶ 1. A gap function D(n, k) is optimal if for every other
gap function D0 (n, k) there is a c D0 with D(n, k) ¶ D0 (n, k) + c D0 . ù
+
Proposition 4.6.8. There is an optimal gap function D(n, k) < H(n).

Proof. The existence is proved using the technique of Theorem 4.1.1. For the
inequality it is sufficient to note that H(n) is a gap function. Indeed, we have
n
XX X
B p (n, k)2−H(n) = 2−H(n) ¶ 1.
n¾1 k=0 n¾1

Definition 4.6.3. Let us fix an optimal gap function and denote it by ∆(n, k).
ù

Now we can state the test characterization theorem for Bernoulli tests.

Theorem 4.6.3. Denoting by b(ξ) the universal class test for the Bernoulli
sequences, we have b(ξ) = b(ξ), where
∗

n
 
log b(ξ) = sup log − H(ξ¶n | n, k, ∆(n, k)) − ∆(n, k),
n k

with k = Sn (ξ).

122
 4.6. Randomness and complexity

Proof. Let δ(n, k) = 2−∆(n,k) .

Claim 4.6.9. Consider lower semicomputable functions γ : B∗ → R+ such that
for all n, k we have
X
γ( y) ¶ 2−∆(n,k) .
y∈B(n,k)

Among these functions there is one that is optimal (maximal to within a multi-
plicative constant). Calling it δ( y) we have

δ( y) = δ(n, k) · m( y | n, k, ∆(n, k)).

∗

Thus, the right-hand side is equal, within a multiplicative constant, to a

lower semicomputable function of y.

Proof. This follows immediately from Proposition 4.5.7. with ν = # (the

counting measure) over the sets B(n, k).
∗
Let us show b(ξ) < b(ξ). We have with k = Sn (ξ) in the first line:

n
 
b(ξ) = sup m(ξ¶n | n, k, ∆(n, k))δ(n, k)
n¾1 k
n  
X n X
= sup δ(n, k) 1 y (ξ)m( y | n, k, ∆(n, k))
n¾1 k=0 k y∈B(n,k)
n   X
XX n
¶ 1 y (ξ)δ(n, k)m( y | n, k, ∆(n, k))
n¾1 k=0
k y∈B(n,k)
n   X
XX n
= 1 y (ξ)δ( y),
∗

n¾1 k=0
k y∈B(n,k)

using the notation of Claim 4.6.9 above. Let t(ξ) denote the right-hand side
here, which is thus a lower semicomputable function. We have for all p:
n
XX X
B pξ t(ξ) = B p (n, k)δ(n, k) m( y | n, k, ∆(n, k)) ¶ 1,
∗

n¾1 k=0 y∈B(n,k)

∗ ∗
so t(ξ) > b(ξ) is a Bernoulli test, showing b(ξ) < b(ξ).

123
4. GENERALIZATIONS

∗
To show b(ξ) < b(ξ) we will follow the method of the proof of Theo-
rem 2.3.4. Replace b(ξ) with a rougher version:

1
b0 (ξ) = max{ 2n : 2n < b(ξ) },
2
then we have 2b0 < b. There are computable sequences yi ∈ B∗ and
ki ∈ N with b0 (ξ) = supi 2ki 1 yi (ξ) with the property that if i < j and
1 yi (ξ) = 1 y j (ξ) = 1 then ki < k j . As in the imitated proof, we have
2b0 (ξ) ¾ i 2ki 1 yi (ξ). The function γ( y) =
P P ki
yi = y 2 is lower semicom-
putable. We have
X X n
XX X
b ¾ 2b (ξ) ¾
0
2 1 yi (ξ) =
ki
1 y (ξ)γ( y) = 1 y (ξ)γ( y).
i y∈N∗ n k=0 y∈B(n,k)
(4.6.7)
n

y∈B(n,k) γ( y)
P
By Theorem 4.2.3 we can assume ¶ k
. Let
 −1
n
δ ( y) = γ( y)
0
¶ 1,
k
X
δ0 (n, k) = δ0 ( y).
y∈B(n,k)

Since 1 ¾ B p b ¾ B p (2b0 ) for all p, we have

n
XX X n
XX X
1¾ γ( y)B pξ 1 y (ξ) = p k (1 − p)n−k γ( y)
n k=0 y∈B(n,k) n k=0 y∈B(n,k)
n
XX X n
XX
= B p (n, k) δ ( y) =
0
B p (n, k)δ0 (n, k).
n k=0 y∈B(n,k) n k=0

∗
Thus δ0 (n, k) is a gap function, hence δ0 (n, k) < 2−∆(n,k) , and by Claim 4.6.9
we have
 −1
n ∗
γ( y) = δ0 ( y) < δ( y) = δ(n, k) · m( y | n, k, ∆(n, k)).
∗

k
∗
Substituting back into (4.6.7) finishes the proof of b(ξ) < b(ξ).

124
 4.7. Cells

4.7 CELLS
This section allows to transfer some of the results on sequence spaces to more
general spaces, by encoding the elements into sequences. The reader who is
only interested in sequences can skip this section.

4.7.1 PARTITIONS
The coordinates of the sequences into which we want to encode our elements
will be obtained via certain partitions.
Recall from Definition A.2.16 that a measurable set A is said to be a µ-
continuity set if µ(∂ A) = 0 where ∂ A is the boundary of A.

Definition 4.7.1 (Continuity partition). Let f : X → R be a bounded com-

putable function, and let α1 < · · · < αk be rational numbers, and let µ be a
computable measure with the property that µ f −1 (α j ) = 0 for all j.
In this case, we will say that α j are µ-continuity points of f . Let α0 = −∞,
αk+1 = ∞, and for j = 0, . . . , k, let Let U j = f −1 (( j, j + 1)). The sequence
of disjoint computably enumerable open sets (U0 , . . . , Uk ) will be called the
partition generated by f , α1 , . . . , αk .
If we have several partitions (Ui0 , . . . , Ui,k ), generated by different func-
tions f i (i = 1, . . . , m) and cutoff sequences (αi j : j = 1, . . . , ki ) made up of
µ-continuity points of f i then we can form a new partition generated by all
possible intersections

Vj1 ,..., jn = U1, j1 ∩ · · · ∩ Um, jm .

A partition of this kind will be called a continuity partition. The sets Vj1 ,..., jn
will be called the cells of this partition. ù

The following is worth noting.

Proposition 4.7.1. In a partition as given above, the values µVj1 ,..., jn are com-
putable from the names of the functions f i and the cutoff points αi j .

Proof. Straightforward.

Let us proceed to defining cells.

125
4. GENERALIZATIONS

Definition 4.7.2 (Separating sequence). Assume that a computable se-

quence of functions b1 (x), b2 (x), . . . over X is given, with the property that
for every pair x 1 , x 2 ∈ X with x 1 6= x 2 , there is a j with b j (x 1 ) · b j (x 2 ) < 0.
Such a sequence will be called a separating sequence. Let us give the corre-
spondence between the set BN of infinite binary sequences and elements of
the set
X 0 = { x ∈ X : b j (x) 6= 0, j = 1, 2, . . . }.
For a binary string s1 · · · sn = s ∈ B∗ , let

Γs

be the set of elements of X with the property that for j = 1, . . . , n, if s j = 0

then b j (ω) < 0, otherwise b j (ω) > 0.
The separating sequence will be called µ-continuity sequence if µ(X 0 ) =
0. ù

This correspondence has the following properties.

(a) ΓΛ = X .
(b) For each s ∈ B, the sets Γs0 and Γs1 are disjoint and their union is con-
tained in Γs .
(c) For x ∈ X 0 , we have {x} = x∈Γs Γs .
T

Definition 4.7.3 (Cells). If string s has length n then Γs will be called a

canonical n-cell, or simply canonical cell, or n-cell. From now on, whenever
Γ denotes a subset of X , it means a canonical cell. We will also use the
notation
l(Γs ) = l(s).
ù

The three properties above say that if we restrict ourselves to the set X 0
then the canonical cells behave somewhat like binary subintervals: they di-
vide X 0 in half, then each half again in half, etc. Moreover, around each point,
these canonical cells become “arbitrarily small”, in some sense (though, they
may not be a basis of neighborhoods). It is easy to see that if Γs1 , Γs2 are two
canonical cells then they either are disjoint or one of them contains the other.
If Γs1 ⊂ Γs2 then s2 is a prefix of s1 . If, for a moment, we write Γ0s = Γs ∩ X 0
then we have the disjoint union Γ0s = Γ0s0 ∪ Γ0s1 .
Let us use the following notation.

126
 4.7. Cells

Definition 4.7.4. For an n-element binary string s, for x ∈ Γs , we will write

s = x ¶n = x 1 · · · x n , µ(s) = µ(Γs ).

The 2n cells (some of them possibly empty) of the form Γs for l(s) = n form
a partition
Pn
of X 0 . ù

Thus, for elements of X 0 , we can talk about the n-th bit x n of the descrip-
tion of x: it is uniquely determined.
Examples 4.7.2.
1. If X is the set of infinite binary sequences with its usual topology, the
functions bn (x) = x n − 1/2 generate the usual cells, and X0 = X.
2. If X is the interval [0, 1], let bn (x) = − sin(2n πx). Then cells are open
intervals of the form (k · 2−n , (k + 1) · 2n ), the correspondence between
infinite binary strings and elements of X 0 is just the usual representation
of x as the binary decimal string 0.x 1 x 2 . . . .
ù
When we fix canonical cells, we will generally assume that the partition
chosen is also “natural”. The bits x 1 , x 2 , . . . could contain information about
the point x in decreasing order of importance from a macroscopic point of
view. For example, for a container of gas, the first few bits may describe,
to a reasonable degree of precision, the amount of gas in the left half of
the container, the next few bits may describe the amounts in each quarter,
the next few bits may describe the temperature in each half, the next few
bits may describe again the amount of gas in each half, but now to more
precision, etc. From now on, whenever Γ denotes a subset of X , it means a
canonical cell. From now on, for elements of X 0 , we can talk about the n-th
bit x n of the description of x: it is uniquely determined.
The following observation will prove useful.

Proposition 4.7.3. Suppose that the space X is effectively compact3 and we

have a separating sequence bi (x) as given above. Then the cells Γs form a basis
of the space X.
3
It was noted by Hoyrup and Rojas that the qualification “effectively” is necessary here.

127
4. GENERALIZATIONS

Proof. We need to prove that for every ball B(x, r), the there is a cell x ∈
Γs ⊂ B(x, r). Let C be the complement of B(x, r). For each point y of C,
there is an i such that bi (x) · bi ( y) < 0. In this case, let J 0 = { z : bi (z)
S < 0 },
J 1 = { z : bi (z) > 0 }. Let J( y) = J p such that y ∈ J p . Then C ⊂ y J( y),
and compactness implies that there is a finite sequence y1 , . . . , yk with C ⊂
Sk
j=1
J( y j ). Clearly, there is a cell

k
[
x ∈ Γs ⊂ B(x, r) \ J( y j ).
j=1

4.7.2 COMPUTABLE PROBABILITY SPACES

If a separating sequence is given in advance, we may restrict attention to the

class of measures that make our sequence a µ-continuity sequence:

Definition 4.7.5. Let M 0 (X ) be the set of those probability measures µ in

M (X ) for which µ(X \ X 0 ) = 0. ù

On the other hand, for each computable measure µ, a computable sep-

arating sequence can be constructed that is a µ-continuity sequence. Re-
call that B(x, r) is the ball of center x and radius r. Let D = {s1 , s2 , . . . } =
{α(1), α(2), . . . } be the canonical enumeration of the canonical dense set D.

Theorem 4.7.1 (Hoyrup-Rojas). There is a sequence of uniformly computable

reals (rn )n∈N such that (B(si , rn ))i,n is a basis of balls that are µ-continuity sets.
This basis is constructively equivalent to the original one, consisting of all balls
B(si , r), r ∈ Q.

Corollary 4.7.4. There is a computable separating sequence with the µ-

continuity property.

Proof. Let us list all balls B(si , rn ) into a single sequence B(sik , rnk ). The func-
tions

bk (x) = d(sik , x) − rnk

give rise to the desired sequence.

128
 4.7. Cells

For the proof of the theorem, we use some preparation. Recall from Defi-
nition A.2.3 that an atom is a point with positive measure.

Lemma 4.7.5. Let X be R or R+ or [0, 1]. Let µ be a computable probability

measure on X . Then there is a sequence of uniformly computable reals (x n )n
which is dense in X and contains no atoms of µ.

Proof. Let I be a closed rational interval. We construct x ∈ I with µ({x}) = 0.

To do this, we construct inductively a nested sequence of closed intervals Jk
of measure < 2−k+1 , with J0 = I. Suppose Jk = [a, b] has been constructed,
with µ(Jk ) < 2−k+1 . Let m = (b − a)/3: one of the intervals [a, a + m] and
[b − m, b] must have measure < 2−k , and we can find it effectively—let it be
Jk+1 .
From a constructive enumeration (I n )n of all the dyadic intervals, we can
construct x n ∈ I n uniformly.

Corollary 4.7.6. Let (X, µ) be a computable metric space with a computable

measure and let ( f i )i be a sequence of uniformly computable real valued func-
tions on X . Then there is a sequence of uniformly computable reals (x n )n that
is dense in R and such that each x n is a µ-continuity point of each f i .

P Consider
Proof. the uniformly computable measures µi = µ ◦ f i−1 and define
ν = i 2 µi . It is easy to see that ν is a computable measure and then, by
−i

Lemma 4.7.5, there is a sequence of uniformly computable reals (x n )n which

is dense in R and contains no atoms of ν. Since ν(A) = 0 iff µi (A) = 0 for all
i, we get µ({ f i−1 (x n )}) = 0 for all i, n.
Proof of Theorem 4.7.1. Apply Corollary 4.7.6 to f i (x) = d(si , x).
Since every ball B(si , r) can be expressed as a computably enumerable
union of the balls of the type B(si , rn ) just constructed, the two bases are
constructively equivalent.

129
5 EXERCISES AND PROBLEMS

Exercise 1. Define, for any two natural numbers r, s, a standard encoding

cnvsr of base r strings x into base s strings with the property

log r
|cnvsr (x)| ¶ |x| + 1. (5.0.1)
log s

Solution. We will use X = N N , X r = Z rN for the sets of infinite strings of

natural numbers and r-ary digits respectively. For a sequence p ∈ X r , let
[p] r denote the real number in the interval [0, 1] which 0.p denotes in the
base r number system. For p in S r , let [p] r = { [pq] r : q ∈ X r }.
For the r-ary string x, let v be the size of the largest s-ary intervals [ y]s
contained in the r-ary interval [x] r . If [z]s is the leftmost among these
intervals, then let cnvsr (x) = z. This is a one-to-one encoding. We have
r −|x| < 2sv, since any 2s consecutive s-ary intervals of length v contain an
s-ary interval of length sv. Therefore
log r log 2
|z| = − logs v < |x| +1+
log s log s
hence, since 2 ¶ s and |x| is integer, we have the inequality (5.0.1).

Exercise 2. A function A from S r × S to S is called an r-ary interpreter. Prove

the following generalization of the Invariance Theorem. For any s, there is a
p.r. s-ary interpreter U such that for any p.r. interpreter A there is a constant
c < ∞ such that for all x, y we have

KU (x | y) ¶ KA(x | y) + c. (5.0.2)

Solution. Let V : Z∗s × Z∗s × S → S be a partial recursive function which is

universal: such that for any p.r. s-ary interpreter A, there is a string a such
that for all p, x, we have A(p, x) = V (a, p, x).

131
5. EXERCISES AND PROBLEMS

The machine computing U(p, x) tries to decompose p into uo v and out-

puts V (u, v, x). Let us verify that U is optimal. Let A be a p.r. r-ary interpreter,
B an s-ary p.r. interpreter such that B(cnvsr (p), x) = A(p, x) for all p, x, a a bi-
nary string such that B(p, x) = U(a, p, x) for all p, x. Let x, y be two strings.
If KA(x | y) = ∞, then the inequality (5.0.2) holds trivially. Otherwise, let p
be a binary string of length KA(x | y)/ log r with A(p, y) = x. Then

U(a o cnvsr (p), y) = V (a, cnvsr (p), y) = B(cnvsr (p), y) = A(p, y) = x.

Since
|cnvsr (p)| ¶ |p| log r/ log s + 1 = KA(x | y)/ log s + 1,
we have

KU (x | y) ¶ (2|a| + KA(x | y)/ log s + 1) log s = KA(x | y) + (2|a|) + 1) log s.

Exercise 3 (Snm -theorem). Prove that there is a binary string b such that
U(p, q, x) = U(b o p o q, x) holds for all binary strings p, q and arbitrary strings
x.

Exercise 4. (Schnorr) Notice that, apart from the conversion between rep-
resentations, what our optimal interpreter does is the following. It treats
the program p as a pair (p(1), p(2)) whose first member is the Gödel num-
ber of some interpreter for the universal p.r. function V (p1 , p2 , x), and the
second argument as a program for this interpreter. Prove that indeed, for
any recursive pairing function w(p, q), if there is a function f such that
|w(p, q)| ¶ f (p) + |q| then w can be used to define an optimal interpreter.
+
Exercise 5. Refute the inequality K(x, y) < K(x) + K( y).

Exercise 6. Prove the following sharpenings of Theorem 1.4.2.

+
K(x, y) < J(K(x)) + K( y | x, K(x)),
+
K(x, y) < K(x) + J(K( y | x, K(x))).

Prove Theorem 1.4.2 from here.

+
Exercise 7. (Schnorr) Prove K(x + K(x)) = K(x). Can you generalize this
result?

132
 +
Exercise 8. (Schnorr) Prove that if m < n then m + K(m) < n + K(n).

Exercise 9. Prove
n + n n 1 n
 
log = k log + (n − k) log + log .
k k n−k 2 k(n − k)
Exercise 10. (Kamae) Prove that for any natural number k there is a string
x such that for for all but finitely many strings y, we have

K(x) − K(x | y) ¾ k.

In other words, there are some strings x such that almost any string y con-
tains a large amount of information about them.

Solution. Strings x with this property are for example the ones which con-
tain information obtainable with the help of any sufficiently large number.
Let E be a r.e. set of integers. Let e0 e1 e2 . . . be the infinite string which is the
characteristic function of E, and let x(k) = e0 . . . e2k . We can suppose that
K(x(k)) ¾ k. Let n1 , n2 , . . . be a recursive enumeration of E without repeti-
tion, and let α(k) = max{i : ni ¶ 2k }. Then for any number t ¾ α(k) we have
+
K(x(k) | t) < log k. Indeed, a binary string of length k describes the number
k. Knowing t we can enumerate n1 , . . . , n t and thus learn x(k). Therefore
+
with any string y of length ¾ α(k) we have K(x) − K(x | y) > k − log k.

Exercise 11.
(a) Prove that a real function f is computable iff there is a recursive function
g(x, n) with rational values, and | f (x) − g(x, n)| < 1/n.
(b) Prove that a function f is semicomputable iff there exists a recursive
function with rational values, (or, equivalently, a computable real funct-
ion) g(x, n) nondecreasing in n, with f (x) = limn→∞ g(x, n).

Exercise 12. Prove that in Theorem 1.5.2 one can write “semicomputable”
for “partial recursive”.

Exercise 13. (Levin) Show that there is an upper semicomputable function

G(p, x, y) which for different finite binary strings p enumerates all upper
semicomputable functions F (x, y) satisfying the inequality (1.5.1). Prove
+
K(x | y) = inf G(p, x, y) + J(|p|).
p

133
5. EXERCISES AND PROBLEMS

Exercise 14. Prove X

m(p) = m(n).
∗

p∈Bn

Exercise 15. (Solovay) Show that we cannot find effectively infinitely many
places where some recursive upper bound of H(n) is sharp. Moreover, sup-
pose that F (n) is a recursive upper bound of H(n). Then there is no recursive
function D(n) ordering to each n a finite set of natural numbers (represented
for example as a string) larger than n such that for each n there is an x in
D(n) with F (x) ¶ H(x). Notice that the function log n (or one almost equal
to it) has this property for K(n).

Solution. Suppose that such a function exists. Then we can select a sequence
n1 < n2 < . . . of integers with the property that the sets D(ni ) are all disjoint.
Let X
a(n) = 2−F (x) .
x∈D(n)

Then the sequence a(nk ) is computable and k a(nk ) < 1. It is now easy
P

to construct a computable subsequenceP mi = nki of nk and a computable se-

quence bi such that bi /a(mi ) → ∞ and i bi < 1. Let us define the semimea-
sure µ setting
µ(x) = 2−F (x) bi /a(mi )
for any x in D(mi ) and 0 otherwise. Then for any x in D(mi ) we have
+
H(x) < − log µ(x) = F (x) − log ci where ci = bi /a(mi ) → ∞, so we arrived
a contradiction with the assumption.

Exercise 16. Prove that there is a recursive upper bound F (n) of H(n) and a
constant c with the property that there are infinitely many natural numbers n
such that for all k > 0, the quantity of numbers x ¶ n with H(x) < F (x) − k
is less than cn2−k .

Solution. Use the upper bound G found in Theorem 1.7.4 and define F (n) =
log n + G(blog nc). The property follows from the facts that log n + H(blog nc)
is a sharp upper bound for H(x) for “most” x less than n and that G(k) is
infinitely often close to H(k).

P an example of a computable sequence an > 0 of with the

Exercise 17. Give
property that n an <P∞ but for any other computable sequence bn > 0, if
bn /an → in f t y then n bn = ∞.

134
Hint: Let P
rn be a recursive, increasing sequence of rational numbers with
limn rn = x m(x) and let an = rn+1 − rn .

Exercise 18 (Universal coding). (Elias) Let f (n) = log2 n + 2 log2 log2 n.

Show that when P runs over all nonincreasing probability distributions over
N then X
lim H(P)−1 P(n) f (n) = 1.
H(P)→∞
n

Exercise 19. (Cover) Let log∗2 n = log2 n+log2 log2 n+. . . (all positive terms).
Prove that X ∗
2− log2 n < ∞,
n
+ +
hence H(n) < log∗2 n. For which logarithm bases does H(n) < log∗2 n hold?

Exercise 20. Prove that Kamae’s result in Exercise 10 does not hold for H(x |
y).

Exercise 21. Prove that for each " there is an m such that if H (P) > m then
| x P(x)H(x)/H (P) − 1| < ".
P

Exercise 22. If a finite sequence x of length n has large complexity then

its bits are certainly not predictable. Let h(k, n) = −(k/n)log(k/n) − (1 −
(k/n))log(1 − (k/n)). A quantitative relation of this sort is the following. If,
for some k > n/2, a program of length m can predict x i from x 1 , . . . , x i−1 for
at least n − k values of i then K(x) < m + nh(k, n) + o(n).

Exercise 23. This is a more open-ended problem, having to do with the

converse of the previous exercise. Show (if true) that for each c there is a
d such that predictability of x at significantly more than n/2 + c places is
equivalent to the possibility to enclose x into a set of complexity o(n) and
size n − dn.

135
A BACKGROUND FROM MATHEMATICS

A.1 TOPOLOGY
In this section, we summarize the notions and results of topology that are
needed in the main text.

A.1.1 TOPOLOGICAL SPACES

A topological space is a set of points with some kind of—not quantitatively

expressed—closeness relation among them.

Definition A.1.1. A topology on a set X is defined by a class τ of its subsets

called open sets. It is required that the empty set and X are open, and that
arbitrary union and finite intersection of open sets is open. The pair (X , τ) is
called a topological space. A set is called closed if its complement is open. ù

Having a set of open and closed sets allows us to speak about closure
operations.

Definition A.1.2. A set B is called the neighborhood of a set A if B contains

an open set that contains A. We denote by A, Ao the closure (the intersection
of all closed sets containing A) and the interior of A (the union of all open
sets in A) respectively. Let
∂ A = A \ Ao
denote the boundary of set A. ù

An alternative way of defining a topological space is via a basis.

Definition A.1.3. A basis of a topological space is a subset β of τ such that

every open set is the union of some elements of β. A neighborhood of a point

137
A. BACKGROUND FROM MATHEMATICS

is a basis element containing it. A basis of neighborhoods of a point x is a

set N of neighborhoods of x with the property that each neighborhood of x
contains an element of N . A subbasis is a subset σ of τ such that every open
set is the union of finite intersections from σ. ù

Examples A.1.1.
1. Let X be a set, and let β be the set of all points of X . The topology with
basis β is the discrete topology of the set X . In this topology, every subset
of X is open (and closed).
2. Let X be the real line R, and let βR be the set of all open intervals (a, b).
The topology τR obtained from this basis is the usual topology of the real
line. When we refer to R as a topological space without qualification, this
is the topology we will always have in mind.
3. Let X = R = R ∪ {−∞, ∞}, and let βR consist of all open intervals (a, b)
and in addition of all intervals of the forms [−∞, a) and (a, ∞]. It is clear
how the space R+ is defined.
4. Let X be the real line R. Let βR> be the set of all open intervals (−∞, b).
The topology with basis βR> is also a topology of the real line, different
from the usual one. Similarly, let βR< be the set of all open intervals (b, ∞).
5. Let Σ be a finite or countable alphabet. On the space ΣN of infinite sequen-
ces with elements in Σ, let τC = { AΣN : A ⊆ Σ∗ } be called the topology
of the Cantor space (over Σ). Note that if a set E has the form E = AΣN
where A is finite then E is both open and closed.
ù
Starting from open sets, we can define some other kinds of set that are
still relatively simple:

Definition A.1.4. A set is called a Gδ set if it is a countable intersection of

open sets, and it is an Fσ set if it is a countable union of closed sets. ù

Different topologies over the same space have a natural partial order re-
lation among them:

Definition A.1.5. A topology τ0 on X is called larger, or finer than τ if τ0 ⊇

τ. For two topologies τ1 , τ2 over the same set X , we define the topology
τ1 ∨ τ2 = τ1 ∩ τ2 , and τ1 ∧ τ2 as the smallest topology containing τ1 ∪ τ2 . In
the example topologies of the real numbers above, we have τR = τ< R
∧τ>R
. ù

138
 A.1. Topology

Most topologies used in practice have some separion property.

Definition A.1.6. A topology is said to have the T0 property if every point is

determined by the class of open sets containing it. This is the weakest one of
a number of other possible separation ù

Both our above example topologies of the real line have this property. All
topologies considered in this survey will have the T0 property. A stronger
such property is the following:

Definition A.1.7. A space is called a Hausdorff space, or T2 space, if for

every pair of different points x, y there is a pair of disjoint open sets A, B
with x ∈ A, y ∈ B. ù

The real line with topology τ>R

in Example A.1.1.4 above is not a Haus-
dorff space. A space is Hausdorff if and only if every open set is the union of
closures of basis elements.

A.1.2 CONTINUITY
We introduced topology in order to be able to speak about continuity:

Definition A.1.8. Given two topological spaces (X i , τi ) (i = 1, 2), a function

f :⊆ X 1 → X 2 is called continuous if for every open set G ⊂ X 2 its inverse
image f −1 (G) is also open. If the topologies τ1 , τ2 are not clear from the
context then we will call the function (τ1 , τ2 )-continuous. Clearly, the prop-
erty remains the same if we require it only for all elements G of a subbasis of
X 2.
If there are two continuous functions between X and Y that are inverses
of each other then the two spaces are called homeomorphic.
We say that function f is continuous at point x if for every neighborhood
V of f (x) there is a neighborhood U of x with f (U) ⊆ V . ù

Clearly, function f is continuous if and only if it is continuous in each

point.
There is a natural sense in which every subset of a topological space is
also a topological space:

Definition A.1.9. A subspace of a topological space (X , τ) is defined by a

subset Y ⊆ X , and the topology τY = { G ∩ Y : G ∈ τ }, called the induced

139
A. BACKGROUND FROM MATHEMATICS

topology on Y . This is the smallest topology on Y making the identity map-

ping x 7→ x continuous. A partial function f :⊆ X → Z with dom( f ) = Y is
continuous iff f : Y → Z is continuous. ù

Given some topological spaces, we can also form larger ones using for
example the product operation:

Definition A.1.10. For two topological spaces (X i , τi ) (i = 1, 2), we define

the product topology on their product X × Y : this is the topology defined by
the subbasis consisting of all sets G1 ×X 2 and all sets X 1 ×G2 with Gi ∈ τi . The
product topology is the smallest topology making the projection functions
(x, y) 7→ x, (x, y) 7→ y continuous. Given topological spaces X , Y, Z we call
a two-argument function f : X × Y 7→ Z continuous if it is continuous as a
functionQfrom X × Y to Z. The product topology is defined similarly for the
product i∈I X i of an arbitrary number of spaces, indexed by some index set
I. We say that a function is (τ1 , . . . , τn , µ)-continuous if it is (τ1 ×· · ·×τn , µ)-
continuous. ù

Examples A.1.2.
1. The space R × R with the product topology has the usual topology of the
Euclidean plane.
2. Let X be a set with the discrete topology, and X N the set of infinite se-
quences with elements from X , with the product topology. A basis of this
topology is provided by all sets of the form uX N where u ∈ X ∗ . The ele-
ments of this basis are closed as well as open. When X = {0, 1} then this
topology is the usual topology of infinite binary sequences.
ù
In some special cases, one of the topologies in the definition of continuity
is fixed and known in advance:

Definition A.1.11. A real function f : X 1 → R is called continuous if it is

(τ1 , τR )-continuous. ù

A.1.3 SEMICONTINUITY
For real functions, a restricted notion of continuity is often useful.

Definition A.1.12. A function f : X → R is lower semicontinuous if the set

{ (x, r) ∈ X ×R : f (x) > r } is open. The definition of upper semicontinuity is

140
 A.1. Topology

similar. Lower semicontinuity on subspaces is defined similarly to continuity

on subspaces. ù

Clearly, a real function f is continuous if and only if it is both lower

and upper semicontinuous. The requirement of lower semicontinuity of f is
equivalent to saying that for each r ∈ R, the set { x : f (x) > r } is open. This
can be seen to be equivalent to the following characterization.

Proposition A.1.3. A real function f over X = (X , τ) is lower semicontinuous

if and only if it is (τ, τ<
R
)-continuous.

Example A.1.4. The indicator function 1G (x) of an arbitrary open set G is

lower semicontinuous. ù
The following is easy to see.

Proposition A.1.5. The supremum of any set of lower semicontinuous functions

is lower semicontinuous.

The following representation is then also easy to see.

Proposition A.1.6. Let X be a topological space with basis β. The function

f : X → R+ is lower semicontinuous if and only if there is a function g : β → R+
with f (x) = sup x∈β g(β).

Corollary A.1.7. Let X be a topological space with basis β and f :⊆ X → R+

a lower semicontinuous function defined on a subset D of X . Then f can be
extended in a lower semicontinuous way to the whole space X .

Proof. Indeed by the above proposition there is a function g : β → R+ with

f (x) = sup x∈β g(β) for all x ∈ D. Let us define f by this same formula for
all x ∈ X .

In the important special case of Cantor spaces, the basis is given by the
set of finite sequences. In this case we can also require the function g(w) to
be monotonic in the words w:

Proposition A.1.8. Let X = ΣN be a Cantor space as defined in Exam-

ple A.1.1.5. Then f : X → R+ is lower semicontinuous if and only if there
is a function g : Σ∗ → R+ monotonic with respect to the relation u v v, with
f (ξ) = supuvξ g(u).

141
A. BACKGROUND FROM MATHEMATICS

A.1.4 COMPACTNESS
There is an important property of topological spaces that, when satisfied, has
many useful implications.

Definition A.1.13. Let (X , τ) be a topological space, and B a subset of X . An

open cover of B is a family of open sets whose union contains B. A subset K
of X is said to be compact if every open cover of K has a finite subcover. ù

Compact sets have many important properties: for example, a continuous

function over a compact set is bounded.
Example A.1.9.
1. Every finite discrete space is compact. An infinite discrete space X = (X , τ)
is not compact, but it can be turned into a compact space X by adding a
new element called ∞: let X = X ∪ {∞}, and τ = τ ∪ { X \ A : A ⊂
X closed }. More generally, this simple operation can be performed with
every space that is locally compact, that each of its points has a compact
neighborhood.
2. In a finite-dimensional Euclidean space, every bounded closed set is com-
pact.
3. It is known that if (Xi )i∈I is a family of compact spaces then their direct
product is also compact.
ù
There are some properties that are equivalent to compactness in simple
cases, but not always:

Definition A.1.14. A subset K of X is said to be sequentially compact if every

sequence in K has a convergent subsequence with limit in K. The space is
locally compact if every point has a compact neighborhood. ù

A.1.5 METRIC SPACES

Metric spaces are topological spaces with more structure: in them, the close-
ness concept is quantifiable. In our examples for metric spaces, and later in
our treatment of the space of probability measures, we refer to [5].

142
 A.1. Topology

Definition A.1.15. A metric space is given by a set X and a distance function

d : X × X → R+ satisfying the triangle inequality d(x, z) ¶ d(x, y) + d( y, z)
and also property that d(x, y) = 0 implies x = y. For r ∈ R+ , the sets

B(x, r) = { y : d(x, y) < r }, B(x, r) = { y : d(x, y) ¶ r }

are called the open and closed balls with radius r and center x.
A metric space is bounded when d(x, y) has an upper bound on X . ù

A metric space is also a topological space, with the basis that is the set
of all open balls. Over this space, the distance function d(x, y) is obviously
continuous.
Each metric space is a Hausdorff space; moreover, it has the following
stronger property.

Definition A.1.16. A topological space is said to have the T3 property if for

every pair of different points x, y there is a continuous function f : X → R
with f (x) 6= f ( y). ù

To see that metric spaces are T3 , take f (z) = d(x, z).

Definition A.1.17. A topological space is called metrizable if its topology can

be derived from some metric space. ù

It is known that a topological space is metrizable if and only if it has the

T3 property.

Notation A.1.18. For an arbitrary set A and point x let

d(x, A) = inf d(x, y),

y∈A
"
A = { x : d(x, A) < " }. (A.1.1)

Examples A.1.10.
1. A discrete topological space X can be turned into a metric space as fol-
lows: d(x, y) = 0 if x = y and 1 otherwise.
2. The real line with the distance d(x, y) = |x − y| is a metric space. The
topology of this space is the usual topology τR of the real line.

143
A. BACKGROUND FROM MATHEMATICS

3. The space R × R with the Euclidean distance gives the same topology as
the product topology of R × R.
4. An arbitrary set X with the distance d(x, y) = 1 for all pairs x, y of dif-
ferent elements, is a metric space that induces the discrete topology on
X.
5. Let X be a bounded metric space, and let Y = X N be the set of in-
P −i sequences x = (x 1 , x 2 , . . . ) with distance function d (x, y) =
finite N

i 2 d(x i , yi ). The topology of this space is the same as the product

topology defined in Example A.1.2.2.
6. Specializing the above example, if Σ is the discrete space defined in Ex-
ample 1 above then we obtain a metrization of the Cantor space of Exam-
ple A.1.15. For every finite sequence x ∈ Σ∗ and every infinite sequence
ξ w x the ball B(ξ, 2−l(x) ) is equal to a basis element that is the open-
closed cylinder set xΣN .
7. Let X be a metric space, and let Y = X N be the set of infinite bounded
sequences x = (x 1 , x 2 , . . . ) with distance function d(x, y) = supi d(x i , yi ).
8. Let X be a topological space, and let C(X ) be the set of bounded continu-
ous functions over X with distance function d 0 ( f , g) = sup x d( f (x), g(x)).
A special case is C[0, 1] where the interval [0, 1] of real numbers has the
usual topology.

P 2sequences x = (x 1 , x 2 , . . . ) of real numbers

9. Let l2 be the set of infinite
with the property that i x i < ∞. The metric is given by the distance
d(x, y) = ( i |x i − yi | ) .
P 2 1/2

ù
In metric spaces, certain previously defined topological objects have
richer properties.
Examples A.1.11. Each of the following facts holds in metric spaces and is
relatively easy to prove.
1. Every closed set is a Gδ set (and every open set is an Fσ set).
2. A set is compact if and only if it is sequentially compact.
3. A set is compact if and only if it is closed and for every ", there is a finite
set of "-balls (balls of radius ") covering it.
ù
In metric spaces, the notion of continuity can be strengthened.

144
 A.1. Topology

Definition A.1.19. A function f : X → Y between metric spaces X , Y is

uniformly continuous if for each " > 0 there is a δ > 0 such that dX (a, b) < δ
implies dY ( f (a), f (b)) < ". ù

It is known that over a compact metric space, every continuous function

is uniformly continuous.

Definition A.1.20 (Lipschitz). Given a function f : X → Y between met-

ric spaces and β > 0, let Lipβ (X , Y ) denote the set of functions (called the
Lipschitz(β) functions, or simply Lipschitz functions) satisfying

dY ( f (x), f ( y)) ¶ β dX (x, y). (A.1.2)

Let Lip(X ) = Lip(X , R) = β Lipβ be the set of real Lipschitz functions over
S

X. ù

As an example, every differentiable real function f (x) with | f 0 (x)| ¶ 1

everywhere is a Lipschitz(1) function.
All these functions are uniformly continuous.
We introduce a certain fixed, enumerated sequence of Lipschitz functions
that will be used frequently as “building blocks” of other functions.

Definition A.1.21 (Hat functions). Let

gu,r," (x) = |1 − |d(x, u) − r|+ /"|+ .

This is a continuous function that is 1 in the ball B(u, r), it is 0 outside the
ball B(u, r + "), and takes intermediate values in between. It is clearly a
Lipschitz(1/") function.
If a dense set D is fixed, let F0 = F0 (D) be the set of functions of the
form gu,r,1/n where u ∈ D, r is rational, n = 1, 2, . . . . Let F1 = F1 (D) be the
maximum of a finite number of elements of F0 (D). Each element f of F1 is
bounded between 0 and 1. Let

E = E (D) = {g1 , g2 , . . . } ⊃ F1 (A.1.3)

be the smallest set of functions containing F0 and the constant 1, and closed
under ∨, ∧ and rational linear combinations. For each element of E , from its
definition we can compute a bound β such that f ∈ Lipβ . ù

For the effective representation of points in a topological space the fol-

lowing properties are important.

145
A. BACKGROUND FROM MATHEMATICS

Definition A.1.22. A topological space has the first countability property if

each point has a countable basis of neighborhoods. ù

Every metric space has the first countability property since we can restrict
ourselves to balls with rational radius.

Definition A.1.23. Given a topological space (X , τ) and a sequence x =

(x 1 , x 2 , . . . ) of elements of X , we say that x converges to a point y if for every
neighborhood G of y there is a k such that for all m > k we have x m ∈ G. We
will write y = limn→∞ x n . ù

It is easy to show that if spaces (X i , τi ) (i = 1, 2) have the first countability

property then a function f : X → Y is continuous if and only if for every
convergent sequence (x n ) we have f (limn x n ) = limn f (x n ).

Definition A.1.24. A topological space has the second countability property if

the whole space has a countable basis. ù

For example, the space R has the second countability property for all
three topologies τR , τ<
R
, τ>
R
. Indeed, we also get a basis if instead of taking all
intervals, we only take intervals with rational endpoints. On the other hand,
the metric space of Example A.1.10.7 does not have the second countability
property.

Definition A.1.25. In a topological space (X , τ), a set B of points is called

dense at a point x if it intersects every neighborhood of x. It is called every-
where dense, or dense, if it is dense at every point. A metric space is called
separable if it has a countable everywhere dense subset. ù

It is easy to see that a metric space is separable if and only if as a topo-

logical space it has the second countability property.
Example A.1.12. In Example A.1.10.8, for X = [0, 1], we can choose as our
everywhere dense set the set of all polynomials with rational coefficients, or
alternatively, the set of all piecewise linear functions whose graph has finitely
many nodes at rational points.
More generally, let X be a compact separable metric space with a dense set
D. Then it can be shown that in the metric space C(X ), the set of functions
E (D) introduced in Definition A.1.21 is dense, and turns it into a complete
(not necessarily compact!) separable metric space. ù

146
 A.2. Measures

Definition A.1.26. In a metric space, let us call a sequence x 1 , x 2 , . . . a

Cauchy sequence if for all i < j we have d(x i , x j ) < 2−i . ù

It is easy to see that if an everywhere dense set D is given then every

element of the space can be represented as the limit of a Cauchy sequence of
elements of D. But not every Cauchy sequence needs to have a limit.

Definition A.1.27. A metric space is called complete if every Cauchy se-

quence in it has a limit. ù

For example, if X is the real line with the point 0 removed then X is not
complete, since there are Cauchy sequences converging to 0, but 0 is not in
X.
It is well-known that every metric space can be embedded (as an every-
where dense subspace) into a complete space.
Example A.1.13. Consider the set D[0, 1] of functions over [0, 1] that are
right continuous and have left limits everywhere. The book [5] introduces
two different metrics for them: the Skorohod metric d and the d0 metric.
In both metrics, two functions are close if a slight monotonic continuous
deformation of the coordinates makes them uniformly close. But in the d0
metric, the slope of the deformation must be close to 1. It is shown that the
two metrics give rise to the same topology; however, the space with metric d
is not complete, and the space with metric d0 is. ù
We will develop the theory of randomness over separable complete metric
spaces. This is a wide class of spaces encompassing most spaces of practical
interest. The theory would be simpler if we restricted it to compact or locally
compact spaces; however, some important spaces like C[0, 1] (the set of
continuouos functions over the interval [0, 1], with the maximum difference
as their distance) are not locally compact.

A.2 MEASURES
For a survey of measure theory, see for example [39].

A.2.1 SET ALGEBRAS

Event in a probability space are members of a class of sets that is required to

be a so-called σ-algebra (sigma-algebra).

147
A. BACKGROUND FROM MATHEMATICS

Definition A.2.1. A (Boolean set-) algebra is a set of subsets of some set

X closed under intersection and complement (and then, of course, under
union). It is a σ-algebra (sigma-algebra) if it is also closed under countable
intersection (and then, of course, under countable union). A semialgebra is a
set L of subsets of some set X closed under intersection, with the property
that the complement of every element of L is the disjoint union of a finite
number of elements of L . ù

If L is a semialgebra then the set of finite unions of elements of L is an

algebra.
Examples A.2.1.
1. The set L1 of left-closed intervals of the line (including intervals of the
form (−∞, a)) is a semialgebra.
2. The set L2 of all intervals of the line (which can be open, closed, left-
closed or right-closed), is a semialgebra.
3. In the set {0, 1}N of infinite 0-1-sequences, the set L3 of all subsets of the
form u{0, 1}N with u ∈ {0, 1}∗ , is a semialgebra.
4. The σ-algebra B generated by L1 , is the same as the one generated by
L2 , and is also the same as the one generated by the set of all open sets: it
is called the family of Borel sets of the line. The Borel sets of the extended
real line R are defined similarly.
5. More generally, the class of Borel sets in an arbitrary topological space is
the smallest σ-algebra containing all open sets.
6. Given σ-algebras A , B in sets X , Y , the product σ-algebra A × B in the
space X × Y is the one generated by all elements A × Y and X × B for
A ∈ A and B ∈ B.
ù

A.2.2 MEASURES
Probability is an example of the more general notion of a measure.

Definition A.2.2. A measurable space is a pair (X , S ) where S is a σ-algebra

of sets of X . A measure on a measurable space (X , S ) is a function µ : B →
R+ that is σ-additive: this means that
S for every
P countable family A1 , A2 , . . . of
disjoint elements of S we have µ( i Ai ) = i µ(Ai ). A measure µ is σ-finite

148
 A.2. Measures

if the whole space is the union of a countable set of subsets whose measure
is finite. It is finite if µ(X ) < ∞. It is a probability measure if µ(X ) = 1. ù

Example A.2.2 (Delta function). For any point x, the measure δ x is defined
as follows:
(
1 if x ∈ A,
δ x (A) =
0 otherwise.

Definition A.2.3. If µ is a measure, a point x is called an atom if µ(x) >

0. ù

Generally, we will consider measures over either a countable set (a dis-

crete measure for which the union of atoms has total measure) or an un-
countable one, with no atoms. But mixed cases are possible.
It is important to understand how a measure can be defined in practice.
Algebras are generally simpler to grasp constructively than σ-algebras; semi-
algebras are yet simpler. Suppose that µ is defined over a semialgebra L and
is additive. Then it can always be uniquely extended to an additive function
over the algebra generated by L . The following is an important theorem of
measure theory.

Proposition A.2.3 (Caratheodory’s extension theorem). Suppose that a non-

negative set function defined over a semialgebra L is σ-additive. Then it can
be extended uniquely to the σ-algebra generated by L .

Examples A.2.4.
1. Let x be point and let µ(A) = 1 if x ∈ A and 0 otherwise. In this case, we
say that µ is concentrated on the point x.
2. Consider the the line R, and the algebra L1 defined in Example A.2.1.1.
Let f : R → R be a monotonic real function. We define a set function over
L1 as follows. LetS[ai , bi ), (i =
P1, . . . , n) be a set of disjoint left-closed
intervals. Then µ( i [ai , bi )) = i f (bi ) − f (ai ). It is easy to see that µ is
additive. It is σ-additive if and only if f is left-continuous.
3. Let B = {0, 1}, and consider the set B N of infinite 0-1-sequences, and the
semialgebra L3 of Example A.2.1.3. Let µ : B ∗ → R+ be a function. Let
us write µ(uB N ) = µ(u) for all u ∈ B ∗ . Then it can be shown that the

149
A. BACKGROUND FROM MATHEMATICS

following conditions are equivalent: µ is σ-additive over L3 ; it is additive

over L3 ; the equation µ(u) = µ(u0) + µ(u1) holds for all u ∈ B ∗ .
4. The nonnegative linear combination of any finite number of measures is
also a measure. In this way, it is easy to construct arbitrary measures
concentrated on a finite number of points.
5. Given two measure spaces (X , A , µ) and (Y, B, ν) it is possible to define
the product measure µ × ν over the measureable space (X × Y, A × B).
The definition is required to satisfy µ × ν(A × B) = µ(A) × ν(B), and is
determined uniquely by this condition. If ν is a probability measure then,
of course, µ(A) = µ × ν(A × Y ).
ù
Let us finally define measureable functions.

Definition A.2.4 (Measureable functions). Let (X , A ), (Y, B) be two mea-

sureable spaces. A function f : X → Y is called measureable if and only if
f −1 (E) ∈ A for all E ∈ B. ù

The following is easy to check.

Proposition A.2.5. Let (X , A ) be a measureable space and (R, B) be the mea-

sureable space of the real numbers, with the Borel sets. Then f : X → R is
measureable if and only if all sets of the form f −1 ((r, ∞)) = { x : f (x) > r }
are measureable, where r is a rational number.

Remark A.2.6. Example A.2.4.3 shows a particularly attractive way to define

measures. Keep splitting the values µ(u) in an arbitrary way into µ(u0) and
µ(u1), and the resulting values on the semialgebra define a measure. Exam-
ple A.2.4.2 is less attractive, since in the process of defining µ on all intervals
and only keeping track of finite additivity, we may end up with a monotonic
function that is not left continuous, and thus with a measure that is not σ-
additive. In the subsection on probability measures over a metric space, we
will find that even on the real line, there is a way to define measures in a
step-by-step manner, and only checking for consistency along the way. ù
A probability space, according to the axioms introduced by Kolmogorov,
is just a measureable space with a normed measure.

Definition A.2.5. A probability space is a triple (X , S , P) where (X , S ) is a

measurable space and P is a probability measure over it.

150
 A.2. Measures

Let (X i , Si ) (i = 1, 2) be measurable spaces, and let f : X → Y be a

mapping. Then f is measurable if and only if for each element B of S2 ,
its inverse image f −1 (B) is in S1 . If µ1 is a measure over (X 1 , S1 ) then
µ2 defined by µ2 (A) = µ1 ( f −1 (A)) is a measure over X 2 called the measure
induced by f . ù

A.2.3 INTEGRAL
The notion of integral also generalizes to arbitrary measures, and is some-
times also used to define measures.
First we define integral on very simple functions.

Definition A.2.6. A measurable function f : X → R is called a step function

if its range is finite.
The set of step functions is closed with respect to linear combinations and
also with respect to the operations ∧, ∨. Any such set of functions is called a
Riesz space. ù

Definition A.2.7. Given a step function f which takes values x i on sets Ai ,

and a finite measure µ, we define
Z Z
X
µ( f ) = µ f = f dµ = f (x)µ(d x) = x i µ(Ai ).
i

This is a linear positive functional on the set of step functions. Moreover,

it can be shown that it is continuous on monotonic sequences: if f i & 0 then
µ f i & 0. The converse can also be shown: Let µ be a linear positive func-
tional on step functions that is continuous on monotonic sequences. Then
the set function µ(A) = µ(1A) is a finite measure.

Proposition A.2.7. Let E be any Riesz space of functions with the property
that 1 ∈ E . Let µ be a positive linear functional on E continuous on monotonic
sequences, with µ1 = 1. The functional µ can be extended to the set E+ of
monotonic limits of nonnegative elements of E , by continuity. In case when E is
the set of all step functions, the set E+ is the set of all nonnegative measurable
functions.

Now we extend the notion of integral to a wider class of functions.

151
A. BACKGROUND FROM MATHEMATICS

Definition A.2.8. Let us fix a finite measure µ over a measurable space

(X , S ). A measurable function f is called integrable with respect to µ if
µ| f |+ < ∞ and µ| f |− < ∞. In this case, we define µ f = µ| f |+ − µ| f |− . ù

The set of integrable functions is a Riesz space, and the positive linear
functional µ on it is continuous with respect to monotonic sequences. The
continuity over monotonic sequences also implies the following theorem.

Proposition A.2.8 (Bounded convergence theorem). Suppose that functions

f n are integrable and | f n | < g for some integrable function g. Then f = limn f n
is integrable and µ f = limn µ f n .

Definition A.2.9. Two measurable functions f , g are called equivalent with

respect to measure µ if µ( f − g) = 0. ù

For two-dimensional integration, the following theorem holds.

Proposition A.2.9 (Fubini theorem). Suppose that function f (·, ·) is inte-

grable over the space (X × Y, A × B, µ × ν). Then for µ-almost all x, the
function f (x, ·) is integrable over (Y, B, ν), and the function x 7→ ν y f (x, y) is
integrable over (X , A , µ) with (µ × ν) f = µ x µ y f .

To express a continuity property of measures, we can say the following

(recall the definition of C(X ) in Example A.1.10.8).

Proposition A.2.10. Let X be a metric space and µ a measure. Then µ is a

bounded (and thus continuous) linear functional over the space C(X ).

A.2.4 DENSITY
When does one measure have a density function with respect to another?

Definition A.2.10. Let µ, ν be two measures over the same measurable

space. We say that ν is absolutely continuous with respect to µ, or that µ
dominates ν, if for each set A, µ(A) = 0 implies ν(A) = 0. ù

Every nonnegative integrable function f defines a new measure ν via

the formula ν(A) = µ( f · 1A). This measure ν is absolutely continuous with
respect to µ. The Radon-Nikodym theorem says that the converse is also
true.

152
 A.2. Measures

Proposition A.2.11 (Radon-Nikodym theorem). If ν is dominated by µ then

there is a nonnegative integrable function f such that ν(A) = µ( f · 1A) for all
measurable sets A. The function f is defined uniquely to within equivalence
with respect to µ.

Definition A.2.11. The function f of the Radom-Nikodym Theorem above is

called the density of ν with respect to µ. We will denote it by

µ(d x) dµ
f (x) = = .
ν(d x) dν

The following theorem is also standard.

Proposition A.2.12 (Chain rule and inverse function).

(a) Let µ, ν, η be measures such that η is absolutely continuous with respect to
µ and µ is absolutely continuous with respect to ν. Then the “chain rule”
holds:
dη dη dµ
= . (A.2.1)
dν dµ dν
ν(d x)
(b) If > 0 for all x then µ is also absolutely continuous with respect to ν
µ(d x)
 −1
µ(d x) ν(d x)
and ν(d x) = µ(d x) .

There is a natural distance to be used between measures, though later we

will see that it is not the preferred one in metric spaces.

Definition A.2.12 (Total variation distance). Let µ, ν be two measures, then

both are dominated by some measure η (for example by η = µ+ν). Let their
densities with respect to η be f and g. Then we define the total variation
distance of the two measures as

D(µ, ν) = η(| f − g|).

It is independent of the dominating measure η. ù

Example A.2.13. Suppose that the space X can be partitioned into disjoint
sets A, B such that ν(A) = µ(B) = 0. Then D(µ, ν) = µ(A) + ν(B) = µ(X ) +
ν(X ). ù

153
A. BACKGROUND FROM MATHEMATICS

A.2.5 RANDOM TRANSITIONS

What is just a transition matrix in case of a Markov chain also needs to be

defined more carefully in the non-discrete cases. We follow the definition
given in [39].

Definition A.2.13. Let (X , A ), (Y, B) be measureable spaces (defined in

Subsection A.2.2). Suppose that a family of probability measures Λ = { λ x :
x ∈ X } on B is given. We call it a probability kernel, (or Markov kernel,
or conditional distribution) if the map x 7→ λ x B is measurable for each B ∈
B. ù

When X , Y are finite sets then λ is a Markov transition matrix. The

following theorem shows that λ assigns a joint distribution over the space
(X × Y, A × B) to each input distribution µ.

Proposition A.2.14. For each nonnegative A × B-measureable function f

over X × Y ,
1. The function y 7→ f (x, y) is B-measurable for each fixed x.
2. The function x 7→ λ xy f (x, y) is A -measurable.
3. The integral f 7→ µ x λ xy f (x, y) defines a measure on A × B.

The above proposition allows to define a mapping over measures.

Definition A.2.14. According to the above proposition, given a probability

kernel Λ, to each measure µ over A corresponds a measure over A × B.
We will denote its marginal over B as

Λ∗ µ. (A.2.2)

For every measurable function g( y) over Y , we can define the measurable

function f (x) = λ x g = λ xy g( y): we write

f = Λg. (A.2.3)

The operator Λ is linear, and monotone with Λ1 = 1. By these definitions,

we have
µ(Λg) = (Λ∗ µ)g. (A.2.4)
An example is the simple case of a deterministic mapping:

154
 A.2. Measures

Example A.2.15. Let h : X → Y be a measureable function, and let λ x be the

measure δh(x) concentrated on the point h(x). This operator, denoted Λh is,
in fact, a deterministic transition, and we have Λh g = g ◦ h. In this case, we
will simplify the notation as follows:

h∗ µ = Λ∗h.

A.2.6 PROBABILITY MEASURES OVER A METRIC SPACE

We follow the exposition of [5]. Whenever we deal with probability measures

on a metric space, we will assume that our metric space is complete and
separable (Polish space).
Let X = (X , d) be a complete separable metric space. Then X gives rise to
a measurable space, where the measurable sets are its Borel sets. It can be
shown that, if A is a Borel set and µ is a finite measure then there are sets
F ⊆ A ⊆ G where F is an Fσ set, G is a Gδ set, and µ(F ) = µ(G).
It can be shown that a measure is determined by its values on the ele-
ments of any a basis of open sets closed under intersections. The following
proposition follows then essentially from Proposition A.2.3.

Proposition A.2.16. Let B be a basis of open sets closed under intersections.

Let B ∗ be the set algebra generated by this basis and let µ be any σ-additive
set function on B ∗ with µ(X ) = 1. Then µ can be extended uniquely to a
probability measure.

WEAK TOPOLOGY

One can introduce the notion of convergence of measures in a number of

ways. We have already introduced the total variation distance in Defini-
tion A.2.12 above. But in some cases, the requirement of being close in this
distance is too strong. Let
M (X)
be the set of probability measures on the metric space X. Let x n be a sequence
of points converging to point x but with x n 6= x. We would like to say that the
delta measure δ x n (concentrated on point x n , see Example A.2.2) converges
to δ x . But the total variation distance D(δ x n , δ x ) is 2 for all n.

155
A. BACKGROUND FROM MATHEMATICS

Definition A.2.15 (Weak convergence). We say that a sequence of probabil-

ity measures µn over a metric space (X , d) weakly converges to measure µ if
for all bounded continuous real functions f over X we have µn f → µ f .
For a bounded continuous function f and real numbers c let

A f ,c = { µ : µ f < c }

A topology of weak convergence (M , τw ) can be defined using a number

of different subbases. The one used in the original definition is the subbasis
consisting of all sets of the form A f ,c above.
We also get a subbasis (see for example [39]) if we restrict ourselves to
the set Lip(X ) of Lipschitz functions defined in (A.1.2). Another possible
subbasis giving rise to the same topology consists of all sets of the form

BG,c = { µ : µ(G) > c } (A.2.5)

for open sets G and real numbers c. Let us find some countable subbases.
Since the space X is separable, there is a sequence U1 , U2 , . . . of open sets
that forms a basis of X. Then we can restrict the subbasis of the space of
measures to those sets BG,c where G is the union of a finite number of basis
elements Ui of X and c is rational. This way, the space (M , τw ) itself has the
second countability property.
It is more convenient to define a countable subbasis using bounded con-
tinuous functions f , since the function µ 7→ µ f is continuous on such func-
tions, while µ 7→ µU is typically not continuous when U is an open set.
Example A.2.17. If X = R and U is the open interval (0, 1), the sequence
of probability measures δ1/n (concentrated on 1/n) converges to δ0 , but
δ1/n (U) = 1, and δ0 (U) = 0. ù
For some fixed dense set D, let F1 = F1 (D) be the set of functions intro-
duced in Definition A.1.21.

Definition A.2.16. We say that a set A is a continuity set of measure µ if

µ(∂ A) = 0: the boundary of A has measure 0. ù

Proposition A.2.18. The following conditions are equivalent:

1. µn weakly converges to µ.
2. µn f → µ f for all f ∈ F1 .

156
 A.2. Measures

3. For every Borel set A, that is a continuity set of µ, we have µn (A) → µ(A).
4. For every closed set F , lim infn µn (F ) ¾ µ(F ).
5. For every open set G, lim supn µn (G) ¶ µ(G).

Definition A.2.17. To define the topological space M (X ) of the set of mea-

sures over the metric space X , we choose as subbasis

σM (A.2.6)

the sets { µ : µ f < r } and { µ : µ f > r } for all f ∈ F1 and r ∈ Q. ù

The simple functions we introduced can also be used to define measure

and integral in themselves. Recall the definition of the set E in (A.1.3).T This
set is a Riesz space as defined in Subsection A.2.3. A reasoning combining
Propositions A.2.3 and A.2.7 gives the following.

Proposition A.2.19. Suppose that a positive linear functional µ with µ1 =

1 is defined on E that is continuous with respect to monotone convergence.
Then µ can be extended uniquely to a probability measure over X with µ f =
R
f (x)µ(d x) for all f ∈ E .

Having a topology over the set of measures we can also extend Proposi-
tion A.2.10:

Proposition A.2.20. Let X be a complete separable metric space and M (X )

the space of bounded measures over X with the weak topology. The function
(µ, f ) 7→ µ f is a continuous function M (X ) × C(X ) → R.

As mentioned above, for an open set G the value µ(G) is not a continuous
function of the measure µ. We can only say the following:

Proposition A.2.21. Let X be a complete separable metric space, and M (X )

the space of bounded measures over X with the weak topology, and G ⊆ X an
open set. The function µ 7→ µ(G) is lower semicontinuous.

DISTANCES FOR THE WEAK TOPOLOGY

The definition of measures in the style of Proposition A.2.19 is not sufficiently

constructive. Consider a gradual definition of the measure µ, extending it
to more and more elements of E , while keeping the positivity and linearity

157
A. BACKGROUND FROM MATHEMATICS

property. It can happen that the function µ we end up with in the limit, is not
continuous with respect to monotone convergence. Let us therefore metrize
the space of measures: then an arbitrary measure can be defined as the limit
of a Cauchy sequence of simple meaures.
One metric that generates the topology of weak convergence is the fol-
lowing.

Definition A.2.18 (Prokhorov distance). The Prokhorov distance ρ(µ, ν) of

two measures is the infimum of all those " for which, for all Borel sets A we
have (using the notation (A.1.1))

µ(A) ¶ ν(A" ) + ".

It can be shown that this is a metric and it generates the weak topology.
In computer science, it has been reinvented by the name of “earth mover
distance”. The following important result helps visualize it:

Proposition A.2.22 (Coupling Theorem, see [49]). Let µ, ν be two probabil-

ity measures over a complete separable metric space X with ρ(µ, ν) ¶ ". Then
there is a probability measure P on the space X×X with marginals µ and ν such
that for a pair of random variables ξ, η having joint distribution P we have

P[ d(ξ, η) > " ] ¶ ".

Since weak topology has the second countability property, the metric
space defined by the distance ρ(·, ·) is separable. This can also be seen di-
rectly: let us define a dense set in weak topology.

Definition A.2.19. For each point x, let us define by δ x the measure which
concentrates its total weight 1 in point x. Let D be a countable everywhere
dense set of points in X . Let DM be the set of finite convex rational combi-
nations of measures of the form δ x i where x i ∈ D, that is those probability
measures that are concentrated on finitely many points of D and assign ra-
tional values to them. ù

It can be shown that DM is everywhere dense in the metric space

(M (X ), ρ); so, this space is separable. It can also be shown that (M (X ), ρ)
is complete. Thus, a measure can be given as the limit of a Cauchy sequence
of elements µ1 , µ2 , . . . of DM .

158
 A.2. Measures

The definition of the Prokhorov distance uses quantification over all Borel
sets. However, in an important simple case, it can be handled efficiently.

Proposition A.2.23. Assume that measure ν is concentrated on a finite set of

points S ⊂ X . Then the condition ρ(ν, µ) < " is equivalent to the finite set of
conditions
µ(A" ) > ν(A) − " (A.2.7)
for all A ⊂ S.

Another useful distance for measures over a bounded space is the Wasser-
stein distance.

Definition A.2.20. Over a bounded metric space, we define the Wasserstein

distance by

W (µ, ν) = sup |µ f − ν f |.
f ∈Lip1 (X )

The Prokhorov and Wasserstein metrics are equivalent.

Proposition A.2.24. The Prokhorov and Wasserstein metrics are equivalent:

the identity function creates a uniformly continuous homeomorphism between
the two metric spaces.

Proof. Let M = sup x, y∈X d(x, y). The proof actually shows for " < 1:

ρ(µ, ν) ¶ " ⇒ W (µ, ν) ¶ (M + 1)",

W (µ, ν) ¶ " 2 ⇒ ρ(µ, ν) ¶ ".

Suppose ρ(µ, ν) ¶ " < 1. By the Coupling Theorem (Proposition A.2.22),

there are random variables ξ, η over X with a joint distribution, and having
respectively the distribution µ and ν, such that P[ d(ξ, η) > " ] ¶ ". Then
we have

|µ f − ν f | = |E f (ξ) − E f (η)| ¶ E| f (ξ) − f (η)|

¶ "P[ ρ(ξ, η) ¶ " ] + M P[ ρ(ξ, η) > " ] ¶ (M + 1)".

Now suppose W (µ, ν) ¶ " 2 < 1. For a Borel set A define g"A(x) = |1 −
ρ(x, A)/"|+ . Then we have µ(A) ¶ µ(g"A) and ν g"A ¶ ν(A" ). Further " g"A ∈

159
A. BACKGROUND FROM MATHEMATICS

Lip1 , and hence W (µ, ν) ¶ " 2 implies "µ(g"A) ¶ "ν(" g"A) + " 2 . This concludes
the proof by

µ(A) ¶ µ(g"A) ¶ ν(" g"A) + " ¶ ν(A" ) + ".

RELATIVE COMPACTNESS

Convergence over the set of measures, even over a noncompact space, is

sometimes obtained via compactness properties.

Definition A.2.21. A set Π of measures in (M (X ), ρ) is called sequentially

compact if every sequence of elements of Π contains a convergent subse-
quence.
A set of Π of measures is called tight if for every " there is a compact set
K such that µ(K) > 1 − " for all µ in Π. ù

The following important theorem is using our assumptions of separability

and completeness of the underlying metric space (X , ρ).

Proposition A.2.25 (Prokhorov). A set of measures is sequentially compact if

and only if it is tight and if and only if its closure is compact in the metric space
(M (X ), ρ).

The following, simplest example is interesting in its own right.

Example A.2.26. The one-element set {µ} is compact and therefore by
Prokhorov’s theorem tight. Here, tightness says that for each " a mass of
size 1 − " of µ is concentrated on some compact set. ù
The following theorem strengthens the observation of this example.

Proposition A.2.27. A finite measure µ over a separable complete metric space

has the property

µ(B) = sup{ µ(K) : compact K ⊆ B }

for all Borel sets B.

In case of compact metric spaces, the following known theorem helps.

160
 A.2. Measures

Proposition A.2.28. The metric space (M (X), ρ) of measures is compact if

and only if the underlying metric space (X , d) is compact.

So, if (X , d) is not compact then the set of measures is not compact. But
still, by Proposition A.2.27, each finite measure is “almost” concentrated on
a compact set.

SEMIMEASURES
Let us generalize then notion of semimeasure for the case of general Polish
spaces. We use Proposition A.2.19 as a starting point.

Definition A.2.22. A function f 7→ µ f defined over the set of all bounded

continuous functions is called a semimeasure if it has the following properties:
(a) Nonnegative: f ¾ 0 implies µ f ¾ 0.
(b) Positive homogenous: we have µ(a f ) = aµ f for all nonnegative real
a > 0.
(c) Superadditive: µ( f + g) ¾ µ f + µg.
(d) Normed: µ1 = 1.
ù

Remark A.2.29. The weaker requirement µ1 ¶ 1 suffices, but if we have µ1 ¶

1 we can always set simply µ1 = 1 without violating the other requirements.
ù

FUNCTIONALS OF MEASURES

For a deeper study of randomness tests, we will need to characterize certain

functionals of finite measures over a Polish space.

Proposition A.2.30. Let X = (X , d) be a complete separable metric space with

M˜(X) the set of finite measures over it. A weakly continuous linear function F :
M˜(X) → R can always be written as F (µ) = µ f for some bounded continuous
function f over X.

Proof. Define f (x) = F (δ x ). The weak continuity of F implies that f (x) is

continuous. Let us show that it is bounded. Suppose it is not. Then there is
a sequence of distinct points x n ∈ X , n = 1, 2, . . . with f (x n ) > 2−n . Let µ

161
A. BACKGROUND FROM MATHEMATICS

δ = f (x n ) >
P −n P −n
be the measure n 2 x n
, then by linearity we have L(µ) n2
=
P
n 1 ∞, which is not allowed.
The function µ 7→ µ f is continuous and coincides with F (µ) on the dense
set of points DM , so they are equal.

162
B CONSTRUCTIVITY

B.1 COMPUTABLE TOPOLOGY

B.1.1 REPRESENTATIONS
There are several equivalent ways that notions of computability can be ex-
tended to spaces like real numbers, metric spaces, measures, and so on. We
use the the concepts of numbering (notation) and representation, as defined
in [55].

Notation B.1.1. We will denote by N the set of natural numbers and by B

the set {0, 1}.
Given a set (an alphabet) Σ we denote by ΣN = ΣN the set of infinite
sequences with elements in Σ.
If for some finite or infinite sequences x, y, z, w, we have z = w x y then
we write w v z (w is a prefix of z) and x / z. After [55], let us define the
wrapping function ι : Σ∗ → Σ∗ by

ι(a1 a2 · · · an ) = 110a1 0a2 0 · · · an 011. (B.1.1)

Note that
l(ι(x)) = (2l(x) + 5) ∨ 6. (B.1.2)
For strings x, x i ∈ Σ∗ , p, pi ∈ ΣN , k ¾ 1, appropriate tupling functions
〈x 1 , . . . , x k 〉, 〈x, p〉, 〈p, x〉, and so on can be defined with the help of 〈·, ·〉
and ι(·). ù

Definition B.1.2. Given a countable set C, a numbering (or notation) of C

is a surjective partial mapping δ :⊆ N → C. Given some finite alphabet
Σ ⊇ {0, 1} and an arbitrary set S, a representation of S is a surjective mapping
χ :⊆ ΣN → S. A naming system is a notation or a representation. ù

163
B. CONSTRUCTIVITY

Here are some standard naming systems:

1. id, the identity over Σ∗ or ΣN .
2. νN , νZ , νQ for the set of natural numbers, integers and rational numbers.
3. Cf : ΣN → 2N , the characteristic function representation of sets of natural
numbers, is defined by Cf(p) = { i : p(i) = 1 }.
4. En : ΣN → 2N , the enumeration representation of sets of natural numbers,
is defined by En(p) = { n ∈ N : 110n+1 11 / p }.
∗
5. For ∆ ⊆ Σ, En∆ : ΣN → 2∆ , the enumeration representation of subsets of
∆∗ , is defined by En∆ (p) = { w ∈ Σ∗ : ι(w) / p }.
Using Turing machines with infinite input tapes, work tapes and output
tapes, one can define names for all computable functions between spaces
that are Cartesian products of terms of the kind Σ∗ and ΣN . (One wonders
whether Σ∗ ∪ ΣN is not needed, since a Turing machine with an infinite input
tape may still produce only a finite output. But in this case one can also en-
code the result into an infinite sequence.) Then, the notion of computability
can be transferred to other spaces as follows.

Definition B.1.3. Let δi : Yi → X i , i = 1, 0 be naming systems of the spaces

X i . Let f :⊆ X 1 → X 0 , g :⊆ Y1 → Y0 . We say that function g realizes function
f if
f (δ1 ( y)) = δ0 (g( y)) (B.1.3)

holds for all y for which the left-hand side is defined.

Realization of multi-argument functions is defined similarly. We say that
a function f : X 1 × X 2 → X 0 is (δ1 , δ2 , δ0 )-computable if there is a computable
function g :⊆ Y1 × Y2 → Y0 realizing it. In this case, a name for f is naturally
derived from a name of g.1 ù

Definition B.1.4. For representations ξ, η, we write ξ ¶ η if there is a com-

putable function f :⊆ ΣN → ΣN with ξ(x) = η( f (x)). In words, we say that
ξ is reducible to η, or that f reduces (translates) ξ to η. There is a similar
definition of reduction for notations. We write ξ ≡ η if ξ ¶ η and η ¶ ξ. ù

1
Any function g realizing f via (B.1.3) automatically has a certain extensionality prop-
erty: if δ1 ( y) = δ1 ( y 0 ) then g( y) = g( y 0 ).

164
 B.1. Computable topology

B.1.2 CONSTRUCTIVE TOPOLOGICAL SPACE

Let us start with the definition of topology with the help of a subbasis of open
sets.

Definition B.1.5. A constructive topological space X = (X , σ, ν) is a topolog-

ical space over a set X with a subbasis σ effectively enumerated as a list
σ = {ν(1), ν(2), . . .}, and having the T0 property (thus, every point is deter-
mined uniquely by the subset of elements of σ containing it).
By definition, a constructive topological space satisfies the second count-
ability axiom. We obtain a basis
σ∩
of the space X by taking all possible finite intersections of elements of σ. It
is easy to produce an effective enumeration for σ∩ from ν. We will denote
this enumeration by ν ∩ . This basis will be called the canonical basis.
For every nonempty subset Y of the space X , the subspace of X will nat-
urally get the same structure Y = (Y, σ, ν) defined by { V ∩ Y : V ∈ σ }.
The product operation is defined over constructive topological spaces in
the natural way. ù

Remark B.1.1. The definition of a subspace shows that a constructive topo-

logical space is not a “constructive object” by itself, since the set X itself is
not necessarily given effectively. For example, any nonempty subset of the
real line is a constructive topological space, as a subspace of the real line. ù
Examples B.1.2. The following examples will be used later.
1. A discrete topological space, where the underlying set is finite or count-
ably infinite, with a fixed enumeration.
2. The real line, choosing the basis to be the open intervals with rational
endpoints with their natural enumeration. Product spaces can be formed
to give the Euclidean plane a constructive topology.
3. The real line R, with the subbasis σR> defined as the set of all open in-
tervals (−∞, b) with rational endpoints b. The subbasis σR< , defined sim-
ilarly, leads to another topology. These two topologies differ from each
other and from the usual one on the real line, and they are not Hausdorff
spaces.
4. This is the constructive version of Example A.1.1.5. Let X be a set with a
constructive discrete topology, and X N the set of infinite sequences with

165
B. CONSTRUCTIVITY

elements from X , with the product topology: a natural enumerated basis

is also easy to define.
ù

Definition B.1.6. Due to the T0 property, every point in our space is de-
termined uniquely by the set of open sets containing it. Thus, there is
a representation γX of X defined as follows. We say that γX (p) = x if
EnΣ (p) = { w : x ∈ ν(w) }. If γX (p) = x then we say that the infinite sequence
p is a complete name of x: it encodes all names of all subbasis elements con-
taining x. From now on, we will call γX the complete standard representation
of the space X. ù

Remark B.1.3. Here it becomes important that representations are allowed

to be partial functions. ù

Definition B.1.7 (Constructive open sets). In a constructive topological

space X = (X , σ, ν), a set G ⊆ X is called constructive open, or lower semi-
computable open in set B if there is a computably enumerable set E with
G = w∈E ν ∩ (w) ∩ B. It is constructife open if it is constructive open in X . ù
S

In the special kind of spaces in which randomness has been developed

until now, constructive open sets have a nice characterization:

Proposition B.1.4. Assume that the space X = (X , σ, ν) has the form Y1 ×· · ·×

Yn where each Yi is either Σ∗ or T ΣN . Then a set G is constructive open iff it is
open and the set { (w1 , . . . , w n ) : i ν(w i ) ⊂ G } is recursively enumerable.

Proof. The proof is not difficult, but it relies on the discrete nature of the
space Σ∗ and on the fact that the space ΣN is compact and its basis consists
of sets that are open and closed at the same time.

It is easy to see that if two sets are constructive open then so is their
union. The above remark implies that a space having the form Y1 × · · · × Yn
where each Yi is either Σ∗ or ΣN , also the intersection of two recursively open
sets is recursively open. We will see that this statement holds, more generally,
in all computable metric spaces.

166
 B.1. Computable topology

B.1.3 COMPUTABLE FUNCTIONS

Definition B.1.8. Let Xi = (X i , σi , νi ) be constructive topological spaces, and

let f : X 1 → X 0 be a function. As we know, f is continuous iff the inverse
image f −1 (G) of each open set G is open in its domain. Computability is
an effective version of continuity: it requires that the inverse image of basis
elements is uniformly constructively open. More precisely, f : X 1 → X 0 is
computable if the set
[
f −1 (V ) × {V }
V ∈σ0∩

is a constructive open subset of X 1 × σ0∩ . Here the basis σ0∩ of X0 is treated

as a discrete constructive topological space, with its natural enumeration.
A partial function is computable if its restriction to the subspace that is
its domain, is computable. ù

The above definition depends on the enumerations ν1 , ν0 . The following

theorem shows that this computability coincides with the one obtained by
transfer via the representations γXi .

Proposition B.1.5 (Hertling). For i = 0, 1, let Xi = (X i , σi , νi ) be constructive

topological spaces. Then a function f : X 1 → X 0 is computable iff it is (γX1 , γX0 )-
computable for the representations γXi defined above.

The notion of computable functions helps define morphisms between con-

structive topological spaces.

Definition B.1.9. Let us call two spaces X 1 and X 0 effectively homeomorphic

if there are computable maps between them that are inverses of each other.
In the special case when X 0 = X 1 , we say that the enumerations of subbases
ν0 , ν1 are equivalent if the identity mapping is a effective homeomorphism.
This means that there are recursively enumerable sets F, G such that
[ [
ν1 (v) = ν0∩ (w) for all v, ν0 (w) = ν1∩ (v) for all w.
(v,w)∈F (w,v)∈G

167
B. CONSTRUCTIVITY

B.1.4 EFFECTIVE COMPACTNESS

Compactness is an important property of topological spaces. In the con-

structive case, however, a constructive version of compactness is frequently
needed.

Definition B.1.10. A constructive topological space X with canonical basis

β has recognizeable covers if the set
[
{ S ⊆ β : S is finite and U=X}
U∈S

is recursively enumerable.
A compact space with recognizeable covers will be called effectively com-
pact. ù

Example B.1.6. Let α ∈ [0, 1] be a real number such that the set of rationals
less than α is recursively enumerable but the set of rationals larger
P than α
are not. (It is known that there are such numbers, for example x∈N 2−H(x) .)
Let X be the subspace of the real line on the interval [0, α], with the induced
topology. A basis of this topology is the set β of all nonempty intervals of the
form I ∩ [0, α], where I is an open interval with rational endpoints.
This space is compact, but not effectively so. ù
The following is immediate.

Proposition B.1.7. In an effectively compact space, in every recursively enu-

merable set of basis elements covering the space one can effectively find a finite
covering.

It is known that every closed subset of a compact space is compact. This

statement has a constructive counterpart:

Proposition B.1.8. Every upper semicomputable closed subset E of an effec-

tively compact space X is also effectively compact.

Proof. U1 , . . . , Uk be a finite set covering E. Let us enumerate a sequence Vi of

canonical basis elements whose union is the complement of E. If U1 ∪· · ·∪ Uk
covers E then along with the sequence Vi it will cover X , and this will be
recognized in a finite number of steps.
On an effectively compact space, computable functions have computable
extrema:

168
 B.1. Computable topology

Proposition B.1.9. Let X be an effectively compact space, let f (x) be lower

semicomputable function on X . Then the infimum (which is always a mini-
mum) of f (x) is lower semicomputable uniformly from the definition of X and
f.

Proof. For any rational number r we must be able to recognize that the min-
imum is greater than r. For each r the set { x : f (x) > r } is a lower semi-
computable open set. It is the union of an enumerated sequence of canonical
basis elements. If it covers X this will be recognized in a finite number of
steps.

It is known that a continuous function map compact sets into compact

ones. This statement also has a constructive counterpart.

Proposition B.1.10. Let X be an effectively compact space, and f a computable

function from X into another constructive topological space Y . Then f (X ) is
effectively compact.

Proof. Let βX and βY be the enumerated bases of the space X and Y re-
spectively. Since f is computable, S there is a recursively enumerable set
E ⊆ βX × βY such that f −1 (V ) = (U,V )∈E U holds for all V ∈ βY . Let
EV = { U : (U, V ) ∈ E }, then f −1 (V ) = EV .
S

Consider some finite cover f (X ) ⊆ V1 ∪ · ·S · ∪ Vk with someSVi ∈ βY . We

will show that it will be recognized. Let U = i EVi , then X ⊆ U , and the
whole set U ⊆ βX is enumerable. ByS compactness, there is a finite number
of elements U1 , . . . , Un ∈ U with X ⊆ i Ui . By effective compactness, every
one of these finite covers will be recognized. And any one such recognition
will serve to recognize that V1 , . . . , Vk is a cover of f (X ).

B.1.5 COMPUTABLE ELEMENTS AND SEQUENCES

Let us define computable elements.

Definition B.1.11. Let U = ({0}, σ0 , ν0 ) be the one-element space turned

into a trivial constructive topological space, and let X = (X , σ, ν) be another
constructive topological space. We say that an element x ∈ X is computable
if the function 0 7→ x is computable. It is easy to see that this is equivalent
to the requirement that the set { u : x ∈ ν(u) } is recursively enumerable. Let
X j = (X j , σ j , ν j ), for i = 0, 1 be constructive topological spaces. A sequence

169
B. CONSTRUCTIVITY

f i , i = 1, 2, . . . of functions with f i : X 1 → X 0 is a computable sequence of

computable functions if (i, x) 7→ f i (x) is a computable function. ù

It is easy to see that this statement is equivalent to the statement that

there is a recursive function computing from each i a name for the comput-
able function f i . To do this formally, one sometimes using the s-m-n theorem
of recursion theory, or calls the method “currification”.
The notion of computability can be relativized.

Definition B.1.12. Let X j , j = 1, 2 be two constructive topological spaces

(or more general, representations) and let x j ∈ X j be given. We say that
x 2 is x 1 -computable if there is a computable function f :⊆ X 1 → X 2 with
f (x 1 ) = x 2 . ù

Remark B.1.11. The requirement that f computes y from every representa-

tion of x makes x-computability a very different requirement of mere Turing
reducibility of y to x. We will point out an important implication of this
difference later, in the notion of µ-randomness. It is therefore preferable
not to use the term “oracle computation” when referring to computations on
representations . ù

B.1.6 SEMICOMPUTABILITY
Lower semicomputability is a constructive version of lower semicontinuity, as
given in Definition A.1.12, but the sets that are required to be open there are
required to be constructive open here. The analogue of Proposition A.1.5 and
Corollary A.1.7 holds also: a lower semicomputable function is the supre-
mum of simple constant functions defined on basis elements, and a lower
semicomputable function defined on a subset can always be extended to one
over the whole space.

Definition B.1.13. Let X = (X , σ, ν) be a constructive topological space. A

function f :⊆ X → R+ is called lower semicomputable if the set { (x, r) :
f (x) > r } is constructive open.
We define the notion of x-lower semicomputable similarly to the notion
of x-computability. ù

Let Y = (R+ , σR< , νR< ) be the effective topological space introduced in Ex-
ample B.1.2.2, in which νR> is an enumeration of all open intervals of the

170
 B.1. Computable topology

form (r, ∞] with rational r. The following characterization is analogous to

Proposition A.1.3.

Proposition B.1.12. A function f : X → R is lower semicomputable if and only

if it is (ν, νR> )-computable.

As a name of a computable function, we can use the name of the enu-

meration algorithm derived from the definition of computability, or the name
derivable using this representation theorem.
The following example is analogous to Example A.1.4.
Example B.1.13. The indicator function 1G (x) of an arbitrary constructive
open set G is lower semicomputable. ù
The following fact is analogous to Proposition A.1.5:

Proposition B.1.14. Let f1 , f2 , . . . be a computable sequence of lower semi-

computable functions (via their names) over a constructive topological space X.
Then supi f i is also lower semicomputable.

The following fact is analogous to Proposition A.1.6:

Proposition B.1.15. Let X = (X , σ, ν) be a constructive topological space with

enumerated basis β = σ∩ and f : X → R+ a lower semicomputable function.
Then there is a lower semicomputable function g : β → R+ (where β is taken
with the discrete topology) with f (x) = sup x∈β g(β).

In the important special case of Cantor spaces, the basis is given by the
set of finite sequences. In this case we can also require the function g(w) to
be monotonic in the words w:

Proposition B.1.16. Let X = ΣN be a Cantor space as defined in Exam-

ple B.1.2.4. Then f : X → R+ is lower semicomputable if and only if there is
a lower semicomputable function g : Σ∗ → R+ (where Σ∗ is taken as a discrete
space) monotonic with respect to the relation u v v, with f (ξ) = supuvξ g(u).

Remark B.1.17. In the above representation, we could require the function

g to be computable. Indeed, we can just replace g(w) with g 0 (w) where
g 0 (w) = max vvw g(v, l(w)), and g(v, n) is as much of g(v) as can be com-
puted in n steps. ù
Upper semicomputability is defined analogously:

171
B. CONSTRUCTIVITY

Definition B.1.14. We will call a closed set upper semicomputable (co-

recursively enumerable) if its complement is a lower semicomputable (r.e.)
open set. ù

The proof of the following statement is not difficult.

Proposition B.1.18. Let Xi = (X i , σi , νi ) for i = 1, 2, 0 be constructive topolog-

ical spaces, and let f : X 1 × X 2 → X 0 , and assume that x 1 ∈ X 1 is a computable
element.
1. If f is computable then x 2 7→ f (x 1 , x 2 ) is also computable.
2. If X0 = R, and f is lower semicomputable then x 2 7→ f (x 1 , x 2 ) is also lower
semicomputable.

B.1.7 COMPUTABLE METRIC SPACE

Following [7], we define a computable metric space as follows.

Definition B.1.15. A constructive metric space is a tuple X = (X , d, D, α)

where (X , d) is a metric space, with a countable dense subset D and an
enumeration α of D. It is assumed that the real function d(α(v), α(w)) is
computable. ù

As x runs through elements of D and r through positive rational numbers,

we obtain the enumeration of a countable basis { B(x, r) : x ∈ D, r ∈ Q } (of
balls or radius r and center x) of X, giving rise to a constructive topological
space X̃.

Definition B.1.16. Let us call a sequence x 1 , x 2 , . . . a Cauchy sequence if

for all i < j we have d(x i , x j ) ¶ 2−i . To connect to the type-2 theory of
computability developed above, the Cauchy-representation δX of the space is
defined in a natural way. ù

It can be shown that as a representation of X̃, it is equivalent to γX̃ : δX ≡

γX̃ .
Examples B.1.19.
1. Example A.1.12 is a constructive metric space, with either of the two
(equivalent) choices for an enumerated dense set.

172
 B.1. Computable topology

2. Consider the metric space of Example A.1.10.6: the Cantor space (X , d).
Let s0 ∈ Σ be a distinguished element. For each finite sequence x ∈ Σ∗
let us define the infinite sequence ξ x = xs0 s0 . . . . The elements ξ x form
a (naturally enumerated) dense set in the space X , turning it into a con-
structive metric space.
ù
Let us point out a property of metric spaces that we use frequently.

Definition B.1.17. For balls B j = B(x j , r j ), j = 1, 2 we will say B1 < B2 if

d(x 1 , x 2 ) < r2 − r1 . In words, we will say that B1 is manifestly included in B2 .
This relation is useful since it is an easy way to see B1 ⊂ B2 . ù

The property can be generalized to some constructive topological spaces.

Definition B.1.18. A constructive topological space (X , β, ν) with basis β

has the manifest inclusion property if there is a relation b < b0 among its
basis elements with the following properties.
(a) b < b0 implies b ⊆ b0 .
(b) The set of pairs b < b0 is recursively enumerable.
(c) For every point x, and pair of basis elements b, b0 containing x there is
a basis element b00 containing x with b00 < b, b0 .
manifestly included in b0 . In such
We express this relation by saying that b is T
a space, a sequence b1 > b2 > . . . with i bi = {x} is called a manifest
representation of x. ù

Note that if the space has the manifest inclusion property then for every
pair x ∈ b there is a manifest representation of x beginning with b.
A constructive metric space has the manifest inclusion property as a topo-
logical space, and Cauchy representations are manifest.
Similarly to the definition of a computable sequence of computable func-
tions in Subsection B.1.5, we can define the notion of a computable sequence
of bounded computable functions, or the computable sequence f i of com-
putable Lipschitz functions: the bound and the Lipschitz constant of f i are
required to be computable from i. The following statement shows, in an ef-
fective form, that a function is lower semicomputable if and only if it is the
supremum of a computable sequence of computable functions.

173
B. CONSTRUCTIVITY

Proposition B.1.20. Let X be a computable metric space. There is a computable

mapping that to each name of a nonnegative lower semicomputable function
f assigns a name of a computable sequence of computable bounded Lipschitz
functions f i whose supremum is f .

Proof sketch. Show that f is the supremum of a computable sequence of

functions ci 1B(ui ,ri ) where ui ∈ D and ci , ri > 0 are rational. Clearly, each
indicator function 1B(ui ,ri ) is the supremum of a computable sequence of com-
putable functions g i, j . We have f = supn f n where f n = maxi¶n ci g i,n . It is easy
to see that the bounds on the functions f n are computable from n and that
they all are in Lipβn for a βn that is computable from n.

The following is also worth noting.

Proposition B.1.21. In a computable metric space, the intersection of two con-

structive open sets is constructive open.

Proof. Let β = { B(x, r) : x ∈ D, r ∈ Q } be a basis of our space. For a pair

(x, r) with x ∈ D, r ∈ Q, let

Γ(x, r) = { ( y, s) : y ∈ D, s ∈ Q, d(x, y) + s < r }.

If U is a constructive open
S set, then there is 0a computably enumerable set
SU ⊂ D × Q with U = (x,r)∈SU B(x, r). Let SU = { Γ(x, r) : (x, r) ∈ SU },
S

then we have U = (x,r)∈S 0 B(x, r). Now, it is easy to see

S
U

[
U ∩V = B(x, r).
(x,r)∈SU0 ∩SV0

The following theorem is very useful.

Theorem B.1.1. A computable metric space X is effectively compact if and

only if from each (rational) " one can compute a finite set of "-balls covering X .

Proof. Suppose first that the space is effectively compact. For each ", let
B1 , B2 , . . . be a list of all canonical balls with radius ". This sequence cov-
ers the space, so already some B1 , . . . , Bn covers the space, and this will be
recognized.

174
 B.2. Constructive measure theory

Suppose now that for every rational " one can find a finite set of "-balls
covering the space. Let S ⊆ β be a finite set of basis elements (balls) cover-
ing the space. For each element G = B(u, r) ∈SS , let G" = B(u, r −") S beS its "-
interior, and S" = { G" : G ∈ S }. Then G = " > 0G" , and X = ">0 S" .
Compactness implies that there is an " > 0 such that already Seps covers the
space. Let B1 , . . . , Bn be a finite set of "/2-balls Bi = B(ci , ri ) covering the
space that can be computed from ". Each of these balls Bi intersects one of
the the sets G" = B(u, r −"), d(u, ci ) ¶ r −"/2. But then Bi ⊆ B(u, r) in a rec-
ognizeable way. Once all the relations d(u, ci ) ¶ r − "/2 will be recognized
we will also recognize that S covers the space.

We can strengthen now Example A.1.12:

Example B.1.22 (Prove!). Let X be an effectively compact computable metric

space. Then the metric space C(X ) with the dense set of functions E (D)
introduced in Definition A.1.21 is a computable metric space. ù

The structure of a constructive metric space will be inherited on certain

subsets:

Definition B.1.19. Let X = (X , d, D, α) be a constructive metric space, and

G ⊂ X a constructive open subset. Then there is an enumeration αG of the
set D ∩ G that creates a constructive metric space G = (G, d, D, αG ). ù

B.2 CONSTRUCTIVE MEASURE THEORY

The basic concepts and results of measure theory are recalled in Section A.2.
For the theory of measures over metric spaces, see Subsection A.2.6. We
introduce a certain fixed, enumerated sequence of Lipschitz functions that
will be used frequently. Let E be the set of functions introduced in Defini-
tion A.1.21. The following construction will prove useful later.

Proposition B.2.1. All bounded continuous functions can be obtained as the

limit of an increasing sequence of functions from the enumerated countable set
E of bounded computable Lipschitz functions introduced in (A.1.3).

The proof is routine.

175
B. CONSTRUCTIVITY

B.2.1 SPACE OF MEASURES

Let X = (X , d, D, α) be a computable metric space. In Subsection A.2.6, the

space M (X) of measures over X is defined, along with a natural enumeration
ν = νM for a subbasis σ = σM of the weak topology. This is a constructive
topological space M which can be metrized by introducing, as in A.2.6, the
Prokhorov distance ρ(µ, ν). Recall that we defined DM as the set of those
probability measures that are concentrated on finitely many points of D and
assign rational values to them. Let αM be a natural enumeration of DM . Then

(M , ρ, DM , αM ) (B.2.1)

is a computable metric space whose constructive topology is equivalent to M.

Let U = B(x, r) be one of the balls in X, where x ∈ DX , r ∈ Q. The function
µ 7→ µ(U) is typically not computable, since as mentioned in (A.2.6), it is
not even continuous. The situation is better with µ 7→ µ f . The following
theorem is the computable strengthening of part of Proposition A.2.20:

Proposition B.2.2. Let X = (X , d, D, α) be a computable metric space, and let

M = (M (X), σ, ν) be the constructive topological space of probability measures
over X. If the function f : X → R is bounded and computable then µ 7→ µ f is
computable.

Proof sketch. To prove the theorem for bounded Lipschitz functions, we can
invoke the Strassen coupling theorem A.2.22.
The function f can be obtained as a limit of a computable monotone in-
creasing sequence of computable Lipschitz functions f n> , and also as a limit of
a computable monotone decreasing sequence of computable Lipschitz func-
tions f n< . In step n of our computation of µ f , we can approximate µ f n>
from above to within 1/n, and µ f n< from below to within 1/n. Let these
bounds be an> and an< . To approximate µ f to within ", find a stage n with
an> − an< + 2/n < ".

Using Example B.1.22, we can extend this as follows:

Proposition B.2.3 (Prove!). Let X be an effectively compact metric space (and

thus C(X) is a computable metric space). Then the mapping (µ, f ) 7→ µ f over
M (X) × C(X) is computable.

Talking about open sets, only a weaker statement can be made.

176
 B.2. Constructive measure theory

Proposition B.2.4. Let G ⊆ X be a constructive open set. The function µ 7→

µ(G) is lower semicomputable.

Remark B.2.5. Using the notion of an enumerative lattice defined by Hoyrup

and Rojas, one can make this a statement about the two-argument function
(µ, G) → µ(G). ù
It is known that if our metric space X is compact then so is the space
M (X) of measures. This can be strengthened:

Proposition B.2.6 (Prove!). If a complete computable metric space X is effec-

tively compact then M (X) is also effectively compact.

B.2.2 COMPUTABLE AND SEMICOMPUTABLE MEASURES

A measure µ is called computable if it is a computable element of the space

of measures. Let {g i } be the set of bounded Lipschitz functions over X intro-
duced in Definition A.1.21.

Proposition B.2.7. Measure µ is computable if and only if so is the function

i 7→ µg i .

Proof. The “only if” part follows from Proposition B.2.2. For the “if” part,
note that in order to trap µ within some Prokhorov neighborhood of size ", it
is sufficient to compute µg i within a small enough δ, for all i ¶ n for a large
enough n.

A non-computable density function can lead to a computable measure:

Example B.2.8. Let our probability space be the set R of real numbers with
its standard topology. Let a < b be two computable real numbers. Let µ
1
be the probability distribution with density function f (x) = b−a 1[a,b] (x) (the
uniform distribution over the interval [a, b]). Function f (x) is not comput-
able, since it is not even continuous. However, the measure µ is comput-
1
Rb
able: indeed, µg i = b−a g (x)d x is a computable sequence, hence Propo-
a i
sition B.2.7 implies that µ is computable. ù
The following theorem compensates somewhat for the fact mentioned
earlier, that the function µ 7→ µ(U) is generally not computable.

177
B. CONSTRUCTIVITY

Proposition B.2.9. Let µ be a finite computable measure. Then there is a com-

putable map h with the property that for every bounded computable function f
with | f | ¶ 1 with the property µ( f −1 (0)) = 0, if w is the name of f then h(w)
is the name of a program computing the value µ{ x : f (x) < 0 }.
Proof. Straightforward.
Can we construct a measure just using the pattern of Proposition B.2.7?
Suppose that there is a computable function (i, j) 7→ mi ( j) with the following
properties:
(a) i < j1 < j2 implies |mi ( j1 ) − mi ( j2 )| < 2− j1 .
(b) For all n, there is a probability measure µn with mi (n) = µn g i for all
i < n.
Thus, the sequences converge, and for each n, all values mi (n) for i ¶ n are
consistent with coming from a probability measure νn assigning this value to
g i . Is there a probability measure µ with the property that for each i we have
lim j mi ( j) = µg i ? Not necessarily, if the space is not compact.
Example B.2.10. Let X = {1, 2, 3, . . . } with the discrete topology. Define a
probability measure µn with µn g i = 0 for i < n and otherwise arbitrarily.
Since we only posed n − 1 linear conditions on a finite number of variables,
it is easy to see that such a µn exists. Then define mi (n) = µn (i) for all i.
Now all the numbers mi (n) converge with n to 0, but µ = 0 is not a
probability measure. ù
To guarantee that the sequences mi ( j) indeed define a probability mea-
sure, progress must be made, for example, in terms of the narrowing of
Prokhorov neighborhoods.

B.2.3 RANDOM TRANSITIONS

Consider random transitions now.

Definition B.2.1 (Computable kernel). Let X, Y be computable metric spaces,
giving rise to measurable spaces with σ-algebras A , B respectively. Let
Λ = { λ x : x ∈ X } be a probability kernel from X to Y (as defined in Sub-
section A.2.5). Let {g i } be the set of bounded Lipschitz functions over Y
introduced in Definition A.1.21. To each g i , the kernel assigns a (bounded)
measurable function
f i (x) = (Λg i )(x) = λ xy g i ( y).

178
 B.2. Constructive measure theory

We will call the kernel Λ computable if so is the assignment (i, x) 7→ f i (x). ù

When Λ is computable, each function f i (x) is of course continuous. The

measure Λ∗ µ is determined by the values Λ∗ g i = µ(Λg i ), which are comput-
able from (i, µ) and so the mapping µ 7→ Λ∗ µ is computable.
The following example is the simplest case, when the transition is actually
deterministic.
Example B.2.11 (Computable deterministic kernel). A computable function
h : X → Y defines an operator Λh with Λh g = g ◦ h (as in Example A.2.15).
This is a deterministic computable transition, in which f i (x) = (Λh g i )(x) =
g i (h(x)) is, of course, computable from (i, x). We define h∗ µ = Λ∗hµ. ù

179
BIBLIOGRAPHY

[1] Yevgeniy A. Asarin. Individual Random Signals: a Complexity Approach.

PhD thesis, Moscow State University, Moscow, Russia, 1988. In Russian.
4.1.1
[2] Ya. M. Barzdin’. The complexity of programs to determine whether na-
tural numbers not greater than n belong to a recursively enumerable
set. Soviet Math. Doklady, 9:1251–54, 1968. 3.2.2
[3] Ya. M. Barzdin’ and R. Freivald. On the prediction of general recursive
functions. Soviet Math. Doklady, 206:1224–28, 1972. 1.1.2
[4] Charles H. Bennett and Martin Gardner. The random number Omega
bids fair to hold mysteries of the universe. Scientific American,
242(5):20–34, Nov 1979. 1.1.3
[5] Patrick Billingsley. Convergence of Probability Measures. Wiley, 1968.
First edition. Second edition 1999. A.1.5, A.1.13, A.2.6
[6] Manuel Blum and Silvio Micali. How to generate cryptographically
strong sequences of pseudo-random bits. SIAM Journal on Computing,
13:850–864, 1984. 1.1.2
[7] Vasco Brattka and Gero Presser. Computability on subsets of metric
spaces. Theoretical Computer Science, 305:43–76, 2003. B.1.7
[8] Gregory J. Chaitin. Information-theoretical limitations of formal sys-
tems. Journal of the ACM, 21:403–424, 1974. 1.1.2, 1.1.3
[9] Gregory J. Chaitin. Scientific American, 232(5):47–52, May 1975.
1.1.2, 1.1.3
[10] Gregory J. Chaitin. A theory of program size formally identical to in-
formation theory. J. Assoc. Comput. Mach., 22:329–340, 1975. 1.1.3,
1.6.5

181
BIBLIOGRAPHY

[11] Gregory J. Chaitin. Algorithmic entropy of sets. Computers and

Mathem. with Applications, 2:233–245, 1976. 1.1.3

[12] Gregory J. Chaitin. Algorithmic information theory. IBM J. Res.& Dev.,

21:350–359, 1977. 1.1.3

[13] Imre Csiszár and János Körner. Information Theory. Academic Press,
New York, 1981. 1.6.4, 1.6.6, 3.1.1

[14] Robert Daley. Math. Syst. Th., 9(1):83–94, 1975. 1.1.2

[15] Pierre Simon de Laplace. A Philosophical Essay on Probabilities. Dover,

1951. 1.1, 1.1.3

[16] Terrence Fine. Theories of Probability. Academic Press, New York, 1973.
1.1.3

[17] Peter Gács. On the symmetry of algorithmic information. Soviet Math.

Dokl., 15:1477–1780, 1974. 1.1.3, 1.6.5, 3.1.5, 3.1.1

[18] Peter Gács. Exact expressions for some randomness tests. Z. Math. Log.
Grdl. M., 26:385–394, 1980. Short version: Springer Lecture Notes in
Computer Science 67 (1979) 124-131. 1.1.3, 2.3.6

[19] Peter Gács. On the relation between descriptional complexity and al-
gorithmic probability. Theoretical Computer Science, 22:71–93, 1983.
Short version: Proc. 22nd IEEE FOCS (1981) 296-303. 1.1.3

[20] Peter Gács. Randomness and probability–complexity of description. In

Samuel Kotz and Norman L. Johnson, editors, Encyclopedia of Statistical
Sciences, Vol. 7, pages 551–556. Wiley, New York, 1986. 1.1.3

[21] Peter Gács. Uniform test of algorithmic randomness over a general

space. Theoretical Computer Science, 341(1-3):91–137, 2005. 4.6.4

[22] Peter Gács and János Körner. Common information is far less than
mutual information. Problems of Control and Inf. Th., 2:149–162, 1973.
3.1.1

[23] Peter Gács, John Tromp, and Paul Vitányi. Algorithmic statis-
tics. IEEE Transactions on Information Theory, 47:2443–2463, 2001.
arXiv:math/0006233 [math.PR]. 3.1, 3.1.2

182
 Bibliography

[24] Oded Goldreich, Shafi Goldwasser, and Silvio Micali. How to construct
random functions. Journal of the Association for Computing Machinery,
33(4):792–807, 1986. 1.1.2
[25] Peter Hertling and Klaus Weihrauch. Randomness spaces. In Proc.
of ICALP’98, volume 1443 of Lecture Notes in Computer Science, pages
796–807. Springer, 1998. 4.1.1, 4.1.4
[26] A.N. Kolmogorov. Three approaches to the quantitative definition of
information. Problems of Inform. Transmission, 1(1):1–7, 1965. 1.1,
1.1.3
[27] Andrei N. Kolmogorov. Foundations of the Theory of Probability.
Chelsea, New York, 1956. 1.1
[28] Andrei N. Kolmogorov. A logical basis for information theory and prob-
ability theory. IEEE Transactions on Information Theory, IT-14:662–664,
1968. 1.1, 1.1.3
[29] Leonid A. Levin. On the notion of a random sequence. Soviet Math.
Dokl., 14(5):1413–1416, 1973. 1.1.3, 4.1.1, 4.1.1
[30] Leonid A. Levin. Laws of information conservation (nongrowth) and
aspects of the foundations of probability theory. Problems of Inform.
Transm., 10(3):206–210, 1974. 1.1.2, 1.1.3, 1.6.5, 3.1.2
[31] Leonid A. Levin. Uniform tests of randomness. Soviet Math. Dokl.,
17(2):337–340, 1976. 4.1.1, 2, 4.3.5
[32] Leonid A. Levin. Randomness conservation inequalities: Information
and independence in mathematical theories. Information and Control,
61(1):15–37, 1984. 1.1.2, 1.1.3, 2.2.3, 3.1.4, 3.1.2, 4.1.1, 1, 2, 4.3.5
[33] Leonid A. Levin. One-way functions and pseudorandom generators.
Combinatorica, 4, 1987. 1.1.2
[34] Leonid A. Levin and V.V. V’yugin. Invariant properties of information
bulks. In Proceedings of the Math. Found. of Comp. Sci. Conf., volume 53
of Lecture Notes on Comp.Sci., pages 359–364. Springer, 1977. 1.1.3
[35] Ming Li and Paul M. B. Vitányi. Introduction to Kolmogorov Complexity
and its Applications (Second edition). Springer Verlag, New York, 1997.
1.1.3, 2.2.1

183
BIBLIOGRAPHY

[36] Donald W. Loveland. A variant of the Kolmogorov concept of complex-

ity. Information and Control, 12:510, 1969. 1.1.3, 3.2
[37] Per Martin-Löf. The definition of random sequences. Information and
Control, 9:602–619, 1966. 1.1.3, 4.1.1, 4.1.1, 4.2.3
[38] Wofgang J. Paul, Joel Seiferas, and Janos Simon. An information-
theoretical approach to time-bounds for on-line computation. J. Com-
puter and System Sciences, 23:108–126, 1981. 1.1.2
[39] David Pollard. A User’s Guide to Measure-Theoretic Probability. Cam-
bridge Series in Statistical and Probabilistic Mathematics. Cambridge
University Press, Cambridge, U.K., 2001. A.2, A.2.5, A.2.6
[40] Claus Peter Schnorr. Zufälligkeit und Wahrscheinlichkeit. Springer Ver-
lag, New York, 1971. Lecture Notes in Mathematics. 1.1.3
[41] Claus Peter Schnorr. Process complexity and effective random tests. J.
Comput. Syst. Sci, 7:376, 1973. 1.1.3
[42] Claus Peter Schnorr. A review of the theory of random sequences. In
Proc.5-th Int.Congr. of Logic, Meth. and Phil. of Sci., London, Ontario,
1975. 1.1.3
[43] Claus Peter Schnorr. General random sequences and learnable sequen-
ces. J. Symb. Logic, 42:329–340, 1977. 1.1.3
[44] Raymond J. Solomonoff. A formal theory of inductive inference i. In-
formation and Control, 7:1–22, 1964. 1.1.3
[45] Raymond J. Solomonoff. Complexity-based induction systems: Com-
parisons and convergence theorems. IEEE Transactions on Information
Theory, IT-24(4):422–432, July 1978. 1.1.2
[46] Robert Solovay. Unpublished manuscript. 1976. 1.1.3, 1.7.2, 1.7.4
[47] Robert Solovay. On random r.e. sets. In A. I. Arruda et al., editor, Non-
Classical Logic, Model Th. and Computability, pages 283–307. North
Holland, 1977. 1.1.3
[48] Tom Stoppard. Rosencrantz and Guildenstern are Dead. A play. 1.1
[49] Volker Strassen. The existence of probability measures with given
marginals. Annals of Mathematical Statistics, 36:423–439, 1965. A.2.22

184
 Bibliography

[50] J. Ville. Etude Critique de la Notion de Collectif. Gauthier-Villars, Paris,

1939. 1.1.3
[51] Richard von Mises and H. Geiringer. The Mathematical Theory of Prob-
ability and Statistics. Academic Press, New York, 1964. 1.1, 1.1.3
[52] Volodimir G. Vovk and V. V. Vyugin. On the empirical validity of the
Bayesian method. Journal of the Royal Statistical Society B, 55(1):253–
266, 1993. 4.5.3
[53] V. V. V’yugin. On Turing-invariant sets. Soviet Math. Doklady,
229(4):1090–1094, 1976. 1.1.3
[54] V. V. V’yugin. Ergodic theorems for individual random sequences. The-
oretical Computer Science, 207(2):343–361, 1998. 4.2.2
[55] Klaus Weihrauch. Computable Analysis. Springer, 2000. 4.1.1, B.1.1,
B.1.1
[56] D. G. Willis. Computational complexity and probability constructions.
J. ACM, pages 241–259, April 1970. 1.1.3
[57] Andrew C. Yao. Theory and applications of trapdoor functions. In IEEE
Symp. on Foundations of Computer Science, pages 80–91, 1982. 1.1.2
[58] Ann Yasuhara. Recursive Function Theory and Logic. Academic Press,
New York, 1971. 1.1.1
[59] Alexander K. Zvonkin and Leonid A. Levin. The complexity of finite
objects and the development of the concepts of information and ran-
domness by means of the theory of algorithms. Russian Math. Surveys,
25(6):83–124, 1970. 1.1.2, 1.1.3, 3.2

185