Lecture Theoretical Computer Science II:

Foundations of Theoretical Computer Science

Lecture notes
of
V. Claus and E.-R. Olderog

translated by Ievgeniia Karashchuk

Winter semester 2010/11

Contents

I Basic definitions 1

1 Modelling in theoretical computer science . . . . . . . . . . . . . . . . . . . . . . . . 1

2 Logic, sets, relations and functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

3 Alphabets, strings and languages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

II Finite automata and regular languages 9

1 Finite automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2 Closure properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3 Regular expressions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

4 Structural properties of regular languages . . . . . . . . . . . . . . . . . . . . . . . . 26

5 Decidability questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

6 Automatic verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

III Context-free languages and push-down automata 37

1 Context-free grammars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

2 Pumping Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

3 Push-down automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

4 Closure properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

5 Transformation in normal forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

6 Deterministic context-free languages . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

7 Questions of decidability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

i
IV The notion of algorithm 71

1 Turing machines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

2 Grammars . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

V Non-computable functions — undecidable problems 97

1 Existence of non-computable functions . . . . . . . . . . . . . . . . . . . . . . . . . . 97

2 Concrete undecidable problem: halting for Turing machines . . . . . . . . . . . . . . 101

3 Recursive enumerability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

4 Automatic program verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

5 Grammar problems and Post correspondence problem . . . . . . . . . . . . . . . . . 112

6 Results on undecidability of context-free languages . . . . . . . . . . . . . . . . . . . 120

VI Complexity 123

1 Computational complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

2 The classes P and NP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127

3 The satisfiability problem for Boolean expressions . . . . . . . . . . . . . . . . . . . 133

ii
Chapter I

Basic definitions

§1 Modelling in theoretical computer science

Characteristics of the theory:

- from the particular to the general

- use of modelling to answer general questions
- analysis and synthesis of models

Question: What is necessary to describe the syntax of a programming language?

Modelling of languages:
We consider Chomsky-languages and in particular

- regular languages
- context-free languages

We will show: Regular languages are recognized by finite automata.

Context-free languages are recognized by push-down automata.

Question: How can we describe parallel and communicating systems?

Modelling of processes and process calculi:

- Operators for parallelism and communication

Question: How can we describe time-critical systems?

Modelling of real-time automata:

- Extension of finite automata with clocks

Question: What tasks can be accomplished using computers?

Modelling of a computer:

- Turing machines (1936)

1
2 I. Basic definitions

- Grammars (1959)

We will show: These approaches are equivalent. There exist non-computable functions. Ex-
amples of non-computable functions are considered. Finite and push-down automata can
be seen as special cases of Turing machines.

Question: How much time and memory does the computation take?
Modelling of complexity:

- “fast”, i. e., polynomial algorithms

- complexity classes P and N P

Open problem: Does P = N P hold?

Topics of the lecture:

• Automata
• Formal languages and grammars
• Computability
• Complexity

§2 Logic, sets, relations and functions

Below we have put together the notations that will be used in these lecture notes.

• In logical formulae we use logical connectives ¬ (negation), ^ (conjunction), _ (disjunc-

tion), ) (implication) and , (equivalence) as well as quantifiers 8 (universal quantifier )
and 9 (existential quantifier ).

• We can describe finite sets by listing their elements, for example, {a, b, c, d} or {co↵ee, tea, sugar}.
The empty set {} is also denoted by ;. An important example of an infinite set is the set
N of natural numbers: N = {0, 1, 2, 3, ...}.
x 2 X denotes that x is an element of the set X, and x 62 X denotes that x is not an
element of the set X.
The principle of extensionality holds, i. e., two sets are equal if they have the same ele-
ments. For example,
{a, b, c, d} = {a, a, b, c, d, d}.

X ✓ Y means that X is a subset of Y , i. e., 8 x 2 X : x 2 Y . Subsets can be selected from

the given sets using logical formulae. For example,

M = {n 2 N | n mod 2 = 0}

describes the set of even natural numbers.

For sets X, Y ✓ Z there exist standard set-theoretic operations
 3

union: X [Y = {z 2 Z | z 2 X _ z 2 Y }
intersection: X \Y = {z 2 Z | z 2 X ^ z 2 Y }
di↵erence: X Y = X \ Y = {z 2 Z | z 2 X ^ z 62 Y }
complement: X = Z X
˙ stands for the disjoint union of X and Y , i. e., additionally X \ Y = ;
Furthermore X [Y
holds.

|X| and card(X) denote the cardinality or size of the set X. |X| is the number of elements
in the set X when X is finite. E.g., |{co↵ee, tea, sugar}| = 3.

P(Z) and 2Z denote the powerset of a set Z, i. e., the set of all subsets of Z: P(Z) =
{X | X ✓ Z}. In particular, ; 2 P(Z) and Z 2 P(Z) holds.

X ⇥ Y denotes the (Cartesian) product of two sets X and Y , consisting of all pairs where
the first element is from X and the second from Y : X ⇥ Y = {(x, y) | x 2 X ^ y 2 Y }.
In general, X1 ⇥ ... ⇥ Xn denotes the set of all n-tuples, where for every i 2 {1, ..., n} it
holds that the set Xi contains the i-th component: X1 ⇥ · · · ⇥ Xn = {(x1 , ..., xn ) | x1 2
X1 ^ ... ^ xn 2 Xn }.

• Relations are special sets. A (2-place or binary) relation R of two sets X and Y is a subset
of the product X ⇥ Y , i. e., R ✓ X ⇥ Y . The infix-notation xRy is often used to denote
element relation (x, y) 2 R. The domain of R is defined by

dom(R) = {x 2 X | 9 y 2 Y : (x, y) 2 R}

and the range of R by

ran(R) = {y 2 Y | 9 x 2 X : (x, y) 2 R}.

The relation R ✓ X ⇥ Y is
left-unique, if 8 x1 , x2 2 X, y 2 Y : (x1 , y) 2 R ^ (x2 , y) 2 R ) x1 = x2 ,
right-unique, if 8 x 2 X, y1 , y2 2 Y : (x, y1 ) 2 R ^ (x, y2 ) 2 R ) y1 = y2 ,
left-total, if X = dom(R),
right-total, if ran(R) = Y .
The identity relation on X is denoted by idX : idX = {(x, x) | x 2 X}. The inverse relation
of R ✓ X ⇥ Y is R 1 ✓ Y ⇥ X that is defined as follows:

1
8x 2 X, y 2 Y : (x, y) 2 R , (y, x) 2 R

The composition of two relations R ✓ X ⇥ Y and S ✓ Y ⇥ Z is defined as follows: for

all x 2 X and z 2 Z it holds

(x, z) 2 R S , 9 y 2 Y : (x, y) 2 R and (y, z) 2 S.

By a 2-place relation on the set X we understand a relation R ✓ X ⇥ X. This relation is

4 I. Basic definitions

reflexive, if 8 x 2 X : (x, x) 2 R,
or in terms of relations: idX ✓ R,
irreflexive, if ¬ 9 x 2 X : (x, x) 2 R,
or in terms of relations: R \ idX = ;,
symmetric, if 8 x, y 2 X : (x, y) 2 R ) (y, x) 2 R,
or in terms of relations: R = R 1 ,
antisymmetric, if 8 x, y 2 X : (x, y) 2 R ^ (y, x) 2 R ) x = y,
or in terms of relations: R \ R 1 ✓ idX ,
transitive, if 8 x, y, z 2 X : (x, y) 2 R ^ (y, z) 2 R ) (x, z) 2 R,
or in terms of relations: R R ✓ R.
R is an equivalence relation if R is reflexive, symmetric and transitive (note the initial
letters r-s-t of these properties). The equivalence relation R on X divides the set X
into disjoint subsets, i. e., each subset contains pairwise equivalent elements. [x]R is an
equivalence class of x for element x 2 X, i. e., the set

[x]R = {y 2 X | (x, y) 2 R}.

An element of an equivalence class is called a representative of this class because the whole
class can be identified with the representative and the relation R. Hence an arbitrary
element can be chosen as a representative element of its class because

8x, y 2 X : (x, y) 2 R , [x]R = [y]R .

By the index of R we mean the set cardinality of all equivalence classes of R on X.

Notation:
Index(R) = |{ [x]R | x 2 X}|
If it is clear from the context which R is considered, we write [x] instead of [x]R .
An equivalence relation R on X is called a refinement of an equivalence relation S on X
if R ✓ S. Then every equivalence class of R is a subset of some equivalence class of S.
The n-th power of R ✓ X ⇥ X is defined inductively:

R0 = idX and Rn+1 = R Rn

The transitive closure R+ and the reflexive transitive closure R⇤ of R are defined as follows:
[ [
R+ = Rn and R⇤ = Rn
n2N\{0} n2N

• Functions are special relations. A relation f ✓ X ⇥ Y is called a partial function (or a

part
partial image) from X to Y if f is right-unique. It is denoted as follows f : X ! Y .
Instead of (x, y) 2 f we write f (x) = y. f is a (total ) function from X to Y if f is
additionally left-total. This is denoted by f : X ! Y .
A function f : X ! Y is
injective, if f is left-unique,
surjective, if f is right-total,
bijective, if f is injective und surjective.
A bijective function is also called bijection.
 5

§3 Alphabets, strings and languages

In this lecture we will pay special attention to analysing formal languages. For this purpose the
following notation will be used.

• Alphabet = finite set of characters (symbols) = character set

We use A, B, ⌃, as typical names for alphabets and a, b as typical names for characters,
i. e., elements of alphabets.

• String over an alphabet ⌃ = finite sequence of characters from ⌃. In particular, there is

the empty string ". We use u, v, w as typical names for strings.
Example : Let ⌃ = {1, 2, +}. Then 1 + 2 and 2 + 1 are strings over ⌃.
⌃⇤ = set of all strings over ⌃. ⌃ ✓ ⌃⇤ holds.
⌃+ = set of all non-empty strings over ⌃, i. e., ⌃⇤ = ⌃+ [ {"}.
|u| = length of the string u = number of characters that occur in u.
In particular: |"| = 0.
The symbols of a string u with the length n we denote by u1 , . . . , un .
The concatenation of two strings u and v consists of the characters from u, followed by
the characters from v and is denoted by u · v or simply by uv.
Example : The concatenation of 1+ and 2 + 0 is 1 + 2 + 0.
A string v is called substring of a string w if 9u1 , u2 : w = u1 vu2 .
A string v is called prefix of a string w if 9u : w = vu.
A string v is called suffix of a string w if 9u : w = uv.
There may be several occurrences of the same substring in string w.
Example : The string w = 1 + 1 + 1 has two occurrences of the substring 1+ and three
occurrences of the substring 1.

• A (formal) language over an alphabet ⌃ is a subset of ⌃⇤ . We use L as a typical name for

a language.
There exist standard set-theoretic operations on languages :
L1 [ L2 (union)
L1 \ L2 (intersection)
L1 L2 or L1 \L2 (di↵erence)
L =df ⌃⇤ L (complement)

Furthermore, there are special operations on languages.

The concatenation of strings is extended to languages L1 and L2 as follows:

L1 · L2 = {u · v | u 2 L1 and v 2 L2 }.

The n-th power of a language L is defined inductively:

L0 = {"} and Ln+1 = L · Ln

6 I. Basic definitions

The (Kleene) star operator (also Kleene-closure or iteration) of a language L is

[
L⇤ = Ln = {w1 . . . wn | n 2 N and w1 , . . . , wn 2 L}.
n2N

For all L it holds that " 2 L⇤ .

Example : For L1 = {1+, 2+} and L2 = {1 + 0, 1 + 1} is
L1 · L2 = {1 + 1 + 0, 1 + 1 + 1, 2 + 1 + 0, 2 + 1 + 1},
L21 = {1 + 1+, 1 + 2+, 2 + 1+, 2 + 2+},
L⇤1 = {", 1+, 2+, 1 + 1+, 1 + 2+, 2 + 1+, 2 + 2+, 1 + 1 + 1+, 1 + 1 + 2+, ...}.

§4 Bibliography

The following books are recommended for further reading:

• J.E. Hopcroft, R. Motwani & J.D. Ullmann: Introduction to Automata Theory, Languages,
and Computation. Addison-Wesley, 2nd Edition, 2001.

• U. Schöning: Theoretische Informatik – kurzgefasst. Spektrum Akademischer Verlag,

5. Auflage, 2008.

• D. Harel: Algorithmics – The Spirit of Computing. Addison-Wesley, 1987.

The following sources were used during the preparation for the lecture:

J. Albert & T. Ottmann: Automaten, Sprachen und Maschinen für Anwender. BI 1983
(nur einführendes Buch).

E. Börger: Berechenbarkeit, Komplexität, Logik. Vieweg, Braunschweig 1986 (2. Auflage).

W. Brauer: Automatentheorie. Teubner Verlag, Stuttgart 1984.

E. Engeler & P. Läuchli: Berechnungstheorie für Informatiker. Teubner, Stuttgart 1988.

H. Hermes: Aufzählbarkeit, Entscheidbarkeit, Berechenbarkeit. 2. Auflage, Springer-

Verlag, Berlin 1971.

J.E. Hopcroft & J.D. Ullmann: Introduction to Automata Theory, Languages, and Com-
putation. Addison-Wesley, 1979.

A.R. Lewis & C.H. Papadimitriou: Elements of the Theory of Computation. Prentice
Hall, Englewood Cli↵s 1981.

K.R. Reischuk: Einführung in die Komplexitätstheorie. Teubner Verlag, Stuttgart 1990.

A. Salomaa: Computation and Automata. Cambridge University Press, Cambridge 1985.

 7

A. Salomaa: Formal Languages. Academic Press, New York 1973.

F. Setter: Grundbegri↵e der Theoretischen Informatik. Springer-Verlag, Heidelberg 1988

(einführendes Buch).

W. Thomas: Grundzüge der Theoretischen Infomatik. Vorlesung im Wintersemester

1989/90 an der RWTH Aachen.
8 I. Basic definitions
Chapter II

Finite automata and regular

languages

In the following chapter we deal with a simple model of a machine: the finite automaton. We
will see that

• finite automata may be widely used in computer science,

• the languages recognized by finite automata have many structural properties, e.g. repre-
sentability by regular expressions,

• questions about computations performed by finite automata are decidable.

Moreover, finite automata can be easily implemented as tables in programs and as circuits.

§1 Finite automata

We want to use finite automata for recognizing languages. We can imagine a finite automaton
as a machine with final states, which reads characters from a tape, can move the head only to
the right and can print no new symbols on the tape.

Therefore, the transition function of finite automata is often defined as mapping : Q ⇥ ⌃ ! Q,

where Q is a set of states and ⌃ is an input alphabet of the automaton. This automaton can be
applied to the string w 2 ⌃⇤ to check its acceptance.

9
10 II. Finite automata and regular languages

A U T O M A T O N
Sketch:
6 - Reading direction

q2Q
finite

The representation of automata as the so called transition systems is more suitable for graphical
representation and definition of the acceptance behaviour of automata.

1.1 Definition (deterministic finite automaton): A deterministic finite automaton (ac-

ceptor), in short DFA, is a 5-tuple

A = (⌃, Q, , q0 , F ) or A = (⌃, Q, !, q0 , F )

with following properties:

1. ⌃ is the finite input alphabet,

2. Q is a finite set of states,

3. : Q ⇥ ⌃ ! Q is the transition function

resp. ! ✓ Q ⇥ ⌃ ⇥ Q is a deterministic transition relation, i. e.,

8q2Q 8a2⌃ 9 exactly q 0 2 Q : (q, a, q 0 ) 2 !,

4. q0 2 Q is the initial state,

5. F ✓ Q is the set of final states.

Both representations of automata are related as follows :

(q, a) = q 0 , (q, a, q 0 ) 2 !
a
The elements (q, a, q 0 ) 2! are called transitions. We mostly write q ! q 0 instead of (q, a, q 0 ) 2 !.
a a
For given a 2 ⌃ we use ! also as a binary relation ! ✓ Q ⇥ Q:
a
8 q, q 0 2 Q : q ! q 0 , (q, a, q 0 ) 2 !

A DFA can be graphically represented by a finite state diagram. It is a directed graph which
contains a vertex labelled with q for every state q of the automaton and a directed labelled edge
a
for every transition q ! q 0 . The initial state q0 is marked with an incoming arrow. Final states
are labelled with an additional circle.
 11

Example : Consider A1 = ({0, 1}, {q0 , q1 }, !, q0 , {q1 }) with

! = {(q0 , 0, q0 ), (q0 , 1, q1 ), (q1 , 0, q1 ), (q1 , 1, q0 )}.

Then A1 is represented by the following state diagram:

✏0 ✏0

CC ⇤ CC ⇤
⌫C ⇤✏⇤ 1 ⌫C ⇤✏⇤ ⇠
- q0
- q1
⇢⇡
1

1.2 Definition (Acceptance and reachability)::

Let A = (⌃, Q, !, q0 , F ) resp. A = (⌃, Q, , q0 , F ) be a DFA.

a
1. We extend the transition relations ! from input symbols a 2 ⌃ to strings of these symbols
w
w 2 ⌃⇤ and define the respective relations ! inductively:
"
• q ! q 0 if and only if q = q 0
"
or in terms of relations: ! = idQ
aw a w
• q ! q 0 if and only if 9 q 00 2 Q : q ! q 00 and q 00 ! q 0
aw a w
or in terms of relations: !=! !

Similarly we define the extended transition function ⇤

⇤
: Q ⇥ ⌃⇤ ! Q with ⇤
(q, ") = q and ⇤
(q, aw) = ⇤
( (q, a), w)

for all q, q 0 2 Q, a 2 ⌃ and w 2 ⌃⇤ . It holds that:

⇤ w
(q, w) = q 0 , q ! q0

2. The language accepted by A is

w
L(A) = {w 2 ⌃⇤ | 9 q 2 F : q0 ! q} resp. L(A) = {w 2 ⌃⇤ | ⇤
(q0 , w) 2 F }

A language L is finitely acceptable if there is a DFA A with L = L(A).

w
3. A state q 2 Q is reachable in A, if 9w 2 ⌃⇤ : q0 ! q.

Example : For the automaton from the last example it holds that:

L(A1 ) = {w 2 {0, 1}⇤ | w has an odd number of 10 s}.

Remark : For all a1 , . . . , an 2 ⌃ and q, q 0 2 Q :

a1 ·...·an a a
q ! q0 , 9 q1 , . . . , qn 2 Q : q !1 q1 ... qn 1
n
! qn = q 0 .
12 II. Finite automata and regular languages

or in terms of relations:
a1 ·...·an a a
! = !1 · · · !
n

a a
The sequence q !1 q1 ... qn 1
n
! qn is also called transitions sequence.

Example (from the field of compiler construction): A compiler for a programming lan-
guage works in several phases, sometimes run in parallel:

• Lexical analysis : In this phase the so called scanner breaks the input text into the sequence
of tokens. These are identifiers, keywords and delimiters.

• Syntax analysis : The generated token sequence is the input to the so called parser , which
decides whether this sequence is a syntactically correct program.

• Code generation : The syntactic structure recognized by the parser is used to generate
machine code.

• Optimisation : Execution time of the program should be improved mostly by local changes
of machine code.

The lexical analysis is a simple task, which can be accomplished by finite automata. As an
example let us consider the typical structure of identifiers in a programming language. We will
consider a syntax diagram of MODULA

'letter ⇠

Ident - letter ? -
6
&digit ⇡

letter . . .  . . . 

⌫
? ⌫
? ⌫
? ⌫
?
a ... z A ... Z

"
- ... "
- "
- ... "
- -

digit . . . 

⌫
? ⌫
?
0 ... 9

"
- ... "
- -

The identifiers constructed this way can be recognized by the following finite automaton:
 13

⌫
?

a, . . . , Z @ 0, . . . , 9
a, . . . , Z ⌫ ⇠ R⌫
✏
@ a, . . . , Z
@
XX
z ⇠⇠
9
⇢⇡
⇠⇠ XX
⇤ COC ⇤ OCC
⇤ C
⇤ ⇤⇤ C
0, . . . , 9 0, . . . , 9

For the sake of clarity we have labelled the edges of the state diagram with several symbols.

Example (from the field of operating systems): If several programs try to access shared
resources in a multitasking environment, then these attempts need to be synchronized. For
example, consider two programs P1 and P2 which share one printer. P1 and P2 should be
synchronized in such a way that they do not send data to the printer simultaneously. We
construct a finite automaton that would monitor the printer usage by P1 and P2 .

• P1 reports the beginning and the end of printing to the automaton by the symbols b1 and
e1

• P2 behaves similarly to P1 and uses symbols b2 and e2 respectively.

At each point of time the behaviour of P1 and P2 regarding the printer is given by the finite
string

w 2 {b1 , e1 , b2 , e2 }⇤ .

The automaton should accept this string, if

• every Pi uses the printer correctly, i. e., symbols bi and ei alternate in w, starting with bi ,
i = 1, 2.

• P1 and P2 do not use the printer simultaneously, i. e., there is neither substring b1 b2 nor
b2 b1 in string w.

For example, w1 = b1 e1 b2 e2 should be accepted, but w2 = b1 b2 e1 e2 should not be accepted.

The following finite automaton fulfills the task:
14 II. Finite automata and regular languages

?⇠
⌫
e' - e$
⇢⇡
1 2
@ b2
b1
@
⇠
⌫
@ ⇠
R⌫
@
e 1 , e2
⇢⇡ ⇢⇡
@
b1 , b2 , e2@@
⌫
b1 , b2 , e1
R
@ ?

⇤ OCC
⇤⇤ C b1 , b2 , e1 , e2

Non-determinism

In many applications finite automata can be simplified if non-determinism is allowed, i. e., we

consider an arbitrary relation
!✓Q⇥⌃⇥Q

as transition relation. It may happen that for some q 2 Q, a 2 ⌃ there are several successor
states q1 , . . . , qn , so that (in graphical representation) it holds that
⌫
: q1
a ⇠⇠⇠
⌫
⇠⇠ ⇠ ⇠
q ..
XXX ⌫
XXX .
z q
X
a n

After reading a the automaton moves non-deterministically from q to one of the successor states
q1 , . . . , qn . A special case is n = 0; then for q 2 Q and a 2 ⌃ there is no successor state q 0
a
with q ! q 0 . In this case the automaton stops and rejects symbol a. These remarks lead to the
following definition.

1.3 Definition : A non-deterministic finite automata (or acceptor), in short NFA, is a 5-tuple

B = (⌃, Q, !, q0 , F ),

where ⌃, Q, q0 and F are defined as in DFAs and for ! it holds that:

! ✓ Q ⇥ ⌃ ⇥ Q.

a w
Transitions are written (q ! q 0 ) and extended to strings (q ! q 0 ) as in DFAs.
 15

A graphical representation by state diagram remains unchanged.

1.4 Definition (acceptance and equivalence):

(i) The language accepted (or recognized) by NFA B = (⌃, Q, !, q0 , F ) is

w
L(B) = {w 2 ⌃⇤ | 9 q 2 F : q0 ! q}.

By NFA we denote the class of languages accepted by NFAs.

(ii) Two NFAs B1 and B2 are called equivalent if L(B1 ) = L(B2 ) holds.

An NFA recognizes a string w if while reading w it can reach one of the final states. It may be
the case that, while reading w, there exist other transition sequences which end up in non-final
states.

Obviously, a DFA is a special case of an NFA. Thus the equivalence between arbitrary finite
automata is defined.

Example (suffix recognition): Consider an alphabet ⌃ and a string v = a1 . . . an 2 ⌃⇤ with

ai 2 ⌃ for i = 1, . . . , n.

We want to recognize the language

Lv = {wv | w 2 ⌃⇤ },

which consists of all strings over ⌃ with the suffix v.

For this purpose we consider the NFA

Bv = (⌃, {q0 , . . . , qn }, !, q0 , {qn }),

where ! is defined by the following state diagram Bv :

a2⌃
✏
CC ⇤
⌫C ⇤✏⇤ ⌫ ⌫ ⇠
⌫
- q0 a1 - q1 a2 - q2 a3 - an -
qn
⇢⇡
···

Bv behaves non-deterministically in the initial state q0 : while reading a string, Bv can decide
in each occurrence of a1 whether to try to recognize the suffix v. In order to do it, Bv goes to
q1 and now waits for a2 . . . an as a suffix. Should this not be the case, then Bv stops at some
i 2 {1, . . . , n} and a 6= ai .

The question arises: Do NFAs accept more languages than DFAs?

The answer is “no”.

1.5 Theorem (Rabin and Scott, 1959): For every NFA there exists an equivalent DFA.
16 II. Finite automata and regular languages

Proof : Consider an NFA B = (⌃, Q, !, q0 , F ). We introduce a DFA A with L(A) = L(B)

using the following powerset construction:

A = (⌃, P(Q), !A , {q0 }, FA ),

where for S, S 0 ✓ Q and a 2 ⌃ it holds that:

a a
S !A S 0 if and only if S 0 = {q 0 2 Q | 9 q 2 S : q ! q 0 },

Let the set of final states be FA = {S ✓ Q | S \ F 6= ;}.

There exists a state in A for each subset S of the set of states Q of B (we denote such a state by
a
S). S !A S 0 if and only if S 0 is the set of all successor states that can be reached by a-transitions
of non-deterministic automaton B from states of S. It can be graphically represented by:
⇠ ⇠

u a - u
PP
PP
S q u
a PPP
P S0
a
results in S !A S 0

u a - u

⇢⇡ ⇢⇡

We see that !A is deterministic, thus

a
8S✓Q 8a2⌃ 9 exactly one S 0 ✓ Q : S !A S 0 .

Furthermore, for all q, q 0 2 Q, S, S 0 ✓ Q, a 2 ⌃ the following holds:

a
(i) If q ! q 0 and q 2 S,
a
then 9 S 0 ✓ Q : S !A S 0 and q 0 2 S 0 .
a
(ii) If S !A S 0 and q 0 2 S 0 ,
a
then 9 q 2 Q : q ! q 0 and q 2 S.

This way we can easily show that L(A) = L(B). Let w = a1 . . . an 2 ⌃⇤ with ai 2 ⌃ for
i = 1, . . . , n. Then it holds that:
w
w 2 L(B) , 9 q 2 F : q0 ! q
, 9 q1 , . . . , q n 2 Q :
a an
q0 !1 q1 . . . qn 1 ! qn with qn 2 F
, {“)” follows from (i) and “(” follows from (ii)}
9 S1 , . . . , S n ✓ Q :
a a
{q0 } !1A S1 . . . Sn 1 !nA Sn with Sn \ F 6= ;
w
, 9 S 2 FA : {q0 } !A S
, w 2 L(A).
 17

Example : We consider the suffix recognition of 01 in strings over the alphabet ⌃ = {0, 1, 2}.
According to the previous example the language L01 = {w01 | w 2 ⌃⇤ } is recognized by the
NFA B01 with the following diagram:
0,✏
1, 2

CC ⇤
⌫C ⇤✏⇤ ⌫ ⌫ ⇠
- q0 0 - q1 1 - q2
⇢⇡

Now we apply the powerset construction from the Rabin-Scott theorem to B01 . As a result
we get the following DFA A01 :

✏
CC ⇤ 0, 1, 2
⌫C ⇤✏⇤

✏ Q
k
1, 2 Q
@ Q 0, 1, 2
6
Q
@ CC ⇤ Q
@ ⌫ ⌫ Q⌫ ⇠
0, 2 Q
C ⇤
✏
⇤
⌫
R
@
- {q0 } 1 -
⇢⇡
{q1 } {q2 } 0, 2
k 1, 2
Q
Q
26 0 Q 6
'$ Q '$
# '$
#
Q 1
?
Q
!
1 -
{q0 , q1 } {q0 , q2 } {q1 , q2 }
&% 0 "! &%
&% "!
I
@
⇤ COC @ 6
⇤⇤ C @ 0
0 @ 1
@ '$
@#
&
2
{q0 ,q1 ,q2 }

"!
&%

The fragments in boxes ⇤ are unreachable from the initial state {q0 } of A01 . Therefore, A01
can be simplified to the following equivalent DFA :

✏
1, 2

⇣
1, 2
⌫
CC ⇤
⇣ ⇣⇣
⇤✏⇤⇣
- ~ ~ - ~
C⇣⇣ 0 -
) 1
2 0
⇤ OCC
⇤⇤ C
0
18 II. Finite automata and regular languages

The number of states of this DFA can not be further minimized. There are examples, where
the DFA given in the proof of the Rabin-Scott theorem cannot be further simplifed, i. e., if
the given NFA has n states, then in the worst case the DFA actually needs 2n states.

Spontaneous transitions

Automata can be defined even more conveniently, if "-transitions in addition to non-determinism

are allowed. These are the transitions, which an automaton makes spontaneously, without
reading a symbol of the input string.

1.6 Definition : A nondeterministic finite automaton (acceptor) with "-transitions, in short

"-NFA, is a 5-tuple
B = (⌃, Q, !, q0 , F ),

where ⌃, Q, q0 and F are defined as in NFAs or DFAs and for the transition relation ! it holds
that:
! ✓ Q ⇥ (⌃ [ {"}) ⇥ Q.

"
A transition q ! q 0 is called "-transition and is represented in the state diagrams as follows:
⌫ ⌫
q " - 0
q

In order to define the acceptance behaviour of "-NFAs, we need an extended transition relation
w
q ) q 0 . For this purpose we make two terminological remarks.

↵ ↵
• We define for every ↵ 2 ⌃ [ {"} a 2-place relation ! over Q, i. e., ! ✓ Q ⇥ Q ,
↵
8 q, q 0 2 Q holds: q ! q 0 , (q, ↵, q 0 ) 2 ! .
↵
Here we use the standard infix notation for 2-place relations, i. e., q ! q 0 instead of
↵ ↵
(q, q 0 ) 2 !. We call ! the ↵-transition relation.

• Therefore, we can use the composition for such transition relations. For all ↵, 2 ⌃[{"}
↵
we define ! ! as follows: For q, q 0 2 Q it holds that
↵ ↵
q! ! q 0 , 9 q 00 2 Q : q ! q 00 and q 00 ! q 0 .

w
1.7 Definition : Let B = (⌃, Q, !, q0 , F ) be an "-NFA. Then a 2-place relation ) ✓ Q ⇥ Q is
defined inductively for every string w 2 ⌃⇤ :

" " "

• q ) q 0 if 9 n 0: q!
| ·{z· · !} q 0
n-times
 19

aw " a w
• q ) q 0 if q ) ! ) q0,

where q, q 0 2 Q, a 2 ⌃ and w 2 ⌃⇤ .

Remark : For all q, q 0 2 Q and a1 , . . . , an 2 ⌃:

" "
(i) q ) q, but q = q 0 does not follow from q ) q 0 .

(ii)
a ...a
n 0 " a " " a "
1
q =) q , q ) !1 ) · · · ) n
! ) q0
a an 0
, q )1 · · · ) q

"
(iii) It is decidable, whether the relation q ) q 0 holds for the given states q, q 0 2 Q.

Proof : (i) and (ii) follow directly from the definition.

“(iii)”: Let k be the number of states in Q. Then:

" " "

q ) q0 , 9 n  k 1: q!
| ·{z· · !} q 0 .
n-times
"
If at least k transitions ! take place one after the other, then the states repeat in the path from
q to q 0 . So in order to find all states reachable by "-transitions from q, it is sufficient to check
" "
the finitely many sequences with maximum k 1 transitions !. Thus the decidability of q ) q 0
is proved. ⇤

1.8 Definition (acceptance): Let B = (⌃, Q, !, q0 , F ) be a "-NFA.

(i) The language accepted (or recognized) by B is

w
L(B) = {w 2 ⌃⇤ | 9 q 2 F : q0 ) q}.

(ii) Two "-NFAs B1 and B2 are called equivalent if L(B1 ) = L(B2 ).

Obviously NFAs are a special case of "-NFAs. Therefore, the equivalence of NFAs and "-NFAs
can be defined.

Example : For the input alphabet ⌃ = {0, 1, 2} we consider the "-NFA B defined by the
following state diagram:

✏0 ✏1 ✏2

CC ⇤ CC ⇤ CC ⇤
⌫C ⇤✏⇤ ⌫C ⇤✏⇤ ⌫ C ⇤✏⇤ ⇠
- q0 ✏ - q1 ✏ - q2
⇢⇡
20 II. Finite automata and regular languages

Then:
L(B) = {w 2 {0, 1, 2}⇤ | 9 k, l, m 0 : w = 0| .{z
. . 0} 1| .{z . . 2} }
. . 1} 2| .{z
k-times l-times m-times

1.9 Theorem : For every "-NFA there exists an equivalent NFA.

Proof : Consider a "-NFA B = (⌃, Q, !, q0 , F ).

We construct the following NFA A = (⌃, Q, !A , q0 , FA ), where for q, q 0 2 Q and a 2 ⌃ it holds

that:

a a " a "
• q !A q 0 if and only if q ) q 0 in B, therefore ) ! ) q 0 in B
"
• q 2 FA if and only if 9 q 0 2 F : q ) q0

By definition A is a NFA without "-transitions with F ✓ FA . It remains to show that: L(A) =

L(B). Let w = a1 . . . an 2 ⌃⇤ with n 0 and ai 2 ⌃ for i = 1, . . . , n. Then:
w
w 2 L(A) , 9 q 2 FA : q0 !A q
a a
, 9 q 2 FA : q0 !1A · · · !nA q
a an
, 9 q 2 FA : q0 )1 · · · ) q
"
, {“)”: choose q 0 with q ) q 0 .
“(”: choose q = q 0 .}
a an 0
9 q 0 2 F : q )1 · · · ) q
0 w 0
, 9q 2F : q)q
, w 2 L(B)

In the special case w = " the above argument is reduced as follows:

" 2 L(A) , 9 q0 2 FA
"
, 9 q 2 F : q0 ) q
, " 2 L(B)

Remarks :

(i) In the proof FA could also be defined as follows:

( "
F [ {q0 } if 9q2F : q0 ) q
FA =
F otherwise

Compare with the proof in Hopcroft & Ullman.

"
(ii) NFA A can be computed from the given "-NFA B, because the relation q ) q 0 is decidable:
Compare with the previous remark.
 21

(iii) If there are "-cycles in "-NFA B, then the set of states of NFA A can be reduced by the
preceding contraction of "-cycles; every "-cycle can be replaced by one state.

Example : We apply the construction introduced in the proof to the "-NFA B from the previous
example. As the result we have the following NFA A:
✏0 ✏1 ✏2
CC ⇤ CC ⇤ CC ⇤
⌫ C ⇤✏ ⇠ ⌫
⇤ C ⇤✏ ⇠ ⌫
⇤ C ⇤✏⇤ ⇠
- q0 0, 1
- q1 1, 2- q2
⇢⇡ ⇢⇡ ⇣⇣1⇢⇡
⇣⇣
" ⇣⇣0, 1, 2
⇣

It is easy to check that L(A) = L(B) actually holds.

Therefore, we have the following result:

DEA = NEA = "-NEA,

i. e., the classes of languages accepted by DFAs, NFAs and "-NFAs coincide; we call it the class
of finitely accepted languages. If we want to show the properties of this class, we can choose the
type of automaton most suited for this purpose.

§2 Closure properties

Now we explore under which operations the class of finitely accepted languages is closed. For
this purpose we consider the set operations union, intersection, complement, di↵erence as well
as operations on languages such as concatenation and iteration (Kleene star operator), which
were introduced in Chapter I.

2.1 Theorem : The class of finitely acceptable languages is closed under the following opera-
tions:

1. union,

2. complement,

3. intersection,

4. di↵erence,

5. concatenation,

6. iteration.

Proof : Let L1 , L2 ✓ ⌃⇤ be finitely acceptable. Then there are DFAs Ai = (⌃, Qi , !i , q0i , Fi )
with Li = L(Ai ), i = 1, 2, and Q1 \ Q2 = ;. We show that L1 [ L2 , L1 , L1 \ L2 , L1 \L2 , L1 · L2
22 II. Finite automata and regular languages

and L⇤1 are finitely acceptable. For L1 [ L2 , L1 · L2 and L⇤1 we shall provide the accepting
"-NFAs. This makes the task less complicated.

1. L1 [ L2 : Let us construct the "-NFA B = (⌃, {q0 } [ Q1 [ Q2 , !, q0 , F1 [ F2 ), where

q0 2
/ Q1 [ Q2 and
! = {(q0 , ", q01 ), (q0 , ", q02 )} [ !1 [ !2

hold. B is graphically represented as follows:

⌫
◆ ⇣

✏ *✓ ⌘
q01 A1
⌫
- q0
⌫
◆ ⇣
HH
✏Hj
H
✓ ⌘
q02 A2

It is clear that L(B) = L1 [ L2 holds.

2. L1 : Consider the DFA A = (⌃, Q1 , !1 , q01 , Q1 \F1 ). Then for all w 2 ⌃⇤ it holds that:
w
w 2 L(A) , 9 q 2 Q1 \F1 : q01 !1 q
{!1 determ.}
w
, ¬9 q 2 F1 : q01 !1 q
, w2 / L(A1 )
, w2 / L1

Therefore, L(A) = L1 holds.

3. L1 \ L2 : is obvious, because L1 \ L2 = L1 [ L2 holds.

4. L1 \L2 : is obvious, because L1 \L2 = L1 \ L2 holds.

5. L1 · L2 : Construct the "-NFA B = (⌃, Q1 [ Q2 , !, q01 , F2 ) with

! = !1 [ {(q, ", q02 ) | q 2 F1 )} [ !2 .

B is graphically represented as follows:

' $
' 
⌫ ⌫
⌫ z⌫
!1 q1 1 XXX✏ !2 q2 1
XX
- q01 from
F1 { ⌫
..
.
..
.
⇣ ⇣ ⇣⇣
q
1 02 from F2 { ⌫
..
.
A1 q1 m ⇣ ✏ A2 q2 n
& & %

It is easy to show that L(B) = L1 · L2 holds.

 23

6. L⇤1 : For the iteration we construct the "-NFA

B = (⌃, {q0 } [ Q1 , !, q0 , {q0 }), where q0 2
/ Q1 and

! = {(q0 , ", q01 )} [ !1 [{(q, ", q0 ) | q 2 F1 }

hold. B is graphically represented as follows:

' $
✏
' 
⌫
⇡
⌫? ⇠ ⌫ !1 q1
- q0
⇢⇡
✏ - q
01 from F { ⌫
..
.
..
.
$
1

6 A1 qn
&

& ✏ %

Again it is easy to show that L(B) = L⇤1 holds.

Thus the proof is complete. ⇤

Remark : There is also an interesting direct construction of accepting DFAs for the union and
intersection . Let A1 and A2 be as in the above proof. Then we consider the following transition
relation ! ✓ Q ⇥ ⌃ ⇥ Q on the Cartesian product Q = Q1 ⇥ Q2 of the sets of states:

for all q1 , q10 2 Q1 and q2 , q20 2 Q2 and a 2 ⌃ it holds that

a a a
(q1 , q2 ) ! (q10 , q20 ) if and only if q1 !1 q10 and q2 !2 q20 .
a
The relation ! models the simultaneous parallel progress of automata A1 and A2 while reading
the symbol a.

Consider the DFAs

A[ = (⌃, Q, !, (q01 , q02 ), F[ ),
A\ = (⌃, Q, !, (q01 , q02 ), F\ )
with
F[ = {(q1 , q2 ) 2 Q | q1 2 F1 or q2 2 F2 },
F\ = {(q1 , q2 ) 2 Q | q1 2 F1 and q2 2 F2 }.
Then it holds that L(A[ ) = L1 [ L2 and L(A\ ) = L1 \ L2 .

Proof : We show that the statement for A\ holds. For any w 2 ⌃⇤ it holds that:
w
w 2 L(A\ ) , 9 (q1 , q2 ) 2 F\ : (q01 , q02 ) ! (q1 , q2 )
w w
, 9 q1 2 F1 , q2 2 F2 : q01 !1 q1 and q02 !2 q2
, w 2 L(A1 ) and w 2 L(A2 )
, w 2 L1 \ L2
⇤
24 II. Finite automata and regular languages

§3 Regular expressions

With the help of regular expressions we can inductively describe the finitely acceptable languages.
For this purpose we consider a fixed alphabet ⌃.

3.1 Definition (regular expressions and languages):

1. The syntax of regular expressions over ⌃ is given as follows:

• ; and " are regular expressions over ⌃.

• a is a regular expression over ⌃ for every a 2 ⌃.
• If re, re1 , re2 are regular expressions over ⌃, then
(re1 + re2 ), (re1 · re2 ), re⇤ are also regular expressions over ⌃.

2. The semantics of a regular expression re over ⌃ is the language L(re) ✓ ⌃⇤ , which is

inductively defined as follows:

• L(;) = ;
• L(") = {"}
• L(a) = {a} for a 2 ⌃
• L((re1 + re2 )) = L(re1 ) [ L(re2 )
• L((re1 · re2 )) = L(re1 ) · L(re2 )
• L(re⇤ ) = L(re)⇤

3. A language L ✓ ⌃⇤ is called regular if there is a regular expression re over ⌃ with L =

L(re).

In order to save space by omitting some brackets we define priorities for the operators:

⇤ binds stronger than · and · binds stronger than +.

Besides we omit the outer brackets and use the associativity of · and +. The concatenation dot ·
is often omitted.

Example : We use regular expressions to describe some previously considered languages.

1. The language of identifiers considered by the lexical analysis is described by the regular
expression
re1 = (a + . . . + Z)(a + . . . + Z + 0 + . . . + 9)⇤ .

2. The language over {b1 , e1 , b2 , e2 } used for synchronization of two programs which share a
printer is described by the following regular expression

re2 = (b1 e1 + b2 e2 )⇤ (" + b1 + b2 ).

 25

3. Let ⌃ = {a1 , . . . an , b1 , . . . , bm } and v = a1 . . . an . Then the language Lv = {wv | w 2 ⌃⇤ }

of the strings with suffix v is described by the regular expression

re3 = (a1 + . . . an + b1 + . . . + bm )⇤ a1 . . . an .

It holds that: L(re3 ) = Lv .

Now we will show that a more general statement holds:

3.2 Theorem (Kleene): A language is regular if and only if it is finitely acceptable.

Proof : We consider a language L ✓ ⌃⇤ .

“)”: For a regular expression re over ⌃ it holds that L = L(re). We show using induction over
the structure of re that L(re) is finitely acceptable.

Basis: L(;), L(") and L(a) are obviously finitely acceptable for a 2 ⌃.

Induction step: Let L(re), L(re1 ) and L(re2 ) be finitely acceptable. Then L(re1 +re2 ), L(re1 ·re2 )
and L(re⇤ ) are also finitely acceptable, because the class of finitely acceptable languages is closed
under union, concatenation and iteration.

“(”: It holds that L = L(A) for a DFA A with n states. W.l.o.g. A = (⌃, Q, !, 1, F ), where
Q = {1, . . . , n}. For i, j 2 {1, . . . , n} and k 2 {0, 1, . . . , n} we define
w
Lki,j = {w 2 ⌃⇤ | i ! j and 8 u 2 ⌃⇤ , 8 l 2 Q :
u
from (9 v : v 6= ", v 6= w and uv = w) and i ! l holds l  k}.

Lki,j consists of all strings w such that automaton A moves from state i to state j while reading
the string w. Furthermore, while reading the string w, only states with the number less or equal
k occur.

Now we show by induction on k that the languages Lki,j are all regular.

k = 0: For strings from L0i,j no intermediate state can be used. Thus it holds that:
( a
{a 2 ⌃ | i ! j} if i 6= j
L0i,j = a
{"} [ {a 2 ⌃ | i ! j} if i = j

Therefore, L0i,j is regular as a finite subset of ⌃ [ {"}.

k ! k +1: Let the languages Lki,j be regular for all i, j 2 {1, . . . , n}. Then for all i, j 2 {1, . . . , n}
it holds that:
⇤
Lk+1 k k k
i,j = Li,j [ Li,k+1 · (Lk+1,k+1 ) · Lk+1,j ,
k

In order to go from the state i to the state j, the intermediate state k + 1 is either unnecessary,
then Lki,j is sufficient for the description; or the state k + 1 is used as an intermediate state one
or several times, then Lki,k+1 · (Lkk+1,k+1 )⇤ · Lkk+1,j is used for description. Therefore, by induction
on k we get:
Lk+1
i,j is regular.
26 II. Finite automata and regular languages

From the regularity of languages Lki,j we conclude that L itself is regular, because it holds that

[
L = L(A) = Ln1,j .
j2F

Thus we have proved the statement of Kleene. ⇤

Remark : As an alternative to the above construction of a regular expression from a given

finite automaton we can solve guarded regular systems of equations.

§4 Structural properties of regular languages

As regular languages are exactly the finitely acceptable languages, we can derive important
characteristics of regular languages from the finiteness of the sets of states of the accepting
automata. First we consider the so called pumping lemma, which gives us a necessary condition
for a language to be regular. Let ⌃ be an arbitrary alphabet.

4.1 Theorem (the pumping lemma for regular languages): For every regular language
L ✓ ⌃⇤ there exists a number n 2 N, so that for all strings z 2 L with |z| n there is a
decomposition z = uvw with v 6= " and |uv|  n and for all i 2 N it holds that uv w 2 L, i. e.,
i

we can “pump” the substring v and the resulting string will be in the regular language L.

In quantifier notation:

8 regular L ✓ ⌃⇤ 9 n 2 N 8 z 2 L with |z| n

9 u, v, w 2 ⌃⇤ : z = uvw and v =
6 " and |uv|  n and 8 i 2 N : uv i w 2 L

Proof : Let L ✓ ⌃⇤ be regular.

According to the Kleene theorem there is a DFA A = (⌃, Q, !, q0 , F ) with L = L(A). We

choose n = |Q| and consider a string z 2 L with |z| n. Then A must go through at least one
state twice while reading z. More precisely it holds that:

Let z = a1 . . . ar with r = |z| n and ai 2 ⌃ for i = 1, . . . , r and let q1 , . . . , qr 2 Q be defined

a
as follows: qi 1 !i qi for i = 1, . . . , r. Then there is j, k 2 {1, . . . , n} with 0  j < k  n  r, so
that qj = qk holds. In graphical representation:
 27

'ak $

⌫ ⌫
? ⌫
aj aj+1 ak 1
- q0 a1- · · · - qj - ··· - qk 1

ak+1
?
..
.

?⇠
ar
⌫
qr
⇢⇡

Let u = a1 . . . aj , v = aj+1 . . . ak , w = ak+1 . . . ar . Then according to the properties of j and k

it holds that v 6= " and |uv|  n. Besides, it is clear that the automaton A, while recognizing,
can go through the qj -loop any number of times, i. e., for all i 2 N it holds that: uv i w 2 L. ⇤

We consider a typical application of the pumping lemma, where we prove that a certain language
is not regular.

Example : The language L = {an bn | n 2 N} is not regular. We provide a proof by contradiction.

Hypothesis: L is regular. According to the pumping lemma, there is an n 2 N with given

properties. Now consider z = an bn . Since |z| n holds, we can break z into z = uvw with v 6= "
and |uv|  n, so that for all i 2 N it holds: uv i w 2 L. However, v consists only of symbols a, so
that in particular uw = an |v| bn 2 L should hold. Contradiction

The above example shows that finite automata cannot count unlimitedly. We shall get ac-
quainted with other applications of the pumping lemma in the section devoted to decidability
properties.

Nerode relation

If we consider the so called Nerode relation, we get a characteristic, i. e., necessary and sufficient
condition for the regularity of languages L ✓ ⌃⇤ .

4.2 Definition :

Let L ✓ ⌃⇤ be some language. The Nerode relation of L is a 2-place relation ⌘L on ⌃⇤ , i. e.,

⌘L ✓ ⌃⇤ ⇥ ⌃⇤ , which is defined as follows for u, v 2 ⌃⇤ :

u ⌘L v if and only if for all w 2 ⌃⇤ it holds that uw 2 L , vw 2 L.

Therefore, it holds that u ⌘L v if u and v can be extended in the same way to strings from L.
In particular, u 2 L , v 2 L (with w = ").

Remark : The Nerode relation is a right congruence, i. e., it has the following properties:

1. ⌘L is an equivalence relation on ⌃⇤ , i. e., reflexive, symmetric and transitive.

28 II. Finite automata and regular languages

2. ⌘L is right compatible with the concatenation, i. e., from u ⌘L v it follows uw ⌘L vw for

all w 2 ⌃⇤

Given that ⌘L is an equivalence relation, we can determine the index of ⌘L , i. e., the number
of equivalence classes of ⌘L .

4.3 Theorem (Myhill and Nerode):

A language L ✓ ⌃⇤ is regular if and only if ⌘L ✓ ⌃⇤ ⇥ ⌃⇤ has a finite index.

Proof : “)”: Let L be regular, i. e., L = L(A) for a DFA A = (⌃, Q, !, q0 , F ). We introduce
the following 2-place relation ⌘A on ⌃⇤ . For u, v 2 ⌃⇤ it holds that:
u v
u ⌘A v if and only if we have a q 2 Q, with q0 ! q and q0 ! q,

i. e., if the automaton A moves from q0 to the same state q after having read the inputs u and v.
Note that q is uniquely defined, because A is deterministic. The relation ⌘A is an equivalence
relation on ⌃⇤ .

We show: ⌘A is a refinement of ⌘L , i. e., for all u, v 2 ⌃⇤ it holds that:

For u ⌘A v it follows that u ⌘L v.

Let u ⌘A v and w 2 ⌃⇤ . Then it holds that:

u w
uw 2 L , 9 q 2 Q 9 q 0 2 F : q0 ! q ! q 0
, {u ⌘A v}
v w
9 q 2 Q, q 0 2 F : q0 ! q ! q 0
, vw 2 L.

Thus there are at least as many equivalence classes of ⌘A as of ⌘L . Therefore, it holds that

Index(⌘L )
 Index(⌘A )
= number of states reachable from q0
 |Q|,

thus ⌘L has a finite index.

“(”: Let L ✓ ⌃⇤ be a language and k 2 N the finite index of ⌘L . We choose k strings

u1 , . . . , uk 2 ⌃⇤ with u1 = " as representatives of the equivalence class of ⌘L . Then ⌃⇤ can be
represented as a disjoint union of these equivalence classes:

⌃⇤ = [u1 ] [˙ . . . [˙ [uk ].

In particular, for every string u 2 ⌃⇤ there is an i 2 {1, . . . , k} with [u] = [ui ].

Now we construct the following equivalence-class automaton

 29

AL = (⌃, QL , !L , qL , FL ):

QL = {[u1 ], . . . , [uk ]},

qL = [u1 ] = ["]
FL = {[uj ] | uj 2 L},

and for i, j 2 {1, . . . , k} and a 2 ⌃ let

a
[ui ] !L [uj ] if and only if [uj ] = [ui a].

Then AL is a DFA and for all strings w 2 ⌃⇤ it holds that:

w
["] !L [uj ] if and only if [uj ] = [w],

more precisely for w = a1 . . . an

w a a
["] !L [uj ] if and only if ["] !1 L [a1 ] . . . !
n
L [a1 . . . an ] = [uj ],

and thus

w 2 L(AL ) =
w
, 9 [uj ] 2 FL : ["] !L [uj ]
, 9 uj 2 L : [uj ] = [w]
, w2L

So AL accepts the language L. Therefore, L is regular. ⇤

We use the method of proof of the Myhill-Nerode theorem in order to minimize the number
of states of a DFA. We mainly refer to the deterministic equivalence-class automaton AL from
the proof.

4.4 Corollary : Let L ✓ ⌃⇤ be regular and k = Index(⌘L ). Then every DFA that accepts L
has at least k states. The minimal number k is reached by the DFA AL . However, there may
exist NFAs with less than k states accepting L.

Proof : In the proof of the Myhill-Nerode theorem we have shown in “)” that every DFA
A = (⌃, Q, !, q0 , F ) accepting L has at least k states. At the same time k = Index(⌘L )  |Q|.
In “(” we have constructed the DFA AL with k states accepting L. ⇤

The equivalence-class automaton AL is the prototype of all DFAs accepting L with minimal
number of states k. We can show that every other DFA accepting L and that has k states is
isomorphic to AL , i. e., we can get it from AL by a bijective renaming of the states.

4.5 Definition : Two DFAs or NFAs Ai = (⌃, Qi , !i , q0i , Fi ), i = 1, 2, are called isomorphic if
there is a bijection : Q1 ! Q2 with the following properties:

• (q01 ) = q02 ,
30 II. Finite automata and regular languages

• (F1 ) = { (q) | q 2 F1 } = F2 ,
a a
• 8 q, q 0 2 Q1 8 a 2 ⌃ : q !1 q 0 , (q) !2 (q 0 ).

The bijection is called the isomorphism from A1 to A2 .

Note that isomorphism is an equivalence relation on finite automata. Now we will show the
announced statement.

4.6 Theorem : Let L ✓ ⌃⇤ be regular and k = Index(⌘L ). Then every DFA A that accepts L
and that has k states is isomorphic to AL .

Proof : We have A = (⌃, Q, !, q1 , F ) with L(A) = L and |Q| = k and the equivalence-class
automaton from the Myhill-Nerode theorem with QL = {[u1 ], . . . , [uk ]} and u1 = ". For every
u
i 2 {1, . . . , k} we define the state qi 2 Q by transition q1 !i qi .

Note that qi is uniquely defined, because the transition relation ! is deterministic.

Now we define the mapping : QL ! Q by

([ui ]) = qi

for i 2 1, . . . , k and show that is an isomorphism from AL to A.

u uj
1. is injective: Let qi = qj . Then it holds that q1 !i qi and q1 ! qi . Therefore, for all
w 2 ⌃⇤ it holds that ui w 2 L , uj w 2 L. Therefore, ui ⌘L uj and [ui ] = [uj ].

2. is surjective: this property follows from (1) and the fact that k = |Q|. Thus it holds in
particular that Q = {q1 , . . . , qk }.

3. ([qL ]) = ([u1 ]) = (["]) = q1

4. (FL ) = F : [uj ] 2 FL , uj 2 L , qj 2 F

5. For all i, j 2 {1, . . . , k} and a 2 ⌃ it holds that:

a a
[ui ] !L [uj ] , qi ! qj .

a
• Proof of “)”: Let [ui ] !L [uj ]. Then by definition of !L : [ui a] = [uj ]. There is an
a
l 2 {1, . . . , k} with qi ! ql . By definition of qi and ql we have the following image:
ui- a- q
q1 qi l

⇢ ⇡
6
ul

Given that the strings ui a and ul lead to the same state ql , it follows that ui a ⌘L ul .
Therefore, it holds that [uj ] = [ui a] = [ul ]. According to the choice of u1 , . . . , uk in AL it
a
follows that uj = ul , even j = l. Thus qi ! qj follows as desired.
 31

a
• Proof of “(”: Let qi ! qj . Thus we have the following image similar to the one above:
ui- a - qj
q1 qi

⇢ ⇡
6
uj

Hence we make a conclusion as above: ui a ⌘L uj . Therefore, it holds that [ui a] = [uj ]; by

a
definition of !L we can derive that [ui ] !L [uj ].

Thus AL and A are isomorphic. ⇤

For every regular language L ✓ ⌃⇤ with k as index of ⌘L there is a DFA which is unique up to
isomorphism and with minimal number of states k accepting L.

4.7 Definition : The minimal automaton for a regular language L ✓ ⌃⇤ is the DFA which
accepts L and has the number of states equal to the index of the Nerode relation ⌘L . This
minimal automaton is unique up to isomorphism.

The minimal automaton for a regular language L ✓ ⌃⇤ from every DFA A = (⌃, Q, !, q0 , F )
accepting L by reduction is algorithmically computable. The reduction includes the following
steps:

1. Eliminate unreachable states.

w
A state q 2 Q is called reachable if there is a string w 2 ⌃⇤ with q0 ! q. The subset of
reachable states of Q is computable, because we can consider only those strings w with
w
q0 ! q which have the length  |Q| (compare with the proof of the pumping lemma).

2. Combine equivalent states.

w w
For q 2 Q, w 2 ⌃⇤ and S ✓ Q we write q ! S if there is a q 0 2 S with q ! q 0 . Two states
q1 , q2 2 Q are called equivalent, in short q1 ⇠ q2 , if for all w 2 ⌃⇤ it holds:

w w
q1 ! F , q2 ! F,

i. e., the same strings lead from q1 and q2 to final states.

u
There is a close relation between equivalence and the Nerode relation ⌘L . Let q0 ! q1
v
and q0 ! q2 . Then it holds that:

q1 ⇠ q2 , u ⌘L v , [u] = [v].

Comparing with the equivalence-class automaton AL we see that equivalent states must
coincide in the minimal automaton. q0 , q1 and q2 in AL are represented by the equivalence
u v
classes ["], [u] and [v] such that from ["] ! [u] and ["] ! [v] it follows that:

[u] ⇠ [v] , u ⌘L v , [u] = [v].

32 II. Finite automata and regular languages

§5 Decidability questions

First of all, we determine that the following constructions are algorithmically computable:

• "-NFA ! NFA ! DFA

• DFA ! minimal automaton

• "-NFAs for the following operations on finitely acceptable languages:

union, complement, intersection, di↵erence, concatenation and iteration

• regular expression ! NFA ! DFA ! regular expression

After that we can move on to decidability questions for the languages represented by finite
automata or regular expressions. Due to the constructions mentioned above we consider only
the languages represented by DFAs. We consider the following problems for regular languages.

acceptance problem Given: DFA A and a string w

Question: Does w 2 L(A) hold ?
emptiness problem Given: DFA A
Question: Does L(A) = ; hold ?
finiteness problem Given: DFA A
Question: Is L(A) finite ?
equivalence problem Given: DFA ’s A1 and A2
Question: Does L(A1 ) = L(A2 ) hold ?
inclusion problem Given: DFA ’s A1 and A2
Question: Does L(A1 ) ✓ L(A2 ) hold ?
intersection problem Given: DFA ’s A1 and A2
Question: Does L(A1 ) \ L(A2 ) = ; hold ?

5.1 Theorem (decidability): For regular languages

• the acceptance problem,

• the emptiness problem,

• the finiteness problem,

• the equivalence problem,

• the inclusion problem,

• the intersection problem

 33

are all decidable.

Proof : Acceptance problem: We apply A to the given string w and decide whether a final state
of A is reached. Therefore, A itself gives us the decision procedure.

Emptiness problem: Let n be the number we get by applying the pumping lemma to the regular
language L(A). Then it holds that:

L(A) = ; , ¬9 w 2 L(A) : |w| < n (⇤)

The proof of (⇤): “)” is obvious. “(” by contraposition: Let L(A) 6= ;. If there is a string
w 2 L(A) with |w| < n, then we have nothing to show. Otherwise there is a string w 2 L(A)
with |w| n. By successive application of the pumping lemma with i = 0 we get a string
w0 2 L(A) with |w0 | < n. This completes the proof.

The decision procedure for L(A) = ; now solves the acceptance problem “w 2 L(A)?” for every
string over the input alphabet of A with |w| < n. If the answer is “no”, then we get “L(A) = ;”.
Otherwise we get “L(A) 6= ;”.

Finiteness problem: Let n be defined as above. Then it holds that:

L(A) is finite , ¬9 w 2 L(A) : n  |w| < 2n (⇤⇤)

The proof of (⇤⇤): “)”: If there was a string w 2 L(A) with |w| n, then L(A) would be
infinite by the pumping lemma. “(” by contraposition: Let L(A) be infinite. Then there are
strings of arbitrary length in L(A), in particular a string w with w 2n. By applying the
pumping lemma successively with i = 0 we get a string w0 with n  |w0 | < 2n, because with
i = 0 the given string is shortened maximally by n letters.

The finiteness problem can be decided by (⇤⇤) by solving of the acceptance problem a finite
number of times.

equivalence problem: First we construct a DFA A with the following property:

L(A) = (L(A1 ) \ L(A2 )) [ (L(A2 ) \ L(A1 )).

Obviously it holds that:

L(A1 ) = L(A2 ) , L(A) = ; (⇤ ⇤ ⇤)

Hence the equivalence problem for automata is reduced to the emptiness problem for A. The
construction of A described above is hard to implement. Alternatively, we can use the following
product construction, which is similar to the last remarks in section 2 of this chapter. Let
Ai = (⌃, Qi , !i , q0i , Fi ), i = 1, 2. Then consider Q = Q1 ⇥ Q2 with the following transition
relation ! ✓ Q ⇥ ⌃ ⇥ Q

for all q1 , q10 2 Q1 and q2 , q20 2 Q2 and a 2 ⌃ it holds that

a a a
(q1 .q2 ) ! (q10 , q20 ) if and only if q1 ! q10 and q2 ! q20 . Then define A = (⌃, Q, !, (q01 , q02 ), F )
with F = {(q1 , q2 ) 2 Q | q1 2 F1 , q2 2 / F2 }.
34 II. Finite automata and regular languages

Then (***) holds for this DFA A:

L(A1 ) = L(A2 ) , 8 w 2 ⌃⇤ : (w 2 L(A1 ) , w 2 L(A2 ))

w w
, 8 w 2 ⌃⇤ : ((9 q1 2 F1 : q0 ! q1 ) , (9 q2 2 F2 : q0 ! q2 ))
w
, 8 w 2 ⌃⇤ 8 (q1 , q2 ) 2 Q : (q01 , q02 ) ! (q1 , q2 ) ) (q1 , q2 ) 62 F
, L(A) = ;

Inclusion problem: We construct a DFA A such that

L(A) = L(A1 ) \ L(A2 )

holds. Due to
L(A1 ) ✓ L(A2 ) , L(A) = ;

the inclusion problem for A1 and A2 can be reduced to the emptiness problem of A.

Intersection problem: We construct a DFA A with

L(A) = L(A1 ) \ L(A2 ),

so that the intersection problem of A1 and A2 can be reduced to the emptiness problem of A.
For A we can use the product construction A\ from the last remark in section 2. ⇤

§6 Automatic verification

The automatic verification can be implemented for programs represented by finite automata. In
general, the verification problem can be formulated as follows:

Given : program P and specification S

Question : Does P fulfil the specification S ?
This problem is also called Model Checking, because in terms of logic the question is whether
the program P is a model of specification S. Here we consider the following special case of this
problem:
^
• Program P = finite automaton (DFA, NFA or "-NFA) A
^
• Specification S = an extended regular expression re, i. e., extended by the operators
re1 \re2 (intersection), re (complement) and ⌃ as an abbreviation
for a1 + . . . + an if ⌃ = {a1 , . . . , an } holds.
^
• P satisfies S = L(A) ✓ L(re). This test can be conducted automatically, because
the inclusion problem for regular languages is decidable.
Example : Consider once again the example from the field of operating systems, where two
programs share a printer by means of the operations b1 and b2 , and report the end of use by
e1 and e2 . We put ⌃ = {b1 , b2 , e1 , e2 } and describe the set of strings over ⌃, which define the
allowed executions by the following extended regular expression:
 35

re = ⌃⇤ b1 b2 ⌃⇤ + ⌃⇤ b2 b1 ⌃⇤ .

The application of the operators re and ⌃ makes the regular expression re simpler. Due to
closure properties of regular languages, we certainly do not leave the class of regular languages.
Hence we can also decide for every given finite automaton A for synchronization of printer usage
by both programs whether L(A) ✓ L(re) holds.

One of the possibilities to define the satisfiability is as follows:

^
• P satisfies S = L(A) = L(re). This test can be automatically conducted, because
the equivalence problem for languages is decidable.
Even the following synthesis problem can be solved automatically for program P represented as
a finite automaton and specification S represented as extended regular expression:

Given : a specification S
Searched : a program P that satisfies S.
36 II. Finite automata and regular languages
Chapter III

Context-free languages and

push-down automata

In the previous chapter we have seen that regular languages have multiple applications in com-
puter science (e.g. lexical analysis and substring recognition) and are particularly easy to use
(representability by finite automata and regular expressions, good closure and decidability prop-
erties). However these are not enough to carry out an important task of computer science, namely
the syntax description of programming languages.

The reason is that programming languages accept bracket structures of arbitrary nesting-depth,
such as for example

• arithmetic expressions of the form 3 ⇤ (4 (x + 1)),

• lists of the form (CAR(CONS x y )) or

• statements of the form

while(b1) {
x = e1;
while(b1) {
y = e2;
z = e3;
}
}

In section 4 we have shown with the help of the pumping lemma that the simplest example of
such a bracket structure, namely the language

L = {an bn | n 2 N}

is no longer regular. Context-free languages are used for the syntax description of programming
languages.

37
38 III. Context-free languages and push-down automata

§1 Context-free grammars

First we consider the corresponding grammars.

1.1 Definition : A context-free grammar is a 4-tuple G = (N, T, P, S), where the following
holds:

(i) N is an alphabet of non-terminal symbols,

(ii) T is an alphabet of terminal symbols with N \ T = ;,

(iii) S 2 N is the start symbol,

(iv) P ✓ N ⇥ (N [ T )⇤ is a finite set of productions or rules.

We use the following formal definitions:

• A, B, C, . . . stand for non-terminal symbols,

• a, b, c, . . . stand for terminal symbols,

• u, v, w, . . . stand for strings with terminal and non-terminal symbols.

We often write down a production (A, u) 2 P as an arrow notation A ! u. If several productions

have the same left side, such as

A ! u1 , A ! u2 , . . . , A ! uk ,

then we write it down shorter as a unique “metarule”

A ! u1 | u2 | . . . | uk

or also

A ::= u1 | u2 | . . . | uk . (⇤)

Thus | is a “metasymbol” for the disjunction of alternatives u1 , . . . , uk , which may not occur in
N [ T.

If the productions of a context-free grammar are represented in the form (⇤), then we deal with
Backus-Naur form or shortly BNF-notation. This notation was introduced in 1960 by John
Backus and Peter Naur to define the programming language ALGOL 60. The extended BNF-
notation, also called EBNF, allows us to make further abbreviations. The EBNF-notation can
be translated 1–1 into the syntax diagram introduced in 1970 by Niklaus Wirth to define the
programming language PASCAL.
 39

Every context-free grammar G has the two-place derivation relation `G on (N [ T )⇤ :

x `G y if and only if 9A ! u 2 P 9w1 , w2 2 (N [ T )⇤ :

x = w1 A w2 and y = w1 u w2 .

By `⇤G we denote the reflexive and transitive closure of `G . We read x `⇤ y as “y can be derived
from x”. The language generated by G is

L(G) = {w 2 T ⇤ | S `⇤G w}.

Two context-free grammars G1 and G2 are called equivalent if L(G1 ) = L(G2 ).

1.2 Definition : A language L ✓ T ⇤ is called context-free if there is a context-free grammar G

with L = L(G).

Example :

(1) The language L = {an bn | n 2 N} is generated by the grammar

G1 = ({S}, {a, b}, P1 , S), where P1 is given as follows:

S ! " | aSb.

For example, it holds that a2 b2 2 L(G1 ), because

S `G1 aSb `G1 aaSbb `G1 aabb.

(2) The arithmetic expressions with variables a, b, c and operators + and ⇤ are generated by
the grammar G2 = ({S}, {a, b, c, +, ⇤, (, )}, P2 , S) with the following P2 :

S ! a | b | c | S + S | S ⇤ S | (S).

For example (a + b) ⇤ c 2 L(G2 ), because

S `G2 S ⇤ S `G2 (S) ⇤ S `G2 (S + S) ⇤ S

`G2 (a + S) ⇤ S `G2 (a + b) ⇤ S `G2 (a + b) ⇤ c.
Now we will consider the derivation of strings in a context-free grammar in more detail.

1.3 Definition : A derivation from A to w in G with the length n 0 is a sequence of derivation

steps

A = z0 `G z1 `G · · · `G zn = w. (⇤⇤)

This derivation is called leftmost derivation if we replace the leftmost non-terminal symbol in
every derivation step zi `G zi+1 , i.e. if zi and zi+1 always have the form

zi = w1 Aw2 and zi+1 = w1 uw2 , where w1 2 T ⇤ .

The rightmost derivations are defined respectively (then it holds that: w2 2 T ⇤ ).

Every derivation can be graphically represented as a tree.

1.4 Definition : A derivation tree from A to w in G is a tree with the following properties:
40 III. Context-free languages and push-down automata

(i) Every node is labelled with a symbol from N [ T [ {"}. The root is labelled with A and
every internal node is labelled with a symbol from N .

(ii) If an internal node labelled with B has k children nodes, which are labelled with the
symbols 1 , . . . , k from left to right, then it holds that

a) k = 1 and 1 = " and B ! " 2 P

or
b) k 1 and 1, . . . , k 2 N [ T and B ! 1... k 2 P.

(iii) The string w is generated by concatenating the symbols on the leaves from left to right.

Illustration

A
··
◆◆· · ·S·S
◆· · S
◆ ·· · S
◆ ·· B S
◆ ·· @S
@
◆ ··· S
◆ B 1 k S
◆ S
◆ " S
| {z }
w

We construct the derivation tree from A to w corresponding to a derivation from A to w of the

form (⇤⇤) by induction on the length n of the derivation.

n = 0 : The trivial derivation A belongs to the trivial derivation A.

n ! n + 1 : Consider a derivation

A = z0 `G . . . `G zn = w1 Bw2 `G w1 uw2 = zn+1 .

Let t be the derivation tree corresponding to the derivation A = z0 `G . . . `G zn .

If u = ", then the entire derivation tree is as follows:

A
⌦⌦ JJ
⌦ t J
⌦ J
⌦ B J
w1 w2

"

If u = 1... k, where 1, . . . , k 2 N [ T , then the entire derivation tree is as follows:

 41

A
⌦⌦ JJ
⌦ t J
⌦ J
⌦ B J
w1 w2
AA
1 ··· k

Example : Derivation trees for the derivations mentioned in the previous example are

S S
@
@ @
@
a S b and S ⇤ S
@
@ @
@
a S b ( S ) c
@
@
" S + S

a b

Remark : There exists the following relationship between derivations and derivation trees:

(i) A `⇤G w , There is a derivation tree from A to w in G.

(ii) In general, several derivations from A to w correspond to the given derivation tree from A
to w. However, this holds for only one leftmost derivation and one rightmost derivation.

Proof :

(i) “)” is obvious due to the above construction.

We show “(” inductively on the depth of the derivation tree.

(ii) Derivation trees abstract from the unimportant order of the rule application if several
non-terminal symbols occur simultaneously. For example, both derivations

S ` G 2 S + S ` G 2 a + S `G 2 a + b

and
S `G 2 S + S ` G 2 S + b ` G 2 a + b
result in the same derivation tree:
S
@
@
S + S

a b

If we have decided to use leftmost or rightmost derivation respectively, then such alterna-
tives are not possible.
42 III. Context-free languages and push-down automata

Now let the syntax of a programming language PL be given by a context-free grammar G. A

compiler for PL generates for every given PL-program a derivation tree in G in the syntax
analysis phase. The meaning or semantics of the PL-program depends on the structure of the
generated derivation tree. The compiler generates the machine code of the PL-program on the
basis of the derivation tree.

For using the programming language PL it is important that every PL-program has an unam-
biguous semantics. Therefore, for every PL-program there should exist exactly one derivation
tree.

1.5 Definition :

(i) A context-free grammar G = (N, T, P, S) is called unambiguous if for every string w 2 T ⇤

there is at most one derivation tree or a leftmost derivation from S to w in G respectively.
Otherwise G is ambiguous.

(ii) A context-free language L ✓ T ⇤ is called unambiguous if there is an unambiguous context-

free grammar G with L = L(G). Otherwise L is called (inherently) ambiguous.

Example : The above given grammar G2 for arithmetic expressions is ambiguous. For the
string a + b ⇤ c 2 L(G2 ) there exist the following two derivation trees:

S S
@
@ @
@
S + S and S ⇤ S
@
@ @
@
a S ⇤ S S + S c

b c a b

These correspond to semantically di↵erent bracketings a + (b ⇤ c) and (a + b) ⇤ c.

Therefore, in programming languages we choose a grammar G3 for arithmetic expressions, where

the following evaluation strategy is chosen:

• The evaluation takes place from left to right. Thus, for example, a + b + c is evaluated as
(a + b) + c.

• Multiplication ⇤ has a higher priority than +. Thus, for example, a + b ⇤ c is evaluated as

a + (b ⇤ c).

If we want to have another evaluation sequence, we must explicitly put brackets ( and ).
 43

Consider G3 = ({E, T, F }, {a, b, c, +, ⇤, (, )}, P3 , E) with the following P3 :

E ! T |E + T
T ! F |T ⇤ F
F ! (E) | a | b | c

G3 is unambiguous, and it holds that L(G3 ) = L(G2 ). For example, a + b ⇤ c has the derivation
tree in G3

E
@
@
E + T
@
@
T T ⇤ F

F F c

a b

Example : (Chomsky, 1964) An inherent ambiguous context-free language is

L = {ai bj ck | i, j, k 0 and (i = j or j = k)}.

For the proof see the literature.

44 III. Context-free languages and push-down automata

§2 Pumping Lemma

There also exists a pumping lemma for context-free languages. It provides a necessary condition
for a given language to be context-free.

2.1 Theorem (Pumping lemma for context-free languages or uvwxy-lemma): For

every context-free language L ✓ T ⇤ there exists a number n 2 N such that for all strings z 2 L
with |z| n there exists a decomposition z = uvwxy with the following properties:

(i) vx 6= ",

(ii) |vwx|  n,

(iii) for all i 2 N it holds that: uv i wxi y 2 L.

Hence we can “pump” the substrings v and x an arbitrary number of times without leaving the
context-free language L.

To prove the pumping lemma we need the following general lemma for trees, which we can then
apply to derivation trees. For finite trees t we define:

• Branching factor of t = maximum number of children nodes of any node in t.

• A path of length m in t is a sequence of edges from the root to a leaf of t with m edges.
The trivial case m = 0 is allowed.

2.2 Lemma : Let t be a finite tree with the branching factor  k, where every path has the
length  m. Then the number of leaves in t is  k m .

Proof : Induction on m 2 N:

m = 0: t consists only of k 0 = 1 nodes.

m ! m + 1: t has j subtrees t1 , . . . , tj with j  k, where the paths have the length  m:

t= r
⇢Z
⇢ Z
r Zr
⇢ Z
⇢ ···
⌦
⌦JJ ⌦⌦JJ
⌦ t J · · · ⌦ tj J
⌦ 1 J ⌦ J
⌦ J ⌦ J

By induction hypothesis the number of leaves in each of the subtrees t1 , . . . , tj is  k m . Therefore,

it holds for t that:
number of leaves  j · k m  k · k m = k m+1 .

⇤
 45

Now we are ready for the

Proof of the pumping lemma: Let G = (N, T, P, S) be a context-free grammar with L(G) =
L. Let:

• k = the length of the longest right side of a production from P , however at least 2

• m = |N |,

• n = k m+1 .

Now let z 2 L with |z| n. Then there is a derivation tree t from S to z in G with no part
corresponding to a derivation of the form B `⇤G B:

S
◆◆ ... SS
◆ S
◆ B S
◆ ◆ S S
◆ ◆ SS
◆ ◆ ..S S
◆ ◆S .
◆S S
◆ ◆ "S ◆" S S
◆ ◆ SB◆ S S
⌦ J
⌦ J

Every part like this could actually be removed from t without changing the derived string z.

Having chosen k and |z| as above, we can see that t has a branching factor  k and k m+1
leaves. Therefore, according to the previously considered lemma there is a path with the length
m + 1 in t. There are m + 1 internal nodes on this path, so that a non-terminal symbol is
repeated, while labeling these nodes. We need this repetition in a special case.

By a repetition tree in t we denote a subtree of t, where the labeling of the root is repeated in
some other node. Now we choose a minimal repetition tree t0 in t, i.e. such a tree that contains
no other repetition tree (as a real subtree). In t0 every path has a length  m + 1.

Let A be the root labeling of t0 . Then t has the following structure:

t= S
⇢ . Z
⇢ .. Z
⇢ Z
⇢ t0 = A Z
⇢⇢ .. Z Z
⇢⇢ . Z Z
⇢ ⇢ A Z Z
⇢ ⇢ Z Z
⇢ ⇢ @ Z Z
⇢ ⇢ @ Z Z
u v w x y
| {z }
z
46 III. Context-free languages and push-down automata

From this structure we get a decomposition z = uvwxy with

S `⇤G uAy `⇤G uvAxy `⇤G uvwxy. (⇤)

We show that this decomposition of z satisfies the conditions of the pumping lemma:

(i) By the choice of t it holds that vx 6= ".

(ii) By the choice of t0 and the preceding pumping lemma it holds that |vwx|  k m+1 = n.

(iii) From (⇤) it follows immediately that for all i 2 N it holds that: uv i wxi y 2 L(G).

The derivation tree from S to uv i wxi y in G for i = 3 looks as follows:

S
⇢ . Z
⇢ .. Z
⇢ Z
⇢ A Z
⇢ ⇢ .. Z Z
⇢ ⇢ . Z Z
⇢ ⇢ A Z Z
⇢ ⇢ Z Z
⇢ ⇢ ⇢ .
⇢ . Z
. Z Z Z
⇢ ⇢ ⇢ Z Z Z
u v⇢ A Zx y
⇢ .. Z
v . @
@ x
A @
A @
v w x

Just as in regular languages, the above pumping lemma can be used to prove that a certain
language is not context-free.

Example : The language L = {an bn cn | n 2 N} is not context-free. We provide a proof by

contradiction.

Assumption: L is context-free. Then according to the pumping lemma there is an n 2 N with

given properties. Now consider z = an bn cn . Because |z| n holds, we can decompose z into
z = uvwxy with vx 6= " and |vwx|  n such that for all i 2 N it holds that uv i wxi y 2 L.
Because |vwx|  n, no a’s or no c’s occur in the substring vwx. Therefore, while pumping up
to uv i wxi y, at most two of the characters a, b, c will be considered. However, such strings are
not in L. Contradiction. ⇤

The pumping lemma allows us to show that context-free grammars are not enough to provide the
full description of the syntax of more advanced programming languages such as Java. Though
we describe the basic structure of the syntactically correct programs with context-free grammars
(in BNF or EBNF notation), there exist side conditions, which are context-sensitive.
 47

Example : The programming language Java, which consists of all syntactically correct Java
programs, is not context-free. We make a proof by contradiction.

Hypothesis: Let Java be context-free. Then there is an n 2 N with the properties mentioned in
the pumping lemma. Now let us consider the following syntactically correct Java-class:

class C {
int X 1| .{z
. . 1};
n
void m() {
X 1| .{z
. . 1} = X 1| .{z
. . 1}
n n
}
}

By every uvwxy-decomposition of this program, the vwx-part influences at most two out of
. . 1}, because |vwx|  n. Therefore, while pumping to
three occurrences of the variables X 1| .{z
n
uv i wxi y, there appear either character strings which do not fulfill the requirements of the Java
syntax diagram, or character strings of the form

class C {
int X 1| .{z
. . 1};
k
void m() {
X 1| .{z
. . 1} = X 1| .{z
. . 1}
l m
}
}

where k, l, m are not all equal. These character strings violate the following condition for syn-
tactically correct Java programs:

“Every variable must be declared before use.” (⇤⇤)

Here either X 1
| .{z
. . 1} or X 1
| .{z
. . 1} (or both of them) is not declared.
l m
Contradiction. ⇤

Such conditions as (⇤⇤) are context-sensitive and thus are mentioned while defining program-
ming languages together with the context-free basic structure of programs. A compiler checks
these context-sensitive conditions by applying appropriate tables for saving declared variables
or general identifiers.
48 III. Context-free languages and push-down automata

§3 Push-down automata

So far we defined context-free languages by the fact that they can be generated by grammars.
Now we want to consider recognizability of such languages by automata. Our goal is to extend
the model of the finite automaton in such a way that it can recognize the context-free languages.

The weak point of finite automata is that they lack memory to store an unlimited amount of
data. Obviously, a finite automaton cannot recognize such a language as

L = {an bn | n 2 N}

because at the moment the first b is read, it does not know any more how many a’s have been
read. The only information saved is the current state, i.e. an element of a finite set of states.

Now we consider the so called push-down automata. These are nondeterministic finite automata
with "-transitions ("-NFAs), extended with memory which can save infinitely long strings but
can be accessed very limitedly. The memory is organized as a stack or a pushdown-list with
access only to the topmost symbol. The transitions of a push-down automaton may depend on
the current state, on the symbol of the input string which has been read, and on the topmost
symbol of the stack; they change the state and the contents of the stack.

Sketch

a a a b b b input string
6 - reading direction
⌃⌅

q2Q - A topmost symbol

finite A
control #
stack

3.1 Definition (push-down automaton): A (nondeterministic) push-down automaton, shortly

PDA, is a 7-tuple
K = (⌃, Q, , !, q0 , Z0 , F )

with the following properties:

(i) ⌃ is the input alphabet,

(ii) Q is a finite set of states,

(iii) is the stack alphabet,

(iv) ! ✓ Q ⇥ ⇥ (⌃ [ {"}) ⇥ Q ⇥ ⇤ is the transition relation,

 49

(v) q0 2 Q is the start state,

(vi) Z0 2 is the start symbol of the stack,

(vii) F ✓ Q is the set of final states.

While speaking about PDAs we typically use the following characters: a, b, c 2 ⌃, u, v, w 2 ⌃⇤ ,

↵ 2 ⌃ [ {"}, q 2 Q, Z 2 , 2 ⇤ . These characters can also be “decorated” with indices and
primes. The elements (q, Z, ↵, q 0 , 0 ) 2 ! are called transitions. Instead of (q, Z, ↵, q 0 , 0 ) 2 !
↵
we often write (q, Z) ! (q 0 , 0 ).
a
We can illustrate it as follows. A transition (q, Z) ! (q 0 , 0 ) means: if q is the current state
and Z is the topmost stack symbol, then the PDA can read the input symbol a, can move to
"
the state q 0 and replace the symbol Z by the string 0 . Similarly a transition (q, Z) ! (q 0 , 0 )
means: in the state q and Z as a topmost stack symbol PDA can make a transition into the
state q 0 and replace Z by 0 on the top of the stack. In this case no input symbol is read.
↵
If 0 = ", then in a transition (q, Z) ! (q 0 , ") we speak about a pop step, because the topmost
↵
stack symbol is removed. If 0 = Z, then we speak about a push step in a transition (q, Z) !
(q 0 , Z), because the string is added to the top of the stack.

In order to define the acceptance behaviour of PDAs, we must – just as in "-NFAs – first of all
extend the transition relation. For this purpose we need the definition of the configuration of a
PDA.

3.2 Definition : Let K = (⌃, Q, , !, q0 , Z0 , F ) be a PDA.

(i) By a configuration of K we understand a pair (q, ) 2 Q ⇥ ⇤ which describes the current

state q and the current stack contents of K.
↵
(ii) For every ↵ 2 ⌃ [ {"}, ! is a 2-place relation on configurations of K defined as follows:
↵
(q, ) ! (q 0 , 0 ) if 9Z 2 , 9 0, 1 2 ⇤
:

=Z 0 and (q, Z, ↵, q 0 , 1) 2 ! and 0

= 1 0
↵
By ! we denote the ↵-transition relation on configurations.
w
(iii) For every string w 2 ⌃⇤ , ) is a 2-place relation on configurations of K defined inductively:
" " "
• (q, ) ) (q 0 , 0 ) if 9n 0 : (q, ) ! . . !}(q 0 , 0 )
| .{z
n-times
aw " a w
• (q, ) ) (q 0 , 0) if (q, ) ) ! ) (q 0 , 0 ), for all a 2 ⌃.
w
By ) we denote the extended w - transition relation on configurations.

Remark : For all ↵ 2 ⌃ [ {"}, a1 , . . . , an 2 ⌃ and w1 , w2 2 ⌃⇤ it holds that:

↵ ↵
(i) (q, ) ! (q 0 , 0 ) implies (q, ) ) (q 0 , 0 ).
50 III. Context-free languages and push-down automata

a ...a " a " " a "

1
(ii) (q, ) =)n
(q 0 , 0 ) if and only if (q, ) ) !1 ) . . . ) ! n
) (q 0 , 0 )
a an
if and only if (q, ) )1 . . . ) (q 0 , 0 )
w w w w
1 2
(iii) (q, ) =) (q 0 , 0 ) if and only if (q, ) )1 )2 (q 0 , 0 )

There exist two variants of language acceptance.

3.3 Definition (acceptance):

Let K = (⌃, Q, , !, q0 , Z0 , F ) be a PDA and w 2 ⌃⇤ .

w
(i) K accepts w if 9q 2 F 9 2 ⇤ : (q0 , Z0 ) =) (q, ).
The language accepted (or recognized) by K is

L(K) = {w 2 ⌃⇤ | K accepts w}.

(ii) K accepts w with the empty stack,

if
w
9q 2 Q : (q0 , Z0 ) =) (q, ").

The language accepted (or recognized) by K with an empty stack is

L" (K) = {w 2 ⌃⇤ | K accepts w with an empty stack}.

Example : We construct a push-down automaton which accepts the mentioned language L =

{an bn | n 2 N}. Let
K = ({a, b}, {q0 , q1 , q2 }, {a, Z}, !, q0 , Z, {q0 }),

where the transition relation ! consists of the following transitions:

a
(1) (q0 , Z) ! (q1 , aZ)
a
(2) (q1 , a) ! (q1 , aa)
b
(3) (q1 , a) ! (q2 , ")
b
(4) (q2 , a) ! (q2 , ")
"
(5) (q2 , Z) ! (q0 , ")

K accepts an bn for n 1 by using the following 2n + 1 transitions, which we have labelled for
the sake of clarity with numbers from 1 to 5:
a a a
(q0 , Z) !1 (q1 , aZ) !2 (q1 , aaZ) . . . !2 (q1 , an Z)
b b b
!3 (q2 , an 1 Z) !4 (q2 , an 2 Z) . . . !4 (q2 , Z)
"
!5 (q0 , ")

an b n "
Thus the relation (q0 , Z) =) (q0 , ") holds for n 1. For n = 0 it trivially holds that (q0 , Z) )
(q0 , Z). Because q0 is the final state of K, it follows that L ✓ L(K).

For the inclusion L(K) ✓ L we must analyse the transition behaviour of K. For this purpose we
index the transitions as above. We can see that K is deterministic, i.e. while sequentially reading
 51

characters of a given input string, only one transition can be applied on each step. Namely the
following transition sequences are possible, where n m 0 holds:
⇣ ⌘n
a a
• (q0 , Z) !1 !2 (q1 , an+1 Z)
⇣ ⌘n ⇣ ⌘m
a a b b
• (q0 , Z) !1 !2 !3 !4 (q2 , an m Z)

⇣ ⌘n ⇣ ⌘n
a a b b "
• (q0 , Z) !1 !2 !3 !4 !5 (q0 , ")

Therefore, K accepts only those strings which have the form an bn . Altogether, it holds that
L(K) = L.

Furthermore, note that the automaton K accepts all strings an bn with the empty stack, except
for the case n = 0: L" (K) = {an bn | n 1}.

Now we want to derive general properties of push-down automata. For this purpose we often
need the following Top Lemma. Intuitively, it points out that changes on the top of the stack
do not depend on the rest of the stack.

3.4 Lemma (top of the stack): Let K = (⌃, Q, , !, q0 , Z0 , F ) be a push-down automaton.

Then for all w 2 ⌃⇤ , q, q 0 2 Q, Z 2 and 2 ⇤ it holds that: if
w
(q, Z) =) (q 0 , "),

then also
w
(q, Z ) =) (q 0 , ).

Proof : Exercise ⇤

Now we show that both variants of language acceptance are equivalent.

3.5 Theorem (acceptance):

(1) For every PDA A we can construct a PDA B with L(A) = L" (B).

(2) For every PDA A we can construct a PDA B with L" (A) = L(B).

Proof : Let A = (⌃, Q, , !A , q0 , Z0 , F ).

(1): The idea of the proof is simple: B works like A and empties the stack in final states.
However, we should take care of the fact that B should not have an empty stack after having
read input strings which A does not accept. Therefore, B uses an additional symbol # to label
the bottom of the stack. To be more precise, we construct:

B = (⌃, Q [ {qB , q" }, [ {#}, !B , qB , #, ;)

with qB , q" 2
/ Q and # 2
/ and the following transition relation:
52 III. Context-free languages and push-down automata

!B = {(qB , #, ", q0 , Z0 #)} “starting A”

[ !A ) “working like A”
[ {(q, Z, ", q" , ") | q 2 F, Z 2 [ {#}}
“emptying the stack”
[ {(q" , Z, ", q" , ") | Z 2 [ {#}}

Then it holds for all w 2 ⌃⇤ , q 2 F and 2 ⇤:

w
(q0 , Z0 ) =)A (q, )

if and only if
" w "
(qB , #) !B (q0 , Z0 #) =)A (q, #) !B (q" , ").

(For the “if-then” direction we use the Top Lemma.) Analysing the applicability of the new
transitions in B we get L(A) = L" (B).

(2): Idea of the proof: B works like A, but uses an additional symbol # to label the bottom
of the stack. As soon as A has emptied its stack, B reads the symbol # and moves to a final
state. Making the exact construction of B should be done as an exercise. ⇤

Now we want to show that the nondeterministic push-down automata accept context-free lan-
guages (with empty stack). First for a given context-free grammar G we construct a push-down
automaton which represents a nondeterministic “top-down” parser of the language L(G).

3.6 Theorem : For every context-free grammar G we can construct a nondeterminstic push-
down automaton K with L" (K) = L(G).

Proof : Let G = (N, T, P, S). We construct K in such a way that it simulates the leftmost
derivation in G:
K = (T, {q}, N [ T, !, q, S, ;),

where the transition relation ! consists of the following types of transitions:

"
(1) (q, A) ! (q, u), if A ! u 2 P,
a
(2) (q, a) ! (q, ") , if a 2 T.
K works as follows: first S is on the stack. A rule application A ! u of the grammar is executed
on the stack by replacing the topmost stack symbol A by u. If a terminal symbol is at the top
of the stack, we compare it with the next symbol of the input string and, if they are equal, we
remove it from the stack. This way we produce stepwise by K a leftmost derivation of the input
string. If this derivation is successful, then K accepts the input string with the empty stack.
The application of transitions of type (1) is nondeterministic if there are several rules with the
same non-terminal symbol A in P . The only state q is not important for the transition behavior
of K, but must be indicated in order for the definition of K to be complete.

To show that L(G) = L" (K) holds, we consider more carefully the relationship between left-
most derivations in G and transition sequences in K. For this purpose we use the following
abbreviations of strings w 2 (N [ T )⇤ :
 53

• wT is the longest prefix of w with wT 2 T ⇤ ,

• wR is the remainder of w, defined by w = wT wR .

Hypothesis 1 For all A 2 N, w 2 (N [ T )⇤ , n 0 and leftmost derivations

A ` G . . . `G w
| {z }
n-times

of the length n holds

T w
(q, A) =) (q, wR ).

Proof by induction on n:
"
n = 0: Then w = A, thus wT = " and wR = A. Trivially (q, A) ) (q, A) holds.

n ! n + 1: We analyse the last step of a leftmost derivation with the length n + 1:

A `G . . . `G w̃ = w̃T Bv `G w̃T uv = w
| {z }
n-times

for B 2 N and u, v 2 (N [ T )⇤ with B ! u 2 P . By the induction hypothesis it holds that

T e
w
(q, A) =) (q, Bv).

The transition type (1) implies that

"
(q, Bv) ) (q, uv).

The transition type (2) also implies that

(uv)T
(q, uv) =) (q, (uv)R ).

Because wT = (w̃T uv)T = w̃T (uv)T and wR = (w̃T uv)R = (uv)R hold, then in general we get
T w
(q, A) =) (q, wR ).

Thus we have proved statement 1.

Hypothesis 2 For all A 2 N, m 0, ↵1 , . . . , ↵m 2 T [ {"}, 0 , . . . , m 2 (N [ T )⇤ and all

transition sequences
↵1 ↵m
(q, A) = (q, 0 ) ! (q, 1 ) · · · ! (q, m )

of the length m holds

A `⇤G ↵1 . . . ↵m m.

Proof by induction on m:

m = 0: Then 0 = A. Trivially A `⇤G A holds.

m ! m + 1: We analyse the last transition

↵m+1
(q, m) ! (q, m+1 ).
54 III. Context-free languages and push-down automata

By the induction hypothesis A `⇤G ↵1 . . . ↵m m.

Case ↵m+1 = "

Then we used transition type (1) and the transition has the form
"
(q, m) = (q, Bv) ! (q, uv) = (q, m+1 )

for some B 2 N and u, v 2 (N [ T )⇤ , where B ! u 2 P . Thus it holds

A `⇤G ↵1 . . . ↵m Bv `G ↵1 . . . ↵m uv = ↵1 . . . ↵m ↵m+1 m+1 .

Case ↵m+1 = a 2 T
Then we used transition type (2) and the transition has the form
a
(q, m) = (q, av) ! (q, v) = (q, m+1 )

for some v 2 (N [ T )⇤ . Then it holds that

A `⇤G ↵1 . . . ↵m av = ↵1 . . . ↵m ↵m+1 m+1 .

Thus we have also proved statement 2.

Particularly from the statements 1 and 2 it follows that for all strings w 2 T ⇤ it holds that:
w
S `⇤G w if and only if (q, S) ) (q, ")
if and only if K accepts w with the empty stack.

Thus L(G) = L" (K) holds as required. ⇤

Example : We consider again the language L = {an bn | n 2 N}. We have already seen that the
language is generated by the context-free grammar G1 = ({S}, {a, b}, P1 , S), where P1 consists
of the productions
S ! " | aSb,
i.e. L(G1 ) = L. The construction used in the proof above gives us the push-down automaton

K1 = ({a, b}, {q}, {S, a, b}, !, q, S, ;),

where the transition relation ! consists of the following transitions:

"
(q, S) ! (q, ")
"
(q, S) ! (q, aSb)
a
(q, a) ! (q, ")
b
(q, b) ! (q, ")

It follows from the proof that L" (K1 ) = L(G1 ). To illustrate this, let us consider the transition
sequence of K1 while accepting a2 b2 :
" a
(q, S) ! (q, aSb) ! (q, Sb)
" a
! (q, aSbb) ! (q, Sbb)
" b b
! (q, bb) ! (q, b) ! (q, ").
 55

Now for every given push-down automaton we construct a respective context-free grammar.

3.7 Theorem : For every push-down automaton K we can construct a context-free grammar G
with L(G) = L" (K).

Proof : Let K = (⌃, Q, , !, q0 , Z0 , F ). We construct G = (N, T, P, S) with T = ⌃ and

N = {S} [ {[q, Z, q 0 ] | q, q 0 2 Q and Z 2 }.

The idea of non-terminal symbols [q, Z, q 0 ] is as follows:

(1) All strings w 2 ⌃⇤ which K can accept from the configuration (q, Z) with empty stack and
w
the state q 0 should be generated in G from [q, Z, q 0 ]: (q, Z) ) (q 0 , ").
↵
(2) Thus a transition (q, Z) ! (r0 , Z1 . . . Zk ) from K is simulated in G by the following pro-
ductions:
[q, Z, rk ] ! ↵[r0 , Z1 , r1 ][r1 , Z2 , r2 ] . . . [rk 1 , Zk , rk ]

for each r1 , . . . , rk 2 Q. The strings which are accepted by K up to the reduction of the
symbol Z1 are generated from [r0 , Z1 , r1 ]; the strings accepted by K up to the reduction
of the symbol Z2 are generated from [r1 , Z2 , r2 ], and so on. The intermediate states
r1 , . . . , rk 1 are those states which K reaches directly after the reduction of the symbols
Z1 , . . . , Z k 1 .

To make it more precise, P consists of the following transitions:

• Type (1): S ! [q0 , Z0 , r] 2 P for all r 2 Q,

↵
• Type (2): For every transition (q, Z) ! (r0 , Z1 . . . Zk ) with ↵ 2 ⌃ [ {"} and k 1 in K:
[q, Z, rk ] ! ↵[r0 , Z1 , r1 ] . . . [rk 1 , Zk , rk ] 2 P for all r1 , ..., rk 2 Q.
↵
• Type (3): (Special case (2) for k = 0.) For every transition (q, Z) ! (r0 , ") in K:
[q, Z, r0 ] ! ↵ 2 P .

To show that L(G) = L" (K) holds, consider the relationship between derivations in G and
transition sequences in K.

Hypothesis 1 For all q, q 0 2 Q, Z 2 , w 2 ⌃⇤ , n 1 and derivations in G

[q, Z, q 0 ] `G . . . `G w
| {z }
n-times

with the length  n holds for K

w
(q, Z) ) (q 0 , ").

Proof by induction on n:
56 III. Context-free languages and push-down automata

n = 1: From [q, Z, q 0 ] `G w it follows (because w 2 ⌃⇤ ) that we deal with the production type
↵ ↵
(3) in G. Therefore, it holds that w = ↵ 2 ⌃ [ {"} and (q, Z) ! (q 0 , "). Thus (q, Z) ) (q 0 , ").

n ! n + 1: We analyse the first step of a derivation with the length n + 1, which should take
place according to the production type (2):

[q, Z, rk ] `G ↵ [r0 , Z1 , r1 ] . . . [rk 1 , Zk , rk ] `G . . . `G ↵ w1 . . . wk = w,

| {z }
n-times
↵
where (q, Z) ! (r0 , Z1 . . . Zk ) in K, rk = q 0 , ↵ 2 ⌃ [ {"}, w1 , . . . , wk 2 ⌃⇤ and

[ri 1 , Zi , ri ] `G . . . `G w i
| {z }
n-times

for i = 1, . . . , k holds. Due to induction it holds in K that

w
(ri 1 , Zi ) )i (ri , ")

for i = 1, . . . , k and thus according to the Top Lemma

↵ w
(q, Z) ! (r0 , Z1 . . . Zk ) )1 (r1 , Z2 . . . Zk )
..
.
w
(rk 1 , Zk ) )k (rk , ").
w
So in conclusion (q, Z) ) (q 0 , ") for K as required.

Hypothesis 2 For all q, q 0 2 Q, Z 2 , n 1, ↵1 , . . . , ↵n 2 ⌃[{"} and all transition sequences

↵ ↵
(q, Z) !1 · · · !n (q 0 , ")

in K with the length n it holds in G that

[q, Z, q 0 ] `⇤G ↵1 . . . ↵n .

Proof by induction on n:
↵
n = 1: Then (q, Z) !1 (q 0 , ") holds. By definition of P in G — see production type (3) — it
follows that [q, Z, q 0 ] `G ↵1 .

n ! n + 1: We analyse the first step of a transition sequence in K with the length n + 1:

↵ ↵ ↵n+1
(q, Z) !1 (r0 , Z1 . . . Zk ) !2 · · · ! (q 0 , "),

where k 1 holds. By definition of P there is a production of type (2) in G

[q, Z, q 0 ] ! ↵1 [r0 , Z1 , r1 ] · · · [rk 1 , Zk , rk ],

where rk = q 0 . We consider in more detail the successive reduction of the stack contents Z1 . . . Zk
of K. Furthermore, there exist transition sequences in K
11↵ ↵1m
(r0 , Z1 ) ! ··· !1 (r1 , ")
........................................
↵k1 ↵kmk
(rk 1 , Zk ) ! ··· ! (rk , ") = (q 0 , ")
 57

with ↵2 . . . ↵n+1 = ↵11 . . . ↵1m1 . . . ↵k1 . . . ↵kmk and m1 , . . . , mk  n. By condition of induction

it holds in G that
[ri 1 , Zi , ri ] `⇤G ↵i1 . . . ↵imi

for i = 1, . . . , k. In conclusion it results in G that

[q, Z, q 0 ] `⇤G ↵1 ↵2 . . . ↵n+1

as required.

The hypotheses 1 and 2 imply: for all q 2 Q and w 2 ⌃⇤ it holds in G

S `G [q0 , Z0 , q] `⇤G w

if and only if in K
w
(q0 , Z0 ) ) (q, ")

holds. Thus we have shown that L(G) = L" (K). ⇤

58 III. Context-free languages and push-down automata

§4 Closure properties

Now we consider, under which operations the class of context-free languages is closed. In contrast
to the regular (i.e. finitely acceptable) languages we have the following results.

4.1 Theorem : The class of context-free languages is closed under the following operations

(i) union,

(ii) concatenation,

(iii) iteration,

(iv) intersection with regular languages.

However, the class of context-free languages is not closed under the operations

(v) intersection,

(vi) complement.

Proof : Let L1 , L2 ✓ T ⇤ be context-free. Then there are context-free grammars Gi =

(Ni , T, Pi , Si ) with L(Gi ) = Li , where i = 1, 2 and N1 \N2 = ;. First we show that L1 [L2 , L1 ·L2
and L⇤1 are context-free. After that we consider the operations intersection and complement.
Let S 2 / N1 [ N2 be a new start symbol.

(i) L1 [ L2 : Consider the context-free grammar G = ({S} [ N1 [ N2 , T, P, S) with

P = {S ! S1 , S ! S2 } [ P1 [ P2 . Obviously L(G) = L1 [ L2 holds.

(ii) L1 · L2 : Consider the context-free grammar G = ({S} [ N1 [ N2 , T, P, S) with

P = {S ! S1 S2 } [ P1 [ P2 . Obviously L(G) = L1 · L2 holds.

(iii) L⇤1 : Consider the context-free grammar G = ({S} [ N1 , T, P, S) with

P = {S ! ", S ! S1 S} [ P1 . Then S `⇤G S1n holds for all n 0 and thus L(G) = L⇤1 .

(iv) L1 \ regSpr : For the intersection with regular languages we use the representation of
context-free and regular languages by push-down automata and finite automata, respec-
tively. Let L1 = L(K1 ) for the (nondeterministic) PDA K1 = (T, Q1 , , !1 , q01 , Z0 , F1 )
and L2 = L(A2 ) for the DFA A2 = (T, Q2 , !2 , q02 , F2 ). We construct from K1 and A2 the
(nondeterministic) PDA

K = (T, Q1 ⇥ Q2 , , !, (q01 , q02 ), F1 ⇥ F2 ),

where the transition relation ! for q1 , q10 2 Q1 , q2 , q20 2 Q2 , Z 2 , ↵ 2 T [ {"} and 0 2 ⇤

is defined as follows:
↵
((q1 , q2 ), Z) ! ((q10 , q20 ), 0 ) in K
 59

if and only if
↵ ↵
(q1 , Z) !1 (q10 , 0 ) in K1 and q2 !2 q20 in A2 .
"
Note that in the special case ↵ = " the notation q2 !2 q20 for the DFA A2 simply means
w
q2 = q20 (compare with the definition of the extended transition relation q !2 q 0 for DFA’s
↵
in section 1). Thus the relation ! of K models the synchronous parallel progress of the
automata K1 and A2 . However, in the special case ↵ = " only K1 makes a spontaneous
"-transition, while the DFA A2 remains in the current state.

We show that for the acceptance by final states it holds that: L(K) = L(K1 ) \ L(A2 ) =
L1 \ L2 . Let w = a1 . . . an 2 T ⇤ , where n 0 and ai 2 T for i = 1, . . . , n. Then it holds
that:
⇤ a a
w 2 L(K) , 9(q1 , q2 ) 2 F, 2 : ((q01 , q02 ), Z0 ) )1 · · · )
n
((q1 , q2 ), )
⇤ a1 an
, 9q1 2 F1 , q2 2 F2 , 2 : (q01 , Z0 ) )1 · · · )1 (q1 , )
a a
and q02 !12 · · · !n2 q2
, w 2 L(K1 ) \ L(A2 ).

(v) not L1 \ L2 : However, the context-free languages are not closed under intersection with
other context-free languages. Consider

L1 = {am bn cn | m, n 0}

and
L2 = {am bm cn | m, n 0}.

It is easy to see that L1 and L2 are context-free. For example, L1 can be generated by the
context-free grammar G = ({S, A, B}, {a, b, c}, P, S} with the following P :

S ! AB,
A ! " | aA,
B ! " | bBc.

The intersection of L1 and L2 is the following language

L1 \ L2 = {an bn cn | n 0},

which is not context-free (as we have shown with the help of the pumping lemma).

(vi) not Li : The context-free languages are also not closed under the complement. This
statement follows directly from (i) and (v) due to the De Morgan’s laws. If context-
free languages were closed against the complement, then they would also be closed under
intersection, because L1 \ L2 = L1 [ L2 holds.

⇤
60 III. Context-free languages and push-down automata

§5 Transformation in normal forms

While considering context-free languages, it is beneficial when the rules of the underlying
context-free grammars are as simple as possible. Therefore, in this section we introduce such
transformations, which transform the given context-free grammars into equivalent grammars
with the rules satisfying additional conditions.

5.1 Definition : An "-production is a rule of the form A ! ". A context-free grammar G =

(N, T, P, S) is called "-free if in G there is

(i) either no "-production at all

(ii) or only the "-production S ! " and then S does not occur on the right-hand side of any
production in G.

In case (i) it holds that " 2

/ L(G) and in case (ii) it holds that " 2 L(G).

5.2 Theorem : Every context-free grammar can be transformed into an equivalent "-free gram-
mar.

Proof : Let G = (N, T, P, S) be context-free. G may contain "-productions, otherwise we have

nothing to do. A symbol A 2 N is called erasable if A `⇤G " holds.

Step 1 First we compute all erasable A 2 N . For this purpose we inductively compute the sets
N1 , N2 , N3 , . . . of erasable non-terminal symbols:

N1 = {A 2 N | A ! " 2 P }
Nk+1 = Nk [ {A 2 N | A ! B1 . . . Bn 2 P mit B1 , . . . , Bn 2 Nk }

These sets represent an ascending sequence bounded from above:

N1 ✓ N2 ✓ N3 ✓ . . . ✓ N.

Because N is finite, there is a minimal k0 with Nk0 = Nk0 +1 .

Statement: A 2 Nk0 () A is erasable.

“)” is obvious from the definition of Nk0 .
“(” is shown by induction on the depth of the derivation tree from A to ".

Step 2 We construct a grammar G0 = (N 0 , T, P 0 , S 0 ) equivalent to G. Let S 0 be a new start

symbol N 0 = {S 0 } [ N . The set of productions P 0 is defined in two steps. First we introduce
the set P0 generated from P by replacing every production

A! 1... n 2P

with 1, . . . , n 2 N [ T and n 1 by the production of the form

A ! ↵1 . . . ↵n ,

where the following holds:

 61

• If i 2 N is erasable, then ↵i = " or ↵i = i.

• If i 2 T or i 2 N are non-erasable, then ↵i = i.

• Not all ↵i ’s are ".

Then it holds that: P0 contains no "-productions and P {A ! " | A 2 N } ✓ P0 . Then we get

P 0 from P0 as follows:

P 0 = {S 0 ! " | S is erasable } [ {S 0 ! u | S ! u 2 P0 } [ P0

Thus G0 is defined completely. It remains to show that: L(G0 ) = L(G).

“✓”: This inclusion is obvious, because S 0 ! u 2 P 0 implies S `⇤G u, and A ! u 2 P 0 with

A 2 N implies A `⇤G u.

“◆”: If " 2 L(G) holds, then S is erasable and thus S 0 ! " 2 P 0 . Therefore, " 2 L(G0 ) holds as
well. Now let w 2 L(G) {"}. Consider a derivation tree from S to w in G:

S
% .. e
% . e
% B e
% L e
% L e
% ⌫A L e
% L e
% L e
% L e
% " L e
| {z }
w

We get a derivation tree from S 0 to w in G0 by replacing S by S 0 and by removing all maximal

subtrees which represent the derivations of the form A `⇤G ":

S0
.. @@
. @
B @
L @
L @
L @
| {z }
w

Thus w 2 L(G0 ) holds. ⇤

62 III. Context-free languages and push-down automata

5.3 Definition : A context-free grammar G = (N, T, P, S) is in Chomsky normal form if the

following holds:

• G is "-free (so that at the most S ! " 2 P is allowed),

• every production in P di↵erent from S ! " has the form

A!a or A ! BC,

where A, B, C 2 N and a 2 T .

5.4 Theorem : Every context-free grammar can be transformed into an equivalent grammar in
Chomsky normal form.

Proof : see literature. ⇤

5.5 Definition : A context-free grammar G = (N, T, P, S) is in Greibach normal form if the

following holds:

• G is "-free (so that at the most S ! " 2 P is allowed),

• every production in P di↵erent from S ! " has the form

A ! aB1 . . . Bk ,

where k 0, A, B1 , . . . , Bk 2 N and a 2 T .

5.6 Theorem : Every context-free grammar can be transformed into an equivalent grammar in
Greibach normal form.

Proof : see literature. ⇤

 63

§6 Deterministic context-free languages

In section 3 we have shown that in general every context-free language can be recognized by
a nondeterministic push-down automaton. The question is, whether we can eliminate the non-
determinism as in the case of finite automata and whether we are always able to construct
equivalent deterministic push-down automata. This question is of practical importance for the
construction of a parser for a given context-free language. First we define the notion of deter-
minism for push-down automata and context-free languages.

6.1 Definition :

(i) A push-down automaton K = (⌃, Q, , !, q0 , Z0 , F ) is called deterministic if the transition

relation ! fulfills the following conditions:
8q 2 Q, Z 2 , a 2 ⌃:
a
( number of transitions of the form (q, Z) ! · · ·
"
+ number of transitions of the form (q, Z) ! · · · )  1
(ii) A context-free language L is called deterministic if there is a deterministic push-down
automaton K with L = L(K) (acceptance by final states).
Example : The language L = {an bn | n 2 N} is deterministic context-free, because in section 3
we have introduced a deterministic push-down automaton K with L(K) = L.

Example : The language PALc = {wcwR | w 2 {a, b}⇤ } of palindromes with c as symbol in the
middle is also deterministic context-free. The notation wR means that the string w should be
read backwards.

Theorem 3.5 does not hold for deterministic push-down automata. Thus we cannot replace L(K)
(acceptance by final states) by L" (K) (acceptance by empty stack).

Now we show that not all context-free languages are deterministic. For this purpose we use the
following theorem.

6.2 Theorem : Deterministic context-free languages are closed under complementation.

Proof sketch: We could use the same approach as in the case of finite automata and con-
sider the deterministic PDA K0 = (⌃, Q, , !, q0 , Z0 , Q F ) for a deterministic PDA K =
⇢
(⌃, Q, , !, q0 , Z0 , F ). Unfortunately, it generally holds that L(K0 ) 6= ⌃⇤ L(K). The reason
for the fact that not all strings from the complement of L(K) are accepted is the existence of
non-terminating computations. For example, if a transition
"
(q, A) ! (q, AA)

is used once, then it must be used repeatedly; then infinitely many elements will be added to the
stack and K will not terminate. If this is the case for some input string w 2 ⌃⇤ , then w 62 L(K).
However the same holds for K0 , i.e. w 62 L(K0 ).

Thus we must first transform K into an equivalent deterministic PDA which terminates after
finitely many steps for every input string. Such a construction is actually possible for push-down
64 III. Context-free languages and push-down automata

automata, because the set

⇤ "
{(q, A) | 9 2 with (q, A) ) (q, A )}
"
can be e↵ectively constructed for K and the respective transitions (q, A) ! (q 0 , 0 ) can be deleted
or replaced by appropriate transitions.

Details: think yourself or refer to literature. ⇤

6.3 Corollary : There exist context-free languages that are not deterministic.

Proof : If all context-free languages were deterministic, then the context-free languages would
be closed under complementation. Contradiction to the theorem from section 4.1 ⇤

6.4 Lemma : Deterministic context-free languages are

(i) closed under intersection with regular languages,

(ii) not closed under union, intersection, concatenation and iteration.

Proof : We prove (i) using the same construction for the nondeterministic case (see section
4.1); the PDA K generated from K1 and A2 is deterministic if K1 is deterministic.

(ii) The context-free languages L1 = {am bn cn | m, n 0} and L2 = {am bm cn | m, n 0} are both

deterministic, however, their intersection is not even context-free. Because L1 \ L2 = L1 [ L2
holds, the deterministic context-free languages are not closed under union. For concatenation
and iteration refer to literature. ⇤

As an example for a nondeterministic context-free language, we consider

PAL = {wwR | w 2 {a, b}⇤ ^ w 6= "},

the language of all non-empty palindromes with even length. The notation wR means that the
string w should be read backwards. PAL is obviously context-free: to generate it we need the
following rules
S ! aa | bb | aSa | bSb.

In order to show that PAL is nondeterministic we use an auxiliary operator Min.

 65

6.5 Definition : For a language L ✓ ⌃⇤ let

Min(L) = {w 2 L | there is no strict prefix v of w with v 2 L},

where v is a strict prefix of w if v 6= w and 9u 2 ⌃⇤ : w = v · u.

6.6 Lemma : If L with " 62 L is a deterministic context-free language, then so is Min(L) as

well.

Proof : Consider a deterministic PDA K = (⌃, Q, , !, q0 , Z0 , F ) with L(K) = L. Because

" 62 L, it holds as well q0 62 F . We transform K into PDA K1 that functions as K, but in
every transition sequence it reaches one of the final states from F at most once and then stops
immediately. Let us consider a new state q1 2 / Q and define K1 = (⌃, Q[{q1 }, , !1 , q0 , Z0 , {q1 })
with
!1 = {(q, Z, ↵, q 0 , 0 ) | q 2 Q F and (q, Z, ↵, q 0 , 0 ) 2 !}
[ {(q, Z, ", q1 , Z) | q 2 F and Z 2 }
K1 is deterministic and L(K1 ) = Min(L(K)) holds, because K is deterministic. ⇤

Now we show:

6.7 Theorem : The context-free language PAL is nondeterministic.

Proof : Hypothesis: PAL is deterministic. Then according to both lemmas considered above
the language
L0 = Min(PAL) \ L((ab)+ (ba)+ (ab)+ (ba)+ )

is context-free and deterministic. (ab)+ stands for the regular expression ab(ab)⇤ and (ba)+
similarly for the regular expression ba(ba)⇤ . Because all strings in L0 are palindromes with even
length without strict prefix, it holds that

L0 = {(ab)i (ba)j (ab)j (ba)i | i > j > 0}.

According to the pumping lemma there exists a number n 2 N for L0 with the properties given
in the pumping lemma. Then we can decompose the string

z = (ab)n+1 (ba)n (ab)n (ba)n+1 2 L0

into z = uvwxy. Because the intermediate part vwx fulfills the conditions |vwx|  n and vx 6= ",
we can show that not all strings of the form uv i wxi y with i 2 N can be in L0 . Therefore, L0 is
not even context-free, not to mention deterministic context-free. Contradiction ⇤

Remark: Because L0 is not context-free, the closure properties imply that context-free languages
are not closed under the operator Min.

For the practical syntax analysis of programming languages we will use the deterministic context-
free languages. There are two di↵erent methods of syntax analysis: in the top-down method
66 III. Context-free languages and push-down automata

we construct the derivation trees from the start symbol of the grammar and in the bottom-up
method we reconstruct from the given string. We want to be able to determine unambiguously
the next derivation step by looking k symbols ahead for some k 0.

For the top-down method we can use the so called LL(k)-grammars. These are context-free
grammars G = (N, T, P, S) where for every intermediate string w1 Av in a leftmost derivation
S `⇤G w1 Av `⇤G w1 w2 = w of a string w 2 T ⇤ from S the part Av and the first k symbols of
the remainder w2 of w define unambiguously the next left derivation step w1 Av `G w1 uv. This
LL(k)-condition can be graphically represented as follows:

S
! . aa
top-down
!!! .. aa
!! aa
w1 A v A
A A
A A
@
@ A A
u
A
k symbols AA
w2
| {z }
w=w1 w2 2T ⇤

For the bottom-up method we can use the so called LR(k)-grammars. These are context-free
grammars G = (N, T, P, S) with the following property. For every intermediate string vw2 in
the bottom-up reconstruction of a right derivation S `⇤G vw2 `⇤G w1 w2 = w of a string w 2 T ⇤
from S, the part v and the first k symbols of the rest w2 define unambiguously the previous
right derivation step ṽAw2 `G ṽuw2 with ṽu = v and A ! u 2 P . This LR(k)-condition can be
graphically represented as follows:

S
⌦ . J
⌦
⌦ . J
.
@
@ J
⌦ J
⌦ J
⌦ A J
⌦ A J
⌦
bottom-up
⌦ ṽ u AA k symbols
J
J
⌦ v w2
⌦
⌦
⌦
w1
| {z }
w=w1 w2 2T ⇤

LL(k) grammars were introduced in 1968 by P.M. Stearns and R.E. Stearns and LR(k)
grammars in 1965 by D.E. Knuth. The following relations hold for languages LL(k) and LR(k)
generated by these grammars:
! !
S ⇢ S
• LL(k) languages 6= LR(k) languages = deterministic context-free languages
k 0 k 0
 67

• It even holds that: LR(1) languages = deterministic context-free languages

68 III. Context-free languages and push-down automata

§7 Questions of decidability

The following constructions are algorithmically computable:

• PDA acceptable by final states $ PDA acceptable by empty stack

• PDA $ context-free grammar

• context-free grammar 7! "-free grammar

• context-free grammar 7! Chomsky or Greibach normal form

The questions of decidability concerning context-free languages can be answered using both
representation by context-free grammars (in normal form) or by push-down automata. We
consider the same problems for regular languages (compare with chapter II, section 5).

7.1 Theorem (decidability): For context-free languages

• the membership problem,

• the emptiness problem,

• the finiteness problem

are decidable.

Proof :
Membership problem: Consider a context-free grammar G = (N, T, P, S) in Greibach normal
form and a string w 2 T ⇤ . The question is: Does w 2 L(G) hold? The case w = " can be decided
immediately, because G is "-free: " 2 L(G) () S ! " 2 P . Now let w 6= ". Then the following
holds:

w 2 L(G) , 9n 1 : S `G . . . `G w
| {z }
n-times
, { In Greibach normal form every derivation step
produces exactly one character of w. }

S `G . . . ` G w
| {z }
|w|-times

In order to decide w 2 L(G) it is enough to check all derivation sequences with the length |w|
in G. This implies the decidability of the decision problem.

Emptiness problem: Consider a context-free grammar G = (N, T, P, S). The question is:
Does L(G) = ; hold? Let n be the number which belongs to the context-free language L(G)
 69

according to the pumping lemma. The same way as in the case of regular languages we show
that:
L(G) = ; () ¬9w 2 L(G) : |w| < n.

Thus we have solved the emptiness problem by solving the decision problem for all strings
w 2 T ⇤ , where |w| < n.

Finiteness problem: Consider a context-free grammar G = (N, T, P, S). The question is: Is
L(G) finite? Let n be the same as above. Then we show the same way as in the case of regular
languages:
L(G) is finite () ¬9w 2 L(G) : n  |w| < 2 · n

Therefore, the finiteness problem can be decided by solving the membership problem a finite
number of times. ⇤

However, unlike the regular languages the following result holds.

7.2 Theorem (undecidability): For context-free languages

• the intersection problem,

• the equivalence problem,

• the inclusion problem

are undecidable.

We can prove this theorem only after formalizing the notion of algorithm.

Another result of undecidability concerns the ambiguity of context-free grammars. For the
practical application of context-free grammars for describing the syntax of programming lan-
guages, it would be beneficial to have an algorithmical test to check the ambiguity of context-free
grammars. However, we will show later that such a test does not exist.

7.3 Theorem : It is undecidable whether a given context-free grammar is ambiguous.

In practice we can easily avoid the problem of having to test the ambiguity by restricting
oneself to LR(1) grammars or its subclasses. The LR(1)-property is algorithmically decidable
and because LR-grammars are always unambiguous (because the last rightmost derivation step
must always be defined unambiguously and thus every string can have only one right derivation)
the problem of ambiguity does not exist.
70 III. Context-free languages and push-down automata
Chapter IV

The notion of algorithm:

What can be computed using
machines?

§1 Turing machines

Turing machines were introduced in 1936 by A.M. Turing (1912–1954). We consider an ele-
mentary model for calculating with pencil and paper. For this purpose we need a tape, where
we can write and change the characters, and a finite program.

Sketch
··· t G T I t t ···
L : means: move one
L - R b b
6 square left on tape.
R: similarly to the right.
q2Q Turing program S : no movement.
finite

1.1 Definition : A Turing machine, shortly TM, is a 6-tuple ⌧ = (Q, ⌃, , , q0 , t) with the
following properties:

(i) Q is a finite non-empty set of states,

(ii) q0 2 Q is the start state,
(iii) is a finite non-empty set, the tape alphabet, with Q \ = ;,
(iv) ⌃ ✓ is the input alphabet,
(v) t2 ⌃ is the blank symbol or blank,

71
72 IV. The notion of algorithm

part
(vi) :Q⇥ ! Q ⇥ ⇥ {R, L, S} is the transition function.
It can be represented as a Turing table or Turing program:
: q1 a1 q10 a01 P1 with qi , qi0 2 Q,
... ... ai , a0i 2 ,
... ... Pi 2 {R, L, S}
... ...
... ...
qn an qn a0n Pn
0

Operating principles of a TM: informal

A configuration of the TM describes the current state, the contents of the tape and the considered
cell.

• Initial configuration:

w
z }| {
··· t t w 0 w 1 ··· w n t ···
6
q0

• Making a step:

··· t w0 w1 · · · wn t ···
6
q0

e.g. (q0 , w0 ) = (q0 , a, R) leads to

··· t a w1 · · · wn t ···
⌥ ⇧
6
q0

• Repetition of this step gives us the following result:

··· t a a ··· a t ···

⌥ ⇧
6
q0
 73

where (q0 , t) is undefined.

Then as the result of the computation we get a string aa . . . a, i.e. the contents of the tape
without blanks.

Example : Let us compute the function

even: { | }⇤ ! {0, 1}
with (
1 if n can be divided by 2
even( |n ) =
0 otherwise

for (n 0). Choose

⌧ = ({q0 , q1 , qe }, { | }, { |, 0, 1, t}, , q0 , t)

using the following Turing table:

: q0 | q1 t R
q0 t qe 1 S
q1 | q0 t R
q1 t qe 0 S

We can easily check that ⌧ computes the function even.

Notion of the configuration

We consider the current state q, the current contents of the tape and the current position of the
writing/reading head. Usually two kinds of notations are used:

(1) We number the squares of the infinite tape sequentially:

··· ···
-1 0 1 2

Contents of the tape: f : Z ! (Z = Set of integers)

Configuration: triple (q, f, i) with i 2 Z
Disadvantage: complicated manipulation

(2) Only a finite part of the tape di↵ers from t. Abstraction of blanks and cells’ numbers.
The configuration

··· t t u1 · · · um v 0 v 1 · · · v n t t ···
6
q
74 IV. The notion of algorithm

u v
z }| { z }| {
can be unambiguously represented as a string u1 . . . um q v0 v1 . . . vn with ui 2 , vj 2
, m, n 0.
Note: Q \ = ;

1.2 Definition : The set K⌧ of the configurations of a ⌧ = (Q, ⌃, , , q0 , t) is given by

⇤ +
K⌧ = ·Q· .

A configuration uqv means that the TM is in state q, the contents of the tape is t1 uvt1 and
the first (leftmost) symbol of the string v is read.

1.3 Definition (Operating principles of a TM ⌧ = (Q, ⌃, , , q0 , t)):

(1) The initial configuration ↵(v) for a string v 2 ⌃⇤ is

(
q0 v if v 6= "
↵(v) =
q0 t otherwise

(2) The transition relation `⌧ ✓ K⌧ ⇥ K⌧ is defined as follows:

K `⌧ K 0 (K 0 is a successor configuration of K)

if 9u, v 2 ⇤ 9a, b 2 9q, q 0 2 Q :

(K = uqav ^ (q, a) = (q 0 , a0 ,S ) ^ K0 = uq 0 a0 v )
_ (K = uqabv ^ (q, a) = (q 0 , a0 ,R) ^ K0 = ua0 q 0 bv)
_ (K = uqa ^ (q, a) = (q 0 , a0 ,R) ^ K0 = ua0 q 0 t )
_ (K = ubqav ^ (q, a) = (q 0 , a0 ,L) ^ K0 = uq 0 ba0 v)
_ (K = qav ^ (q, a) = (q 0 , a0 ,L) ^ K0 = q 0 ta0 v )
By `⌧ ⇤ we denote the reflexive transitive closure of `⌧ , i.e. it holds

K `⌧ ⇤ K 0 if 9K0 , . . . , Kn , n 0 :
K = K 0 ` ⌧ . . . `⌧ K n = K 0

Further on by `⌧ + we will denote the transitive closure of `⌧ , i.e. it holds that

K `⌧ + K 0 if 9K0 , . . . , Kn , n 1 :
K = K 0 ` ⌧ . . . `⌧ K n = K 0

(3) A final configuration is a configuration K 2 K⌧ , which has no successor configurations.

(4) The result (or the visible output) of a configuration uqv is

!(uqv) = uv,

where u is the shortest string with u = t . . . tu and v is the shortest string with v =
vt . . . t. Thus we eliminate q, as well the blanks in the beginning and in the end of the
string; however, the blanks located between characters are not removed.
 75

Remark : Because is a (partial) function, `⌧ is right-unique, i.e. from

K `⌧ K1 ^ K `⌧ K2 it follows that K1 = K2 .

1.4 Definition : The function computed by TM ⌧ = (Q, ⌃, , , q0 , t) is

part
h⌧ : ⌃⇤ ! ⇤

with 8
>
< w if 9 final configuration K 2 K⌧ :
h⌧ (v) = ↵(v) `⌧ ⇤ K ^ w = !(K)
>
:
undef. otherwise
We also write Res⌧ for h⌧ (“result function” of ⌧ ).

Remark : Because `⌧ is right-unique, h⌧ is a partial function.

Illustration of the result function:

h⌧ - w 2
⌃⇤ 3 v ⇤

6
↵ !
?
K⌧ 3 ↵(v) `⌧ ··· `⌧ K 2 K ⌧

1.5 Definition : A set M 2 ⌃⇤ is called a domain of ⌧ if the following holds:

M = {v 2 ⌃⇤ | h⌧ (v) is defined}

A set N 2 ⇤ is called range of values of ⌧ if the following holds:

⇤
N = {w 2 | 9v 2 ⌃⇤ : h⌧ (v) = w}.

1.6 Definition (Turing-computability):

Let A, B be alphabets.

part
(i) A partially defined function h : A⇤ ! B ⇤ is called Turing-computable if there exists a
TM ⌧ = (Q, ⌃, , , q0 , t) with A = ⌃, B ✓ and h = h⌧ , i.e. h(v) = h⌧ (v) for all v 2 A⇤ .
part
(ii) TA,B =def {h : A⇤ ! B ⇤ | h is Turing-computable}

(iii) Let T be the class of all Turing-computable functions (for arbitrary alphabets A, B).
76 IV. The notion of algorithm

1.7 Definition (Turing-decidability):

Let A be an alphabet.

(i) A set L ✓ A⇤ is Turing-decidable if the characteristic function of L

L : A⇤ ! {0, 1}

is Turing-computable. Here L is the following total function:

(
1 if v 2 L
L (v) =
0 otherwise

(ii) A property E : A⇤ ! { true, false } is Turing-decidable if the set {v 2 A⇤ | E(v) = true }

of the strings with the property E is Turing-decidable.

Remarks on computability

(1) Computability of functions with several arguments:

part
k-tuples as input can be described by a function h : (A⇤ )k ! B ⇤ . For the computability
of such functions we simply modify the definition of the initial configuration using separator
#:
↵(v1 , . . . , vk ) = q0 v1 #v2 # . . . #vk

If v1 = ", then q0 observes the first separator. Except for this modification we can use the
previous definition of computability.

(2) Computability of number theoretic functions:

part
f :N !N
We use the bar representation of natural numbers:

b |n = | . . . |
n=
| {z }
n times

Then f is Turing-computable if the function

part
h f : { | }⇤ ! { | } ⇤

with
hf : ( |n ) = |f (n)

is Turing-computable.

Constructing Turing machines:

the flowchart notation

First we define useful elementary TMs. Let = {a0 , . . . , an } be the tape alphabet with a0 = t.
 77

• Small right-machine r:
makes one step to the right and halts. Turing table:
r q 0 a 0 qe a 0 R
... ... ... ... ...
q0 a n qe a n R

• Small left-machine l:
makes one step to the left and the halts. Turing table:
l q0 a 0 qe a0 L
... ... ... ... ...
q0 a n qe an L

• Printer-machine a for a 2 :
prints the symbol a and then halts. Turing table:
a q 0 a 0 qe a S
... ... ... ... ...
q0 a n qe a S

Furthermore we assume that all constructed TMs have exactly one final state, i.e. there exists
one state qe so that for all final configurations uqv it holds that:

q = qe .

It is obvious that TMs r, l, a fulfill this condition. Such TMs can be combined as follows:
a
⌧1 ! ⌧2 means that first ⌧1 works. If ⌧1 terminates on a cell with symbol a, ⌧2 starts.
⌧1 ! ⌧2 means that first ⌧1 works. If ⌧1 terminates, ⌧2 starts.
⌧1 ⌧2 is an abbreviation for ⌧1 ! ⌧2 .
6=a
⌧1 ! ⌧2 means that first ⌧1 works. If ⌧1 terminates on a cell with a symbol di↵erent
from a, ⌧2 starts.

On the basis of the given TMs we can construct flowcharts. The nodes of this flowchart are
a 6=a
labelled with the names of TMs. The edges are labelled with arrows of the form !, ! or !.
Loops are allowed. One TM ⌧ in the flowchart is labelled with an arrow ! ⌧ as a start-TM.

Illustration:

⇠
?
⌧0
a⌦ Jb
⇡
⌦ ^
J
⌧0 ⌧1 ⌧2

A flowchart describes a “large” TM. We can get its Turing table as follows:

Step 1: For every occurrence of a TM ⌧i in the flowchart construct the respective Turing table.
78 IV. The notion of algorithm

Step 2: Make the states in di↵erent tables disjoint.

Step 3: Generate the table of TM by writing all tables (in any order) one below each other.
a
Step 4: Combining: For each ⌧1 ! ⌧2 in the flowchart add to the table of TM line

qe⌧1 aq0⌧2 aS.

Let qe⌧1 be the final state (renamed according to Step 2) of ⌧1 and q0⌧2 the start state
6=a
(renamed according to Step 2) of ⌧2 . Similarly for ⌧1 ! ⌧2 and ⌧1 ! ⌧2 .

Example :

• Large right-machine R:
first makes a step to the right. Then R moves to the right until a blank a0 = t is observed.

R ?
r
6= t

Construction of the Turing table of R:

R q0 a 0 qe a 0 R
... ... ... ... ... r
q0 a n qe a n R

qe a 1 q0 a 1 S ⇤
⇥
... ... ... ... ... 6= a0
qe a n q0 a n S

• Large left-machine L: similarly to the large right-machine.

L ?
l
6= t

Application: Consider a TM for the computation of the “minus” function:

f :N⇥N !N

with (
. x y for x y
f (x, y) = x y =
0 otherwise

The initial configuration of the TM is:

tq0 k . . . k # k . . . k t
| {z } | {z }
x-times y-times

Idea: erase x-dashes as long as y-dashes are there. Then erase the remaining y-dashes and #.

Construction of the TM using a flowchart:

 79
' $

? (y > 0) (x > 0)
- Rl - tLr -t

# (y = 0) # (x = 0)
? ?
-
t tr (y > 0)
t
⇢
6 ⇡

As an exercise we recommend to construct the whole Turing table.

Variations of Turing machines

There exist many variations of the TM definition. These variations are often more convenient
if we want to prove that a certain function is computable. We can show that these variations
do not increase the power of a TM, which was defined in Definition 1.1. First we consider TM
with several tapes.

1.8 Definition (k-tape Turing machine): A k-tape TM ⌧ = (Q, ⌃, , , k, q0 , t) is a 7-tuple

with the following properties:

(i)-(v) Q, ⌃, , q0 , t as in Definition 1.1.

(vi) k 2 N is the number of tapes
(vii) The transition function is now given as follows:

k part k
:Q⇥ ! Q⇥ ⇥ {R, L, S}k

Illustration of for k = 3:

··· c d a ··· ··· c d d ···

⌧ ⌧
6 6

··· a c a ··· `⌧ ··· a d a ···

⇢ ⇠ ⇢ ⇠
6 6

··· c d a ··· ··· a d a ···

6 ' % 6 ' %
⌦ ⌦

q q’

This transition occurs for

(q, (a, c, c)) = (q 0 , (d, d, a), (L, R, S))

80 IV. The notion of algorithm

Configuration of a k-tape TM ⌧ :

K = (u1 qv1 , . . . , uk qvk ) 2 K⌧

with u1 , . . . , uk 2 ⇤, q 2 Q, v1 , . . . , vk 2 +.

Operating principles of a k-tape TM ⌧ :

(1) Initial configuration for a string v 2 ⌃⇤ is

↵k (v) = (↵(v), q0 t, . . . , q0 t),

| {z }
(k 1)-times

i.e. v is written on the first tape.

(2) Transition relation `⌧ ✓ K⌧ ⇥ K⌧ : similarly

(3) Final configuration: no successor configuration (as before)

(4) The result !k of a k-tape configuration is

!k (u1 qv1 , . . . , uk qvk ) = !(u1 qv1 ),

i.e. we take the result of the first tape; we consider the remaining tapes only as auxiliary
tapes for computation.

part
The computed function of a k-tape TM is h⌧ : ⌃⇤ ! ⇤ with
8
>
< w if 9 final config. K 2 K⌧ :
h⌧ (v) = ↵k (v) `⌧ ⇤ K ^ w = !k (K)
>
:
undef. otherwise

Remark: Several tapes can be used for computing functions with several arguments; in this
case we distribute the input strings on the tapes appropriately.

Goal: We want to prove that every function computed by a k-tape TM can be computed by a
normal 1-tape TM.

Idea of the proof: Every k-tape TM can be “simulated” using a constructed 1-tape TM.
Therefore, first we want to specify the notion of simulation.

Notion of simulation

The notion of simulation is essential for proving that di↵erent machines have the same power.

1.9 Definition : We consider (1- or k-tape) TM ⌧ and ⌧ 0 over the same input alphabet ⌃, with
the sets of configurations K⌧ and K⌧ 0 , with transition relations `⌧ , `⌧ 0 , initial configurations
↵, ↵0 and result functions !, ! 0 . Informally, ⌧ is the “higher” TM and ⌧ 0 the “lower” TM.
 81

A simulation of ⌧ by ⌧ 0 is a totally defined function

sim : K⌧ ! K⌧ 0

with the following properties:

(1) 8v 2 ⌃⇤ : ↵0 (v) `⌧ 0 ⇤ sim(↵(v))

(2) 8K1 , K2 2 K⌧ : K1 `⌧ K2 implies sim(K1 ) `⌧ 0 + sim(K2 ),

i.e. every step of ⌧ can be “simulated” by several steps of ⌧ 0 .

(3) 8 final config. K 2 K⌧ 9 final config. K 0 2 K⌧ 0 :

sim(K) `⌧ 0 ⇤ K 0 and !(K) = ! 0 (K 0 ).

Illustration:

' $
h⌧ = h⌧ 0
v - w
6

↵ !

?
↵(v) `⌧ ··· `⌧ K

↵0 sim sim !0

? ? ?
↵0 (v) `⌧ 0 ⇤ sim(↵(v)) `⌧ 0 + ··· `⌧ 0 + sim(K) `⌧ 0 ⇤ K 0

1.10 Theorem (Simulation theorem): Let ⌧ and ⌧ 0 (1- or k-tape) be TMs over the same
input alphabet ⌃. Let there be a simulation sim of ⌧ by ⌧ 0 . Then the functions h⌧ and h⌧ 0
computed by ⌧ and ⌧ 0 , respectively, coincide, i.e.

8v 2 ⌃⇤ : h⌧ (v) = h⌧ 0 (v)

(The equality here means that either both sides are undefined or they have the same value).

Proof : Let v 2 ⌃⇤ .
Case 1: h⌧ (v) is undefined.
Then there is an infinite sequence of configurations

↵(v) = K1 `⌧ K2 `⌧ K3 `⌧ . . .

of ⌧ . Therefore, due to properties (1) and (2) of sim , we have an infinite sequence of configu-
rations
↵0 (v) `⌧ 0 ⇤ sim(K1 ) `⌧ 0 + sim(K2 ) `⌧ 0 + sim(K3 ) `⌧ 0 + . . .
82 IV. The notion of algorithm

of ⌧ 0 . Thus h⌧ 0 (v) is also undefined.

Case 2: h⌧ (v) = w is defined.
Then there is a finite sequence of configurations

↵(v) = K1 `⌧ . . . `⌧ Kn , n 1,

of ⌧ , where Kn is a final configuration and w = !(Kn ) holds. Due to the properties (1) - (3) of
sim , we have a finite sequence of configurations

↵0 (v) `⌧ 0 ⇤ sim(K1 ) `⌧ 0 + . . . `⌧ 0 + sim(Kn ) `⌧ 0 ⇤ Kn0

of ⌧ 0 , where Kn0 is a final configuration with !(Kn ) = ! 0 (Kn0 ). Thus it holds that

h⌧ (v) = w = !(Kn ) = ! 0 (Kn0 ) = h⌧ 0 (v).

Now we can show:

1.11 Theorem : For every k-tape TM ⌧ there is a 1-tape TM ⌧ 0 with h⌧ = h⌧ 0 .

Proof : Let ⌧ = (Q, ⌃, , , k, q0 , t). We construct ⌧ 0 = (Q0 , ⌃0 , 0 , 0 , q 0 , t)

0 so that there is a
simulation sim of ⌧ by ⌧ 0 . Definition of sim:
K = ( u1 qa1 v1 , is sim(K) =
... , simulated u1 q ae1 v1 # . . . #uk aek vk
uk qak vk ) by
2 K⌧ 2 K⌧ 0
That is we mark the symbols a1 , . . . , ak read by k heads using new symbols ae1 , . . . , aek , we write
down the contents of k tapes one after each other on the tape of ⌧ 0 , separated by #, and we set
the state q in such a way that ae1 is observed.

Thus we choose: 0 = [ {ã | a 2 } [ {#}. We sketch the choice of Q0 and 0 according to the
following idea of simulation. A step of ⌧ of the form
K1 = (u1 q1 a1 v1 , `⌧ K2 = (u1 q2 b1 v1 ,
... , ... ,
uk q1 a k v k ) u k q 2 bk v k )
generated by
(q1 , (a1 , . . . , ak )) = (q2 , (b1 , . . . , bk ), (S, . . . , S))
is simulated by ⌧ 0 in 2 phases of steps.

• Reading phase:
sim(K1 ) = u1 q1 ae1 v1 # . . . #uk aek vk
>⌧ 0
u1 [read, q1 ]ae1 v1 # . . . #uk aek vk
>⇤⌧ 0
u1 ae1 v1 # . . . #uk [read, q1 , a1 , . . . , ak ]aek vk
 83

• Changing phase:
Now we will change the configuration according to (q, (a1 , . . . , ak )).

u1 ae1 v1 # . . . #uk [read, q1 , a1 , . . . , ak ]aek vk

>⌧ 0
u1 ae1 v1 # . . . #uk [change, q2 , b1 , . . . , bk , S, . . . , S]aek vk
>⇤⌧ 0
u1 [change, q2 ]be1 v1 # . . . #uk bek vk
>⌧ 0
sim(K2 ) = u1 q2 be1 v1 # . . . #uk bek vk

Thus a step of ⌧ is simulated by at most as many steps of ⌧ 0 as the double length of all strings
written on the k tapes.

Similarly we deal with other -transitions. However, if in R- or L-steps we must “stick” an

empty field t to one of the k tapes, then we must shift the contents of the neighboring #-parts
on the tape of ⌧ 0 .

Now we consider the initial and final configurations.

• Initial configuration: Let v 2 ⌃⇤ and v 6= ".

Then for ⌧
↵(v) = (q0 v , and sim(↵(v)) = q0 v#t# . . . #t
q0 t,
...
q0 t)
For ⌧ : ↵ (v) = q00 v. Of course we can program ⌧ 0 in such a way that
0 0

↵0 (v) `⌧ 0 ⇤ sim(↵(v))

holds.

• Final configuration: If K 2 K⌧ is a final configuration of the form

K = (u1 qe a1 v1 ,
... ,
uk qe ak vk ),
then we can understand that

sim(K) = u1 qe ae1 v1 # . . . #uk aek vk

holds only in the reading phase. Then we must bring sim(K) to the result form of a 1-tape
TM by erasing all symbols starting from the first #. Thus we can program ⌧ 0 in such a
way that
sim(K) `⌧ 0 ⇤ u1 qe0 a1 v1
holds and K 0 = u1 qe0 a1 v1 is the final configuration of ⌧ 0 with !(K) = ! 0 (K 0 ).
84 IV. The notion of algorithm

Thus we have:
Q0 = Q [ {q00 , qe0 }
[ {[read, q, a1 , . . . , aj ] | q 2 Q, a1 , . . . , aj 2 , 0  j  kf f }
[ {[change, q, b1 , . . . , bj , P1 , . . . , Pj ] | q 2 Q, 0  j  k, b1 , . . . , bj 2 ,
P1 , . . . , Pj 2 {R, L, S} }
[ {. . . further states . . . }
Altogether we have now shown that sim is a simulation of ⌧ by ⌧ 0 . Thus according to the
simulation theorem it follows that h⌧ = h⌧ 0 .

⇤
 85

Other variations of Turing machines

Below we will consider other variations of Turing machines:

• Turing machines with k heads on one tape:

t F R E I B U R G t
6 6✏ 6

Definition of the transition function : as in Definition 1.8 of k-tape TM, but the definition
of a configuration is di↵erent, because k cells on the tape should be marked.

• Turing machines with a two-dimensional computational space divided into cells (“comput-
ing on graph paper”), possibly with several heads:

⇤ ⇤⇤ ⇤⇤⇤ = filled
⇤ ⇤⇤ ⇤⇤⇤ ⇤ ⇤⇤ ⇤⇤⇤ ⇤ ⇤⇤ ⇤⇤⇤ ⇤ ⇤⇤ ⇤⇤⇤
⇤ ⇤⇤ ⇤⇤ ⇤⇤ ⇤⇤ ⇤⇤ ⇤⇤ ⇤⇤ ⇤⇤ ⇤⇤ ⇤⇤ ⇤⇤ ⇤⇤ ⇤⇤ ⇤⇤ ⇤
= blank
⇤ ⇤⇤ ⇤⇤⇤ ⇤ ⇤⇤ ⇤⇤⇤ ⇤ ⇤⇤ ⇤⇤⇤ ⇤ ⇤⇤ ⇤⇤⇤ ⇤ ⇤⇤ ⇤⇤⇤ ⇤ ⇤⇤ ⇤⇤⇤ ⇤ ⇤⇤ ⇤⇤⇤
⇤ ⇤⇤ ⇤⇤ ⇤ 6⇤ ⇤⇤ ⇤⇤ ⇤ 6
6⇤ ⇤⇤ ⇤⇤⇤ ⇤ ⇤⇤ ⇤⇤⇤
' %

• Turing machine with one-sided infinite tape:

···
6
q

A normal TM (with two-sided infinite tape) can be simulated as follows by a TM with

one-sided infinite tape:
2-sided configuration K = am . . . a1 qb1 . . . bn with m 0, n 1
is (for m  n) simulated by the
b1 bm bm+1 bn
1-sided configuration sim(K) = q ... ...
a1 am t t
i.e. pairs of symbols of the 2-sided tape are
the symbols of the 1-sided tape.
86 IV. The notion of algorithm

• Turing machines with final states:

⌧ = (Q, ⌃, , , q0 , t, F ),

where F ✓ Q is the set of final states and all remaining components are defined as before.

We will use TM with final states in order to “accept” languages L ✓ ⌃⇤ .

1.12 Definition :

(1) A configuration K = uqv of ⌧ is called accepting if q 2 F .

(2) Let v 2 ⌃⇤ . The TM ⌧ accepts v if 9 accepting final configuration K : ↵(v) `⌧ ⇤ K.

(3) The language accepted by ⌧ is

L(⌧ ) = {v 2 ⌃⇤ | ⌧ accepts v}.

A language L ✓ ⌃⇤ is called Turing-acceptable if there is a TM ⌧ with final states for which

L = L(⌧ ) holds.

(4) If we speak about sets, then by T we denote the languages accepted by Turing machines.

1.13 Theorem : Let L ✓ ⌃⇤ and L = ⌃⇤ L.

L and L are Turing-acceptable , L is Turing-decidable.

Proof : “(”: Let L be decidable by ⌧ . By adding final state transitions we get an accepting
TM for L and L.
“)”: L is accepted by ⌧1 and L by ⌧2 . Now let us construct ⌧ with 2 tapes in such a way that
a step of ⌧1 on tape 1 and simultaneously a step of ⌧2 on tape 2 is made. If ⌧1 accepts the string,
then ⌧ produces the value 1. Otherwise ⌧2 accepts the string, and ⌧ produces the value 0. Thus
⌧ decides the language L. ⇤

Remark : If only L is Turing-acceptable, then it does not imply that L is also Turing-decidable.
The accepting TM might not terminate for strings from L .

• Non-deterministic Turing machines:

We consider 1-tape TM with final states. The transition function is extended as
follows:
: Q⇥ ! P(Q ⇥ ⇥ {R, L, S}).

A tuple
(q 0 , b, R) 2 (q, a)

means that the TM, in case if the symbol a is read in state q, can do the following:

go to state q 0 , write b, move to the right.

 87

If there is another tuple

(q 00 , c, S) 2 (q, a),
then the TM can do the following instead:

go to state q 00 , write c, stay.

The choice between these two possible steps of the TM is non-deterministic, i.e. the TM
can arbitrarily choose one of the two possible steps. The transition relation `⌧ of a non-
deterministic TM ⌧ is not right-unique any more. Apart from that, the language L(⌧ )
accepted by ⌧ is defined as for a deterministic TM.

1.14 Theorem : Every language accepted by a non-deterministic TM is also acceptable by a

deterministic TM.

Proof : Consider a non-deterministic 1-tape TM ⌧ . Then there exists a maximal number r of

non-deterministic choices which ⌧ has according to in state q and symbol a, i.e.

8q 2 Q 8a 2 : | (q, a)|  r,
9q 2 Q 9a 2 : | (q, a)| = r

By r we denote the degree of non-determinism of ⌧ . For each pair (q, a) we number the possible
choices from 1 to (at most) r consequently according to (q, a). Then we can represent every
finite sequence of non-deterministic choices as strings over the alphabet {1, . . . , r}.

Example: For r = 5, for example, 1 · 3 · 5 means:

- in the 1st step make the 1st choice,

- in the 2nd step make the 3rd choice,

- in the 3rd step make the 5th choice.

Now let v 2 ⌃⇤ . Then ⌧ accepts the string v if there is an accepting computation of ⌧ :

↵(v) `⌧ ⇤ K accepting.

The set of all computations of ⌧ starting in ↵(v) can be represented as a tree with nodes labelled
with configurations:

↵ K"
↵(v) =
⌦
@
@
K↵
1 · · · Kr
⌦
@ @
@ @
K1·1 · · · K1·r Kr·1 · · · Kr·r

↵
..........................

⌦
@
K accepting.
88 IV. The notion of algorithm

⌥⌅
By ⌃` ⇧
we mark the choices which have lead to the accepting configuration K.

Now we “construct” a deterministic TM ⌧ 0 so that ⌧ 0 generates and traverses every computation

tree of ⌧ using breadth-first search:

0. !
1. !
2. -
3. -
···

For this purpose ⌧ 0 needs 3 tapes.

• Tape 1 stores the input string v.

• Tape 2 is used for systematic generation of all strings over {1, . . . , r}. In order to model
the breadth-first search the strings are generated as follows:

(i) according to the length, the shorter first,

(ii) when the length is equal, then we use lexicographic order. Thus we get the following
sequence:
", 1, 2, . . . , r, 1 · 1, 1 · 2, . . . , 1 · r, . . . , r · 1, . . . , r · r, . . .

• Tape 3 is used for simulation of ⌧ by ⌧ 0 . First the input string v is copied on the tape 3,
then a sequence of transitions of ⌧ is simulated according to the string encoding choices
from {1, . . . , r}, which is written on tape 2.

If ⌧ has an accepting computation v, then its string of choices is generated by ⌧ 0 on tape 2, and
then ⌧ 0 accepts v. If there is no accepting computation v of ⌧ , then ⌧ 0 will run infinitely long
and generate all strings over {1, . . . , r} on tape 2.
Thus L(⌧ ) = L(⌧ 0 ) holds. ⇤

Remark : The above simulation of ⌧ by ⌧ 0 has exponential complexity in the number of steps:
in order to find an accepting computation of ⌧ with n steps, ⌧ 0 must generate and traverse a
computation tree with the breadth r and depth n, i.e. with rn nodes.

The variations of Turing machines considered here

• k-tape TM

• several heads

• 2-dimensional TM

• Final states: for acceptance

• non-deterministic
 89

§2 Grammars

In the previous chapter we got acquainted with context-free grammars. They are a special case
of Chomsky-Grammars introduced in 1959 by American linguist Noam Chomsky. There exist
several types of such grammars (Types 0–3). Here we consider the most general Type 0.

2.1 Definition (Grammar):

A (Chomsky-0- or shortly CH0-)grammar is a 4-tuple G = (N, T, P, S) with

(i) N is an alphabet of non-terminal symbols,

(ii) T is an alphabet of terminal symbols with N \ T = ;,

(iii) S 2 N is the start symbol,

(iv) P ✓ (N [ T )⇤ ⇥ (N [ T )⇤ is a finite set of productions or rules, where for (u, v) 2 P u

must contain at least one non-terminal symbol.

Productions (u, v) 2 P are written down using arrow notation u ! v, as in context-free gram-
mars. Every grammar G has a derivation relation `G on (N [ T )⇤ :

x `G y if and only if 9u ! v 2 P 9w1 , w2 2 (N [ T )⇤ :

x = w1 u w2 and y = w1 v w2 .

By `⇤G we again denote the reflexive and transitive closure of `G . We read x `⇤ y as “y can be
derived from x”. It holds

x `⇤ y if 9z0 , . . . , zn 2 ⌃⇤ , n 0 : x = z 0 ` G . . . ` G zn = y

The sequence x = z0 `G . . . `G zn = y is also called a derivation of y from x (or from x to y) of

the length n. Particularly, it holds: x `⇤G x with a derivation of the length 0.

2.2 Definition :

(i) The language generated from G is

L(G) = {w 2 T ⇤ | S `⇤G w},

i.e. we are interested only in strings over the terminal symbols; the non-terminal symbols
are used as auxiliary symbols within derivations.

(ii) L ✓ T ⇤ is called Chomsky-0- (or shortly CH0-) language if there is a Chomsky-0-grammar

G with L = L(G). The class of all CH0-languages is also shortly denoted by CH0.

2.3 Theorem (T ✓ CH0 ):

Every Turing-acceptable language L ✓ T ⇤ is a Chomsky-0-language.
90 IV. The notion of algorithm

Proof : Let L be accepted by a TM ⌧ = (Q, ⌃, , , q0 , t, F ) with F = {qe } so that for all

configurations uqv 2 K⌧ it holds:

uqv is a final configuration () u = " and q = qe .

We can easily see that for every TM we can provide a TM with this property, which accepts the
same language. Now we construct a Chomsky-0-grammar G = (N, T, P, S) with L(G) = L in
4 steps.

Step 1: (Generate double start configurations)

Let us define a partial grammar G1 = (N1 , T, P1 , S) with S, c|, $, q0 , t 2 N1 so that for all w 2 T ⇤

S `⇤G1 wc|↵(w)$

holds and even for all v 2 (N [ T )⇤ :

S `⇤G1 v and v contains q0

) 9w 2 T ⇤ : v = wc|↵(w)$.
The start configuration ↵(w) is defined as usual:
(
q0 w if w 6= "
↵(w) =
q0 t otherwise.
Choose N1 = {S, c|, $, q0 , t, A, B} [ {Ca | a 2 T } and P1 as follows:

P1 = { S ! c|q0 t$,
S ! aAc|Ca $ for all a 2 T,
Ca b ! bCa for all a, b 2 T,
Ca $ ! Ba$ for all a 2 T,
bB ! Bb for all b 2 T,
Ac|B ! c|q0 ,
Ac|B ! aAc|Ca for all a 2 T }
Operating principle of P1 :
S `G1 c|q0 t$ = "c|↵(")$
or
S `⇤G1 aAc|Ca $
S `⇤G1 w Ac|B w$
S `⇤G1 w aAc|Ca w$
S `⇤G1 w aAc|wCa $
S `⇤G1 w aAc|wBa$
S `⇤G1 w a Ac|B wa$
S `⇤G1 w ac|q0 wa$

Step 2: (Simulate transition relation `⌧ )

Let us define G2 = (N2 , T, P2 , S) with N2 = {S, c|, $} [ Q [ so that for all w 2 T ⇤ , v 2 ⇤ it
holds that:
↵(w) `⌧ ⇤ qe v () wc|↵(w)$ `⇤G2 wc|qe v$.
 91

We can not simply choose P2 = `⌧ , because `⌧ is infinite. For example, (q, a) = (q 0 , a0 , S)

implies that for all u, v 2 ⇤ : uqav `⌧ uq 0 a0 v. We can focus on the finite and choose a rule of
the form
qa ! q 0 a0 .

More precisely we define:

P2 = {qa ! q 0 a0 | q, q 0 2 Q and a, a0 2 and (q, a) = (q 0 , a0 , S) }

[ {qab ! a0 q 0 b | q, q 0 2 Q and a, a0 , b 2 and (q, a) = (q 0 , a0 , R) }

[ { qa$ ! a0 q 0 t$ | q, q 0 2 Q and a, a0 2 and (q, a) = (q 0 , a0 , R) }

|{z}
TM is at the right end of the tape

[ {bqa ! q 0 ba0 | q, q 0 2 Q and a, a0 , b 2 and (q, a) = (q 0 , a0 , L) }

[ { c|qa ! c|q 0 ta0 | q, q 0 2 Q and a, a0 2 and (q, a) = (q 0 , a0 , L) }

|{z}
TM is at the left end of the tape

Step 3: (Erase final configuration)

Define G3 = (N3 , T, P3 , S) with N3 = {S, c|, $, qe , t, D} so that for all w 2 T ⇤ , v 2 ⇤ it holds
that:
wc|qe v$ `⇤G3 w.

Choose P3 as follows:
P3 = { c|qe ! D,
Da ! D for all a 2 ,
D$ ! " }
Operating principle:
wc|qe v$ `G3 wDv$ `⇤G3 wD$ `G3 w.

Step 4: ( Compute) G from G1 , G2 , G3 )

Now let us define G = (N, T, P, S) as follows:

N = N1 [ N2 [ N3 ,
P = P1 [˙ P2 [˙ P3 (disjoint union).

Then for all w 2 T ⇤ , v 2 ⇤ it holds that:

↵(w) `⌧ ⇤ qe v () S `⇤G wc|↵(w)$ (rules P1 )

`⇤G wc|qe v$ (rules P2 )
⇤
`G w (rules P3 )
For “(” note that the rules from at most one set of rules P1 , P2 or P3 can be applied to a given
string over N [ T .
92 IV. The notion of algorithm

Therefore we get L(G) = L. ⇤

part
2.4 Corollary : Let the function h : T ⇤ ! T ⇤ be computed by a Turing machine. Then the
graph of h, i.e. the set

L = {w#h(w) | w 2 T ⇤ and h(w) is defined },

is a Chomsky-0-language.

Proof : If h is computed by a TM ⌧ , then there is a 2-tape TM ⌧ 0 which accepts L. The TM

⌧ 0 operates as follows:

(1) ⌧ 0 leaves a given input string of the form w#v unchanged on the 1st tape.

(2) ⌧ 0 copies the part w on the initially empty 2nd tape and then simulates ⌧ on this tape.

(3) If ⌧ terminates, then the result h(w) of the final configuration is compared to the part v of
the 1st tape. If h(w) = v holds, then ⌧ 0 accepts the input w#v on the 1st tape. Otherwise,
⌧ 0 does not accept the input w#v on the 1st tape.

Thus according to the above theorem L is a Chomsky-0-language. ⇤

2.5 Theorem (CH0 ✓ T ):

Every Chomsky-0-language L ✓ T ⇤ is Turing-acceptable.

Proof : L is generated by a Chomsky-0-grammar G = (N, T, P, S): L(G) = L.

We construct a non-deterministic 2-tape TM ⌧ which accepts L. The TM ⌧ operates as follows:

(1) ⌧ leaves a given input string w 2 T ⇤ unchanged on the 1st tape.

(2) On the initially empty 2nd tape, ⌧ iteratively generates strings over N [T according to the
rules from P , starting with the start string S. In every step ⌧ chooses non-deterministically
a substring u from the last generated string and a rule u ! v from P and then replaces u
by v. If the input string w occurs in some step, then w is accepted.

Thus it holds: ⌧ accepts L. ⇤

In addition to general Chomsky-0-grammars G = (N, T, P, S), where for rules p ! q 2 P it

holds that p, q 2 (N [ T )⇤ and p contains at least one non-terminal symbol, there also exist
other classes of grammars.
 93

2.6 Definition : A Chomsky-0-grammar (or shortly CH0-grammar) G = (N, T, P, S) is called

(for " see below)

(i) context-sensitive (Chomsky-1- or shortly CH1-grammar) if and only if

8 p ! q 2 P 9 ↵ 2 N, u, v, w 2 (N [ T )⇤ , w 6= " : p = u↵v ^ q = uwv
(i.e. a non-terminal symbol ↵ 2 N is replaced by the non-empty string w “in context”
u . . . v).

(ii) context-free (Chomsky-2- or shortly CH2-grammar) if and only if

8 p ! q 2 P : p 2 N (q = " is allowed).

(iii) right-linear (Chomsky-3- or shortly CH3-grammar) if and only if

8p ! q 2 P : p 2 N ^ q 2 T⇤ · N [ T⇤

For context-sensitive grammars G = (N, T, P, S) there exists the following special rule, so that
" 2 L(G) is possible: "-production may only be of the form S ! ". Then all other derivations
begin with a new non-terminal symbol S 0 :
S ! S0
S0 ! . . .

2.7 Definition : A language L is called context-sensitive, context-free, right-linear, if there

exists a grammar of corresponding type which generates L.

Let the classes of languages be defined for some alphabet T :

CH0 = Class of languages, generated by Chomsky-0- grammars ,

CS = ” ” ” , ” ” context-sensitive ” ,
CF = ” ” ” , ” ” context-free ” ,
RLIN = ” ” ” , ” ” right-linear ”
By definition it already holds that (however, the inclusion CF ✓ CS can be proved only on the
basis of the later theorems):

2.8 Corollary : RLIN ✓ CF ✓ CS ✓ CH0

Now we want to show that the Chomsky-3- (i.e. right-linear) languages are exactly the finitely
acceptable (i.e. regular) languages from Chapter II.

2.9 Lemma : Every finitely acceptable language is a Chomsky-3-language.

Proof : Consider a DFA A = (⌃, Q, !, q0 , F ). Let us construct the following Chomsky-3-

grammar G:
G = (Q, ⌃, P, q0 ),
where for q, q 0 2 Q and a 2 ⌃ it holds that
a
P = {q ! aq 0 | q ! q 0 }
[ {q ! " | q 2 F }.
94 IV. The notion of algorithm

It is easy to show that L(A) = L(G) holds. ⇤

For the reverse direction we use the following lemma.

Remark : Every Chomsky-3-language can be generated by a grammar G = (N, T, P, S) with

P ✓ N ⇥ (T · N [ {"}).

Proof : Exercise. ⇤

2.10 Lemma : Every Chomsky-3-language is finitely acceptable.

Proof : Consider a Chomsky-3-language L ✓ T ⇤ . According to the remark L can be generated

by a grammar G = (N, T, P, S) with P ✓ N ⇥ (T · N [ {"}): L = L(G). Now we construct
the following "-NFA B:
B = (T, N [ {qe }, !, S, {qe }),

where qe 2
/ N holds and the transition relation ! for , 2 N and a 2 T is defined as follows:
a
• ! if and only if !a 2 P
"
• ! qe if and only if !" 2 P

Again it is easy to show that L(G) = L(B) holds. ⇤

From these two lemmas we get:

2.11 Theorem (Chomsky-3 = DFA): A language is Chomsky-3 (right-linear), if and only if

it is finitely acceptable (regular).
 95

Summary of the chapter

We have proved the following result:

part
2.12 Main theorem : For function f : A⇤ ! B ⇤ the following statements are equivalent

• f is Turing-computable.

• The graph of f is a Chomsky-0-language.

In the literature it is often spoken about “recursion”.

part
2.13 Definition : A function f : A⇤ ! B ⇤ is called recursive if f is Turing-computable or the
graph of f is a Chomsky-0-language, respectively.

A set L ✓ A⇤ is called recursive if L : A⇤ ! {0, 1} is recursive, i.e. if L is Turing-decidable.

Remark : Sometimes when dealing with partially defined recursive functions, we speak about
“partially recursive” functions; by “recursive” we mean only total recursive functions. That is
why we should make sure that we understand what exactly “recursive” means.

Church’s thesis (1936) says:

The functions which are intuitively computable using algorithms are exactly the recursive
functions, i.e. “recursion” is the mathematic formalization of the notion “algorithm”.

Church’s thesis cannot be formally proved, because “intuitively computable” is not a mathe-
matically precise notion, but can only be confirmed by observations.

However, these observations are so grave that Church’s thesis is universally accepted. Particu-
larly it holds that:

- In more than 70 years of experience with computable functions no counterexamples have

been found.

- Every further formalization of the notion of algorithm has proved to be equivalent to

recursion: see below further examples for such formalization.

- Recursive functions have properties, for example, closure under the µ-operator (while
loop), which should also hold for the algorithms.

Therefore, we can use the following informal but universally accepted notions for functions f
and sets L:

• f is “computable” means that f is partially recursive.

• L is “decidable” means that L is recursive.

96 IV. The notion of algorithm

In addition to the already considered formalizations of the notion “computable”, there exist
other equivalent formalizations:

• µ-recursive functions:
According to the works of Gödel, Herbrand, Kleene and Ackermann the set of all
part
computable funcions f : Nn ! N is defined inductively. For this purpose we first of
all introduce very simple basic functions (such as successor function, projection function,
constant function). Then we define how we can get new functions from the functions we
already know (by the principles of composition, primitive recursion and µ-recursion).

• Register machines with k 2 registers:

Every register contains one natural number. The machine can test each of these registers,
whether its contents is equal to or larger than 0. Depending on this test and the current
state the machine can do the following operations with the values of the register

– leave unchanged
– increase by 1
– reduce by 1, if we get no negative number.

Note that 2 registers are already enough for the computation of all computable functions
part
f : N ! N (except for encoding). 2-register machines are also called 2-counter machines
or Minsky-machines.

• -calculus:
This calculus introduced by Church describes the computation using higher-order func-
tions, i.e. functions, which take other functions as parameters. Today the -calculus is
used to describe the semantics of functional languages.

• First-order predicate calculus:

The notion of satisfiability of formulae turns out to be equivalent to the notion of com-
putability.
Chapter V

Non-computable functions —
undecidable problems

§1 Existence of non-computable functions

In this section we consider totally defined functions f : N ! N.

1.1 The question of Gödel, Turing, Church 1936 :

Is every function f : N ! N algorithmically computable?

More precise: Is there a Turing machine which computes f or is the graph of f a Chomsky-0-language,
respectively? In other words: Is f recursive?

The answer is “no”. Below we will show that there exist non-computable functions f : N ! N.
More precisely:

1.2 Theorem (Existence of non-computable functions):

There exist functions f : N ! N which are not Turing-computable.

Idea of the proof : The cardinality of the set of all functions is greater (namely uncountable)
than the cardinality of the set of Turing-computable functions (which is countable).

In order to prove this statement we need to make some preparations. Therefore, we put together
definitions and results from the set theory.

97
98 V. Non-computable functions — undecidable problems

1.3 Definition : Let M and N be sets. Then we define

1. M ⇠ N (M and N have the same cardinality ) if 9 bijection :M !N

2. M N (the cardinality of M is less than or equal to the cardinality of N ) if 9N 0 ✓ N :

M ⇠ N 0 , i.e. M has the same cardinality as N 0 .

3. M N (the cardinality of N is greater than the cardinality of M ) if M N and M 6⇠ N .

4. M is (at most) countable if M N.

5. M is uncountable if N M.

6. M is finite if M N.

7. M is infinite if N M

Remark :

• M N , 9 injection : M ! N.

• Every finite set is countable.

• Every uncountable set is infinite.

1.4 Theorem (Schröder-Bernstein): M N and N M imply M ⇠ N .

Proof : See, for example, P.R. Halmos, Naive Set Theory. ⇤

1.5 Corollary : An infinite set M is countable if and only if N ⇠ M , i.e. it holds that
M = { (0), (1), (2), . . .} for a bijection : N ! M .

Examples : The following sets are countable:

• N,

• {n 2 N | n is even} and

• N ⇥ N (compare with the proof of Lemma 1.6).

However the following sets are over uncountable:

• P(N),

• N ! N ( set of functions from N to N: see Lemma 1.7).

First we show:

1.6 Lemma :
 99

The set of Turing-computable functions f : N ! N is countable.

Proof : Let us assume that we have a countable set {q0 , q1 , q2 , . . .} of states and a countable
set { 0 , 1 , 2 , . . .} of tape symbols, where 0 = t and 1 = |. Let t he sets { 0 , 1 , 2 , . . .} and
{q0 , q1 , q2 , . . .} be disjoint.

Every Turing-computable function f : N ! N can be computed by a Turing machine of the

form

⌧ = (Q, {|}, , , q0 , t) with

Q = {q0 , . . . , qk } for some k 0,
= { 0 , . . . , l } for some l 1

Let T be the set of all those Turing machines.

It is sufficient to show: T is countable, because the set of Turing-computable functions can

not be larger than the set of respective Turing machines.

Let us define for k 0, l 1:

Tk,l = {⌧ 2 T | ⌧ has k + 1 states and l + 1 tape symbols}.

(Remark: we add 1 to k and l, because the states and tape symbols are counted from 0.)

Thus for every ⌧ 2 Tk,l it holds that

⌧ = ({q0 , . . . , qk }, {|}, { 0 , . . . , l }, , q0 , t),

| {z } | {z }
Q

where is one of the finitely many functions

:Q⇥ !Q⇥ ⇥ {R, L, S}.

Thus it holds that: Tk,l is finite.

S
Because T = k 0 Tk,l holds, T is countable due to a bijection : N ! T , which is defined
l 1
according to the following diagonal scheme to arrange sets Tk,l :

l 1 2 3 4
@u
k@ u u u
0

1 u u u

2 u u

3 u

counts the Turing machines in T in the following order:

100 V. Non-computable functions — undecidable problems

1st diagonal: all Turing machines in T0,1 ;

2nd diagonal: all Turing machines in T0,2 , then in T1,1 ;
3rd diagonal: all Turing machines in T0,3 , then in T1,2 ,
then in T2,1 ;
and so on.

Thus we have proved Lemma 1.6. ⇤

Remark : Therefore, the basic idea for the countability of T is the countability of sets of all
pairs {(k, l) | k 0, l 1} according to the diagonal scheme. This scheme goes back to G.
Cantor’s proof for countability of N ⇥ N.

However, it holds:

1.7 Lemma :

The set of all functions f : N ! N is uncountable.

Proof :

Let F = {f | f : N ! N}.
Hypothesis : F is countable, i.e. there is a bijection : N ! F, so that F = {f0 , f1 , f2 , . . .}
with fn = (n) for n 2 N holds. Now let us define g 2 F by g(n) = fn (n) + 1 for all n 2 N.
According to the hypothesis there is an index m 2 N with g = fm . However, then it holds that
fm (m) = g(m) = fm (m) + 1. Contradiction! ⇤

Remark :

Here we use Cantor’s diagonal method , i.e. the values of fx (y) are considered on the “diagonals”
x = y; then the contradiction arises at some point m :
fx (y)
Y 0 1 2 3
X @@u
0 f0 (0)
@
@e u
f1 (1)
1 @
@
@u
f2 (2)
2 @
@u
@ f3 (3)
3
@@

The Lemmas 1.6 and 1.7 imply the theorem about the existence of non-computable functions.

This raises the following question:

Do there exist functions important for computer science which are non-computable?
 101

§2 Concrete undecidable problem: halting for Turing machines

Below we consider the “binary” alphabet B = {0, 1}. Now we consider sets (languages) L ✓ B ⇤
which are undecidable and for which the problem:

Given : w 2 B ⇤
Question : Does w 2 L hold ?

is not algorithmically decidable.

First of all we want to consider the halting problem for Turing machines with the input alphabet
B:

Given : Turing machine ⌧ and string w 2 B ⇤

Question : Does ⌧ applied to w halt, i.e. is h⌧ (w) defined?

The relevance of this problem for computer science: is it decidable whether a given program
terminates for given input values?

First we translate the halting problem into an appropriate language L ✓ B ⇤ . In order to do it

we need a binary encoding of Turing machines. Let us assume that we have countable sets

• {q0 , q1 , q2 , . . .} of states and

• { 0, 1 , 2 , . . .} of tape symbols, now with 0 = 0, 1 = 1, 2 = t.

We consider only Turing machines ⌧ = (Q, B, , , q0 , t) with Q = {q0 , . . . , qk } for k 0,

= { 0 , . . . , l } for l 2.

2.1 Definition : The standard encoding of such a Turing machine ⌧ is the following string w⌧
over N [ {]}:

w⌧ = k]l
.
.
.
] i ] j ] s ] t ] nr(P ) for (qi , j) = (qs , t, P )
.
.
.

where P 2 {R, L, S} and nr(R) = 0, nr(L) = 1, nr(S) = 2 holds and the substrings of the form
. . . of w⌧ are written down in lexicographical order of pairs (i, j).
102 V. Non-computable functions — undecidable problems

Thus the binary encoding of ⌧ is the string bw⌧ 2 B ⇤ generated from w⌧ by making the following
substitution:

]!"
n ! 01n+1 for n 2 N

Example :

Consider the small right-machine r = ({q0 , q1 }, B, B [ {t}, , q0 , t) with the Turing-table:

:
q0 0 q1 0 R
q0 1 q1 1 R
q0 t q1 t R
Then

w⌧ = 1 ] 2
]0 ]0]1]0]0
]0 ]1]1]1]0
]0 ]2]1]2]0

and

bw⌧ = 011 0111

01 01 011 01 01
01 011 011 011 01
01 0111 011 0111 01

Remark :

1. Not every string w 2 B ⇤ is a binary encoding of a Turing machine, for example w = " is
not.

2. It is decidable whether a given string w 2 B ⇤ is a binary encoding of a Turing machine,

i.e. the language BW = {w 2 B ⇤ | 9 T uring machine; ⌧ : w = bw⌧ } is decidable.

3. The binary encoding is injective, i.e. bw⌧1 = bw⌧2 implies ⌧1 = ⌧2 .

2.2 Definition (Special halting problem or self-application problem for Turing ma-
chines): The special halting problem or self-application problem for Turing machines is the
language
K = {bw⌧ 2 B ⇤ | ⌧ applied to bw⌧ halts}.

2.3 Theorem : K ✓ B ⇤ is undecidable.

 103

Proof : Again we use Cantor’s diagonal method in a proof by contradiction.

Hypothesis: K ✓ B ⇤ is decidable. Then there exists a Turing machine ⌧ which computes

⇤
K : B ! {0, 1}. Recall that

(
1 if w 2 K
K (w) =
0 otherwise

We change ⌧ into a Turing machine ⌧ 0 as follows. In a flow chart notation

⌧ 0: @
R
@
⌧
=1
?
r
⇤ OCC
⇤⇤ C

Thus it holds: ⌧ 0 gets into an infinite loop if ⌧ halts with 1, and ⌧ 0 halts (with 0) if ⌧ halts with
0.

Then it holds:

⌧ 0 applied to bw⌧ 0 halts , ⌧ applied to bw⌧ 0 halts with 0

, K (bw⌧ 0 ) = 0
, bw⌧ 0 62 K
, ⌧ 0 applied to bw⌧ 0 does not halt.
Contradiction!

Now we will show a general method, which lets us prove that some problems are undecidable
by using some other problems (e.g. K) which are known to be undecidable. This method is the
so-called reduction.

2.4 Definition (Reduction): Let L1 ✓ ⌃⇤1 and L2 ✓ ⌃⇤2 be languages. Then L1 is reducible to
L2 , shortly L1  L2 , if there is a total computable function f : ⌃⇤1 ! ⌃⇤2 so that for all w 2 ⌃⇤1
it holds that: w 2 L1 , f (w) 2 L2 . We also write: L1  L2 using f .

Idea: L1 is a special case of L2 .

2.5 Lemma (Reduction): Let L1  L2 . Then it holds that:

(i) If L2 is decidable, then L1 is also decidable.

(ii) If L1 is undecidable, then L2 is also undecidable.

104 V. Non-computable functions — undecidable problems

Proof : Because (ii) is the contraposition of (i), it is enough to show (i). Thus let L1  L2
using total computable function f and let L2 be decidable, i.e. L2 is computable. Then the
composition L2 (f ) = f L2 (first apply f , then L2 ) is computable. It holds

L1 =f L2 ,

because for all w 2 ⌃1

( ) ( )
1 if w 2 L1 1 if f (w) 2 L2
L1 (w) = = = L2 (f (w))
0 otherwise 0 otherwise

Thus L1 is computable, i.e. L1 is decidable. ⇤

We apply reduction to the general halting problem for Turing machines.

2.6 Definition (general halting problem for TM): The (general) halting problem for Turing
machines is the language

H = {bw⌧ 00u 2 B ⇤ | ⌧ applied to u halts}.

2.7 Theorem : H ✓ B ⇤ is undecidable.

Proof : According to the reduction lemma, it is enough to show K  H, i.e. K is a special

case of H. Here it is trivial: choose f : B ⇤ ! B ⇤ with f (w) = w00w. Then f is computable and
it holds:
bw⌧ 2 K , f (bw⌧ ) 2 H.

2.8 Definition : The blank tape halting problem for Turing machines is the language

H0 = {bw⌧ 2 B ⇤ | ⌧ applied to the blank tape halts}.

2.9 Theorem : H0 is undecidable.

Proof : We show H  H0 . First we describe for a given Turing machine ⌧ and a string u 2 B ⇤
a Turing machine ⌧u , which operates as follows:

• Applied to the blank tape ⌧u first writes u on the tape. Then ⌧u operates as ⌧ (applied to
u).

• It is not important how ⌧u operates if the tape is not blank at the beginning.

We can construct ⌧u from ⌧ and u. Thus there exists a computable function f : B ⇤ ! B ⇤ with

f (bw⌧ 00u) = bw(⌧u ) .

 105

Then it holds

bw⌧ 00u 2 H
, ⌧ applied to u halts.
, f (bw⌧ 00u) 2 H0 .

Thus H  H0 using f holds. ⇤

We got acquainted with three variants of halting problems, namely K, H and H0 . All three
variants are undecidable.

For K we have shown it directly using a diagonal method , and for H and H0 we have shown it
by reduction: K  H  H0 .

In all three variants we dealt with the question whether there exists a decision method which
decides halting for every given Turing machine.

In all cases the answer is “no”.

Perhaps we can at least construct a decision procedure for a given Turing machine ⌧ which
decides whether ⌧ halts.

2.10 Definition : Consider a Turing machine ⌧ . The halting problem for ⌧ is the language

H⌧ = {w 2 B ⇤ | ⌧ applied to w halts}.

For certain Turing machines H⌧ is decidable, for example, for the small right-machine r it holds
that
Hr = B ⇤

and for the Turing machine ⌧ with :

⌧ ?
r

H⌧ = ;.

However, there also exist Turing machines ⌧ for which H⌧ is undecidable. These are the “pro-
grammable” or universal Turing machines.

2.11 Definition : A Turing machine ⌧uni with the input alphabet B is called universal if for
the function h⌧uni computed by ⌧uni the following holds:

h⌧uni (bw⌧ 00u) = h⌧ (u),

i.e. ⌧uni can simulate every other Turing machine ⌧ applied to input string u 2 B ⇤ .
106 V. Non-computable functions — undecidable problems

Relevance for computer science :

The Turing machine ⌧uni is an interpreter of Turing machines, which itself is written as a Turing
machine:

bw⌧ -
⌧uni - h⌧ (u)
u -

Thus the construction of ⌧uni corresponds to writing an interpreter of a simple programming

language in this language.

2.12 Lemma : Universal Turing machines ⌧uni can be constructed e↵ectively.

Proof : Obviously ⌧uni applied to a string w 2 B ⇤ works as follows :

• ⌧uni determines whether w has the form w = bw⌧ 00u for a Turing machine ⌧ (with input
alphabet B) and a string u 2 B ⇤ .

• If no, then ⌧uni goes into an infinite loop.

• If yes, then ⌧uni simulates the Turing machine ⌧ applied to u.

The last can be described in more detail as follows:

1. ⌧uni changes the tape to

bw⌧ c|q0 u$,
where q0 is the initial state of ⌧ .

2. In general ⌧uni sees on the tape

bw⌧ c|vl qavr $
where vl qavr represents the current configuration of the Turing machine ⌧ . ⌧uni remembers
the pair (q, a) and determines the value (q, a) of the transition function (which is binary
encoded in bw⌧ ) of ⌧ .

3. If (q, a) is defined, then ⌧uni returns to the configuration vl qavr of ⌧ and then makes the
necessary transition. Usually ⌧uni must run back and forth several times, because it can
store only a finite amount of information in its memory Q. As a result we have a new
label of the tape of the form
bw⌧ c|vl 0 q 0 a0 vr 0 $,
and ⌧uni operates further as described in (2).

4. If (q, a) is undefined, i.e. vl qavr is a final configuration of ⌧ , then ⌧uni erases the substrings
bw⌧ c| and $ and halts in a final configuration of the form

vl qe avr .
 107

It is obvious that the results of ⌧uni and ⌧ are equal:

!(vl qavr ) = !(vl qe avr ).

Thus it holds that:

h⌧uni (bw⌧ 00u) = h⌧ (u)
as desired. ⇤

2.13 Theorem : H⌧uni ✓ B ⇤ is undecidable.

Proof : According to the definition of ⌧uni it holds that H  H⌧uni using f = idB ⇤ . For the
Turing machine ⌧uni constructed in the proof it even holds that H = H⌧uni , because ⌧uni gets into
an infinite loop if a given string w 2 B ⇤ is not from H, i.e. does not have the form w = bw⌧ 00u.
⇤

Thus we have shown: K  H = H⌧uni  H0

§3 Recursive enumerability

In this section we deal with a weakening of the notion of decidability (recursion) for languages
L over an alphabet A.

3.1 Definition : A language L ✓ A⇤ is called recursively enumerable, shortly r.e., if L = ; or

there exists a total computable function : N ! A⇤ with

L = (N) = { (0), (1), (2), . . .},

i.e. L is the range of values of .

Let us revise the notion “countable” for comparison. L ✓ A⇤ is countable, if L N or equivalent:

if there is a total function : N ! A⇤ with

L = ; or L = (N) = { (0), (1), (2), . . .}.

Thus the di↵erence is that by “countable” we mean that the function may not necessarily be
computable.

We want to consider the new notion of recursive enumerability in more detail.

3.2 Definition (Semi-decidability):

A language L ✓ A⇤ is called semi-decidable if the partial characteristic function of L

part
L : A⇤ ! {1}

is computable. The partial function L is defined as follows:

8
<1 if v 2 L
L (v) =
:undef. otherwise
108 V. Non-computable functions — undecidable problems

Thus a semi-decision procedure for a set L ✓ A⇤ is a “yes-procedure”, while a decision procedure

is a “yes-no-procedure”.

Remark : For all languages L ✓ A⇤ it holds that:

1. L is semi-decidable , L is Turing-acceptable.

2. L is decidable , L and L = A⇤ L are Turing-acceptable or semi-decidable.

Proof : (1) follows from the definition of “semi-decidable”. (2) follows from the respective
theorem about Turing-acceptability. ⇤

3.3 Lemma : For all languages L ✓ A⇤ it holds that: L is recursively enumerable , L is

semi-decidable.

Proof : “)”: Let L be recursively enumerable using the function : N ! A⇤ . Let f :

part
A⇤ ! {1} be computable by the following algorithm:

• Input: w 2 A⇤

• Apply to n = 0, 1, 2, . . . successively.

• If at some point (n) = w holds, halt with output 1. (Otherwise the algorithm does not
halt.)

f= L holds. Thus L is semi-decidable.

“(” : ( dovetailing )

Let L be semi-decidable by the Turing machine ⌧ . If L 6= ;, then we must provide a total

computable function : N ! A⇤ with (N) = L.

Let w0 be some fixed string from L. is computed by the following algorithm:

• Input : n 2 N

• Determine whether n is a prime number encoding of a pair (w, k) 2 A⇤ ⇥ N. Every such

pair can be uniquely encoded by a multiplication of prime numbers.

Details: think yourself.

• If no, the output is w0 .

• Otherwise, determine whether ⌧ applied to w halts in at most k steps (which means the
value 1, i.e. “w 2 L”, is produced).
If yes, the output is w.
Otherwise, the output is w0 .
 109

We show: (N) = L.
“✓” : The above algorithm produces only strings from L.
“◆” : Let w 2 L. Then there is a number of steps k 2 N, such that ⌧ applied to w halts in
k steps and thus produces 1 (“w 2 L”). Let n be the prime number encoding of (w, k). Then
w = (n) 2 (N) holds. ⇤

3.4 Theorem (Characterizing recursive enumerability): For all languages L ✓ A⇤ the

following statements are equivalent:

1. L is recursively enumerable.

2. L is the range of results of a Turing machine ⌧ , i.e. L = {v 2 A⇤ | 9w 2 ⌃⇤ with h⌧ (w) =

v}.

3. L is semi-decidable.

4. L is the halting range of a Turing machine ⌧ , i.e. L = {v 2 A⇤ | h⌧ (v) exists}.

5. L is Turing-acceptable.

6. L is Chomsky-0.

3.5 Corollary : For all languages L ✓ A⇤ it holds that:

L is decidable (recursive) , L and L = A⇤ L are recursively enumerable.

The following extension of the reduction-lemma holds:

3.6 Lemma : Let L1  L2 . Then it holds : If L2 is recursively enumerable, then L1 is also

recursively enumerable.

Proof : Let L1  L2 using f . Then L1 =f L2 holds (first apply f , then L2 ). ⇤

Now we show that the halting problems for Turing machines are recursively enumerable.

3.7 Theorem : H0 ✓ B ⇤ is recursively enumerable.

Proof : H0 is semi-decidable by the Turing machine ⌧0 , applied to w 2 B ⇤ , which operates as

follows:

• ⌧0 determines whether w is of the form w = bw⌧ for some Turing machine ⌧ .

• If no, then ⌧0 goes into an infinite loop.

• If yes, then ⌧0 lets the Turing machine ⌧ run on the blank tape. If ⌧ halts, then ⌧0 produces
the value 1. Otherwise, ⌧0 runs further indefinitely.

⇤
110 V. Non-computable functions — undecidable problems

Thus we get the following main result about the halting of Turing machines.

3.8 Main theorem (Halting of Turing machines):

1. The halting problems K, H, H0 and H⌧uni are recursively enumerable,

but not decidable.

2. The complementary problems K, H, H0 and H⌧uni are countable, but not recursively enu-
merable.

Proof :

“(1)”: It holds that K  H = H⌧uni  H0 .

“(2)”: If the complementary problems were recursively enumerable, then the halting problems
would be decidable according to “(1) r.e.” and the above corollary. Contradiction! ⇤

§4 Automatic program verification

Can the computer verify programs, i.e. determine whether a program P satisfies the given
specification S ? We consider the following verification problem in more detail:

Given : Program P and specification S

Question : Does P satisfy the specification S ?

We formalize this problem as follows:

^
• Program P = Turing machine ⌧ with input alphabet B = {0, 1}
^
• Specification S = subset S of TB,B , the set of all Turing-computable functions
part
h : B⇤ ! B⇤
^
• P satisfies S = h⌧ 2 S
The answer is given in Rice’s theorem (1953, 1956):

The verification problem is undecidable except for two trivial exceptions:

• S = ;: answer always “no”.

• S = TB,B : answer always “yes”

4.1 Theorem (Rice’s Theorem): Let S be an arbitrary non-trivial subset of TB,B , i.e. it
holds that ; ⇢ S ⇢ TB,B . Then the language

BW (S) = {bw⌧ | h⌧ 2 S} ✓ B ⇤

of the binary encodings of all Turing machines ⌧ , whose computed function h⌧ is in the set of
functions S, is undecidable.
 111

Proof : In order to show the undecidability of BW (S), we reduce the undecidable problem H0
or its complement H0 = B ⇤ H0 , respectively, to BW (S).

Therefore, first we consider an arbitrary function g 2 TB,B and an arbitrary Turing machine ⌧ .
Let ⌧g be a TM computing g. Furthermore, let ⌧ 0 = ⌧ 0 (⌧, g) be the following TM depending on
⌧ and g. When applied to a string v 2 B ⇤ , ⌧ 0 (⌧, g) operates as follows:

1. First v is ignored and ⌧ 0 operates as ⌧ applied to the blank tape.

2. If ⌧ halts, then ⌧ 0 operates as ⌧g applied to v.

Let ⌦ 2 TB,B be the totally undefined function. Then the function ⌧ 0 (⌧, g) computed by h⌧ 0 (⌧,g)
is computable by the following case distinction:
(
g if ⌧ halts on the blank tape
h⌧ 0 (⌧,g) =
⌦ otherwise
For a given g there exists a total computable function fg : B ⇤ ! B ⇤ with

fg (bw⌧ ) = bw⌧ 0 (⌧,g) ,

i.e. fg computes the binary encoding of a Turing machine ⌧ 0 (⌧, g) from a given binary encoding
of a Turing machine ⌧ .

We use this function fg for reduction. We distinguish between two cases.

Case 1 : ⌦ 62 S
Choose some function g 2 S. It is possible, because S =
6 ;.
We show: H0  BW (S) using fg .

bw⌧ 2 H0 , ⌧ halts applied to the blank tape

, h⌧ 0 (⌧,g) = g
, {It holds h⌧ 0 (⌧,g) 2 {g, ⌦}, ⌦ 62 S and g 2 S.}
h⌧ 0 (⌧,g) 2 S
, bw⌧ 0 (⌧,g) 2 BW (S)
, fg (bw⌧ ) 2 BW (S)

Case 2 : ⌦ 2 S
Choose any function g 62 S. It is possible, because S =
6 TB,B .
We show: H 0  BW (S) using fg .

bw⌧ 62 H0 , ⌧ does not halt applied to the blank tape

, h⌧ 0 (⌧,g) = ⌦
, {It holds h⌧ 0 (⌧,g) 2 {g, ⌦}, ⌦ 2 S and g 62 S.}
h⌧ 0 (⌧,g) 2 S
, bw⌧ 0 (⌧,g) 2 BW (S)
, fg (bw⌧ ) 2 BW (S)
112 V. Non-computable functions — undecidable problems

Thus we have proved the Rice’s theorem. ⇤

It is clear that Rice’s theorem proves that it is not worth trying to algorithmically verify the
semantics of Turing machines or programs on the basis of their syntactic form, i.e. their input
/ output behavior.

Nevertheless, automatic program verification (also called :“model checking”, where “model” is
equivalent to “program”) is a very active field of research these days. For example, it is possible
to analyse programs whose behavior can be described by finite automata.

§5 Grammar problems and Post correspondence problem

First we consider two problems for Chomsky-0-grammars.

5.1 Definition (decision problem for Chomsky-0):

The decision problem for Chomsky-0-grammars is:
Given: Chomsky-0-grammar G = (N, T, P, S) and a string w 2 T ⇤
Question: Does w 2 L(G) hold?
5.2 Theorem : The decision problem for Chomsky-0-grammars is undecidable.

Idea of the proof : With appropriate binary encoding of grammar G and string w the halting
problem for Turing machines can be reduced to the following:
H  decision problem.

For the direct proof compare also the proofs of Theorem 1.2 and Lemma 1.6.

5.3 Definition : The derivation problem for Chomsky-0-grammars is as follows:

Given: Chomsky-0-grammar G = (N, T, P, S) and
two strings u, v 2 (N [ T )⇤
Question: Does u `⇤G v hold?
5.4 Theorem : The derivation problem for Chomsky-0-grammars is undecidable.

Proof : Obviously the reduction decision problem  derivation problem holds, because for all
grammars G = (N, T, P, S) and strings w 2 T ⇤ it holds that:

w 2 L(G) , S `⇤G w.

Next we consider some sort of a puzzle game, the Post correspondence problem. It was intro-
duced by E. Post (1946) and is abbreviated as PCP (from “Post Correspondence Problem”).
The point is to construct a string in two di↵erent ways.

For subsequent encoding purposes let us assume that we have a countable infinite set of symbols.

SYM = {a0 , a1 , a2 , . . .}.

 113

We always consider strings over SYM. Let X ✓ SYM be an alphabet.

5.5 Definition : An input/instance of the PCP is a finite sequence Y of pairs of strings, i.e.

Y = ((u1 , v1 ), . . . , (un , vn ))

with n 1 and ui , vi 2 SYM⇤ for i = 1, . . . , n. If ui , vi 2 X ⇤ holds, then Y is called an input of

the PCP over X.

A correspondence or solution of Y is a finite sequence of indices

(i1 , . . . , im )

with m 1 and i1 , . . . , im 2 {1, . . . , n}, such that

u i 1 u i 2 . . . ui m = v i 1 v i 2 . . . v i m

holds. Y is called solvable if there exists a solution of Y.

5.6 Definition :

1. The Post correspondence problem (PCP) is the following problem:

Given: Input Y of PCP
Question: Does Y have a correspondence ?

2. The PCP over X is the following problem:

Given: Input Y of PCP over X
Question: Does Y have a correspondence ?

3. The modified PCP (shortly MPCP) is the following problem:

Given: Input Y = ((u1 , v1 ), . . . , (un , vn )) of PCP,

where ui 6= " for i = 1, . . . , n
Question: Does Y have a correspondence (i1 , . . . , im ) with i1 = 1 ?
Thus the correspondence should begin with the first pair of strings,
such that ui1 ui2 . . . uim = vi1 vi2 . . . vim holds.

Example : First we consider

Y1 = ((10, 00), (1, 101), (011, 11)),

i.e.
u1 = 10 , v1 = 00
u2 = 1 , v2 = 101
u3 = 011 , v3 = 11
Then (2, 3, 1, 3) is a correspondence of Y1 , because it holds that:
114 V. Non-computable functions — undecidable problems

u2 u3 u1 u3 = 101110011 = v2 v3 v1 v3

(2) Next we consider

Y2 = ((00, 10), (1, 0), (101, 0)).

Y2 has no correspondence, because for every pair of strings (u, v) 2 Y2 it holds that: neither u
is the initial string of v, nor v is the initial string of u.

Even simple inputs of PCP can have a high degree of complexity. For example, the shortest
solution for
Y = ((001, 0), (01, 011), (01, 101), (10, 001))
already has 66 indices (compare with Schöning, 4th edition, p. 132). Thus we consider the
question whether the PCP is decidable.

Remark : For a one-element alphabet X, the PCP is decidable over X.

Proof : Let X = {I}. Every string I n over X can be identified with a natural number n. Thus
every input Y of PCP over X can be considered as a sequence of pairs of natural numbers:

Y = ((u1 , v1 ), . . . , (un , vn ))

with n 1 and ui , vi 2 N for i = 1, . . . , n. Y is solvable if and only if there is a set of indices

(i1 , . . . , im ) with m 1 and ij 2 {1, . . . , n} for j = 1, . . . , m, such that
m
X m
X
uij = v ij
j=1 j=1

holds. Therefore, we deal with solvability of a problem for natural numbers.

We show:

Y is solvable , (a) 9j 2 {1, . . . , n} : uj = vj

or
(b) 9k, j 2 {1, . . . , n} :
uk < vk and uj > vj

“)”: Proof by contradiction: If neither (a) nor (b) hold, then all pairs (u, v) 2 Y have the
property u < v or all pairs (u, v) 2 Y have the property u > v, respectively. Thus the strings
composed of u-pairs would be shorter or longer than the strings composed of v-pairs. Thus Y
is not solvable.

“(”: If uj = vj holds, the trivial sequence of indices (j) is the solution of Y . If uk < vk and
ul > vl hold, then the sequence

( k, . . . , k , l, . . . , l )
| {z } | {z }
(ul vl ) times (vk uk ) times
 115

is the solution of Y , then it holds that

uk (ul vl ) + ul (vk uk ) = vk (ul vl ) + vl (vk uk ).

Obviously the properties (a) and (b) are decidable. Thus the PCP is decidable over a one-element
alphabet. ⇤

Now we consider the general case of the PCP. Our goal is the following theorem:

5.7 Theorem : The PCP over X is undecidable for every alphabet X with |X| 2.

We show the theorem by reduction of the derivation problem for Chomsky-0-grammars on the
PCP over {0, 1}. We consider the following three reductions:

Derivation problem  MPCP  PCP  PCP over {0, 1}.

We start with the first reduction.

5.8 Lemma : Derivation problem  M P CP

Proof : We introduce an algorithm which constructs an input YG,u,v for MPCP for a given
Chomsky-0-grammar G = (N, T, P, S) with (N [ T ✓ SYM) and given strings u, v 2 (N [ T )⇤
such that the following holds:

(1) u `⇤G v , YG,u,v has a correspondence starting with the first pair of strings.

We use a symbol ], which does not occur in N [ T . Then YG,u,v consists of the following pairs
of strings:

• The first pair of strings: (], ]u])

• Productions: (p, q) for all p ! q 2 P

• Copy: (a, a) for all a 2 N [ T [ {]}

• Last pair: (]v], ])

The exact sequence of pairs of strings in YG,u,v is irrelevant, except for the first pairs of strings.
Now we show that (1) holds.

“)”: It holds that u `⇤G v. Then there is a derivation from u to v of the form

(2) u = u0 p0 v0 `G u0 q0 v0 = u1 p1 v1
`G u 1 q 1 v 1 = u 2 p2 v 2
...
`G u k q k v k = v

where p0 ! q0 , . . . , pk ! qk 2 P holds. There exists the following correspondence of YG,u,v ,

where we write down both components of pairs of strings one after another in two rows :
116 V. Non-computable functions — undecidable problems

⇠ ⇠ ⇠ ⇠
u0 p0 v0 u1 p1 v1 ...
⇢ ⇡⇢⇡
l⇢ ⇡⇢⇡
# # #
b
⌦
b
⌦
b
⇠ ⇠ ⇠ ⇠
l

u0 q0 v0 u1 q1 v1 ...
⇢ ⇢⇡
⇡ ⇢ ⇢⇡
⇡
# u # # #

⇠ ⇠
... uk pk vk
⇢⇡
⇢ ⇡
# # v #
⌦
⇠ ⇠
... uk qk vk #
⇢⇡
⇢ ⇡
#

The “angle pieces” with u and v represent the first pair of strings and the last pair, respectively.
The substrings ui , vi , ] in round frames are constructed character-by-character by copy-pairs
(a, a) for a 2 N [ T [ {]}. The substrings pi , qi in angled frames are placed by the respective
pair of productions (pi , qi ). The connecting lines mean that both strings belong to the same pair
of strings.

“(” : Now consider a correspondence of YG,u,v which begins with the first pair of strings. The
form of the pairs of strings in YG,u,v implies that the correspondence must be constructed as
shown under “)” with the exception that the substrings u and ui qi vi respectively can repeat
infinitely many times by applying only copy-pairs (a, a) :
⇠ ⇠ ⇠ ⇠ ⇠
ui qi vi ui qi vi ...
b⇢⇡ ⇢ ⇡⇢⇡l⇢ ⇡⇢⇡
# # #
⌦
b
⌦
b
b b
⇠ ⇠ ⇠ ⇠ ⇠ ⇠
h
hhh l
b
b
b
... ui qi vi ui qi vi ui qi vi ...
⇢ ⇢⇡
⇡ ⇢ ⇢⇡
⇡ ⇢ ⇢⇡
⇡
# # #

In order to make the correspondence complete, we must be able to place the final pair. For this
purpose we must insert pairs of productions (pi , qi ) such that the correspondence describes a
derivation (2) from u to v.

Note that “(” holds only because we begin with the first pair of strings in the sense of M P CP .
Without this restriction, for example, the copy-pair (], ]) gives us a trivial correspondence which
does not imply u `⇤G v. ⇤

5.9 Lemma : M P CP  P CP

Proof : We introduce an algorithm which constructs an input YP CP of the P CP for the given
input Y = ((u1 , v1 ), . . . , (un , vn )) of the M P CP , i.e. with ui 6= " for i = 1, . . . , n, such that the
following holds:
 117

(3) Y has a correspondence which starts with the first pair of strings
, YP CP has a correspondence.

Idea : Let us construct YP CP in such a way that every correspondence necessarily starts with
the first pair of strings.

For this purpose we use new symbols and which may not appear in any of the strings of Y .
We define two mappings

⇢ : SYM⇤ ! SYM⇤ ,
: SYM⇤ ! SYM⇤ ,

where for w 2 SYM⇤

• ⇢(w) is generated from w by inserting the symbol to the right of every character of w,

• (w) is generated from w by inserting the symbol to the left of every character of w.

For example it holds that

⇢(GT I) = G T I and (GT I) = G T I.

For the mappings ⇢ and the following holds: for all u, v 2 SYM ⇤

1. ⇢(") = " and (") = "

2. ⇢(uv) = ⇢(u)⇢(v) and (uv) = (u) (v)

3. u = v , ⇢(u) = (v)

The statements 1 and 2 mean that ⇢ and are string homomorphisms, i.e. " and the concate-
nation are preserved.

From Y we construct the following YP CP with pairs of strings which are numbered from 0 to
n + 1:
YP CP = ( ( ⇢(u1 ), (v1 )), 0 pair of strings
(⇢(u1 ), (v1 )) , 1st pair of strings
... , ...
(⇢(un ), (vn )) , n-th pair of strings
( , )) , n + 1-th pair of strings
Now we show that (3) holds:

“)”: Let (1, i1 , . . . , im ) be a correspondence of Y , i.e. u1 ui2 . . . uim = v1 vi2 . . . vim . According to
the statement 3 it holds that :

⇢(u1 ui2 . . . uim ) = (v1 vi2 . . . vim )

118 V. Non-computable functions — undecidable problems

By applying statement 2 several times, we get :

...
⇢(u1 ) ⇢(ui2 ) ⇢(uim )
(v1 ) (vi2 ) ... (vim )
=

By using frames we have put together strings which appear in a pair of strings of YP CP . We see
that:
(1, i2 , . . . , im , n + 1)
is a correspondence of YP CP .

“(”: We show this direction only for the case vi 6= " for i = 1, . . . , n:
Let (i1 , . . . , im ) be some correspondence of YP CP . Then i1 = 0 and im = n + 1 hold, because
only the pairs of strings in the 1st pair of strings ( ⇢(u1 ), (v1 )) start with the same symbol
and only the strings in the (n + 1)-th pair of strings ( , ) end with the same symbol. Let
k 2 {2, . . . , m} be the smallest index with ik = n + 1. Then (i1 , . . . , ik ) is also a correspondence
of YP CP , because appears only as the last symbol in the appropriate correspondence string.
The form of the pairs of the strings implies that:

ij 6= 1 for j = 2, . . . , k 1.

Otherwise, there would be two consequent ’s in the substring ⇢(uij 1 ) ⇢(u1 ) which could not
be reproduced by placing (vi )’s, because vi 6= ".

Thus the correspondence string has the following structure

...
⇢(u1 ) ⇢(ui2 ) ⇢(uik 1 )
(v1 ) (vi2 ) ... (uik 1 )
=

By applying the statements (ii) and (iii) we conclude that

u 1 u i 2 . . . ui k 1
= v 1 v i2 . . . v ik 1 .

Thus (1, i2 , . . . , ik 1) is a correspondence of Y . In the case that there is a vi = ", the reasoning
is more difficult.

5.10 Lemma : P CP  P CP over {0, 1}

 119

Proof : For reduction we use a binary encoding over SYM, i.e. an injective computable function

bw : SYM⇤ ! {0, 1}⇤ ,

with which we have got acquainted while considering binary encoding of Turing machines.

Now consider an input Y = ((u1 , v1 ), . . . , (un , vn )) of the P CP . Then we define

bw(Y ) = ((bw(u1 ), bw(v1 )), . . . , (bw(un ), bw(vn )))

as input of P CP over {0, 1}. Obviously it holds that:

Y is solvable , bw(Y ) is solvable.

From the three lemmas above we get the undecidability of the following problems:

• M P CP ,

• P CP ,

• P CP over {0, 1} and

• P CP over X with |X| 2.

In particular we have proved the above theorem.

120 V. Non-computable functions — undecidable problems

§6 Results on undecidability of context-free languages

Now we can prove the results on undecidability of context-free languages mentioned in Chapter
III. In contrast to the regular languages the following holds:

6.1 Theorem : For context-free (i.e. Chomsky-2-) languages

• the intersection problem,

• the equivalence problem,

• the inclusion problem

are undecidable.

Proof :
Intersection problem: Consider two context-free grammars G1 and G2 . The question is:
Does L(G1 ) \ L(G2 ) = ; hold? We show that the Post correspondence problem can be reduced
to the intersection problem:

P CP  intersection problem

This implies the undecidability of the intersection problem.

Now consider an arbitrary input Y = ((u1 , v1 ), . . . , (un , vn )) of the P CP with ui , vi 2 X ⇤ for

an alphabet X. We provide an algorithm which for every input Y constructs two context-free
grammars G1 and G2 , such that the following holds

Y has a correspondence () L(G1 ) \ L(G2 ) 6= ;. (⇤)

The idea is that G1 generates all strings which can be produced by putting ui ’s one after another,
and G2 generates all strings which can be produced by putting vi ’s one after another. In order
for the necessary relation (⇤) to hold, G1 and G2 must also record the indices i of the placed ui
and vi . For this purpose we use n new symbols a1 , . . . , an 2 / X and choose them as the set of
terminal symbols of G1 and G2 :
T = {a1 , . . . , an } [ X
Then put Gi = ({S}, T, Pi , S), i = 1, 2, where P1 is given by the following productions:

S ! a1 u1 | a1 Su1 | . . . | an un | an Sun

P2 is given by the following productions:

S ! a1 v1 | a1 Sv1 | . . . | an vn | an Svn

Obviously it holds that

L(G1 ) = {aim . . . ai1 ui1 . . . uim | m 1 and i1 , . . . , im 2 {1, . . . , n}}

 121

and
L(G2 ) = {aim . . . ai1 vi1 . . . vim | m 1 and i1 , . . . , im 2 {1, . . . , n}}.

This implies that:

Y has the correspondence (i1 , . . . , im )

, aim . . . ai1 ui1 . . . uim = aim . . . ai1 vi1 . . . vim 2 L(G1 ) \ L(G2 ).

Therefore, (⇤) holds and so does the undecidability of the intersection problem. We have proved
a stronger result: the undecidability of the intersection problem for deterministic context-free
languages. As it is easy to check, the languages L(G1 ) and L(G2 ) are deterministic. ⇤

Equivalence problem: Consider two context-free grammars G1 and G2 . The question is: Does
L(G1 ) = L(G2 ) hold? We show the undecidability of this problem using the following reduction:

Intersection problem for deterministic context-free languages

 equivalence problem.

Consider two deterministic push-down automata K1 and K2 . We show that we can construct
from them two (not necessarily deterministic) context-free grammars G1 and G2 such that the
following holds:
L(K1 ) \ L(K2 ) = ; () L(G1 ) = L(G2 ).

We use the fact that for K2 we can construct the complement push-down automaton K2 with
L(K2 ) = L(K2 ) (compare with Chapter III, section 6). Then from K1 and K2 we can algorith-
mically construct context-free grammars G1 and G2 with

L(G1 ) = L(K1 ) [ L(K2 ) and L(G2 ) = L(K2 ).

Thus it holds that

L(K1 ) \ L(K2 ) = ; , L(K1 ) ✓ L(K2 )

, L(K1 ) ✓ L(K2 )
, L(K1 ) [ L(K2 ) = L(K2 )
, L(G1 ) = L(G2 )

as required.

Inclusion problem: Consider two context-free grammars G1 and G2 . The question is: Does
L(G1 ) ✓ L(G2 ) hold? Obviously the reduction

equivalence problem  inclusion problem

holds. Therefore, the inclusion problem is also undecidable. ⇤

Another result of undecidability concerns the ambiguity of context-free grammars. For the
practical use of context-free grammars for syntax description of programming languages it would
122 V. Non-computable functions — undecidable problems

be beneficial to have an algorithmic ambiguity test. However, we show that such a test does not
exist.

6.2 Theorem : It is undecidable whether a given context-free grammar is ambiguous.

Proof : We show the following reduction

P CP  ambiguity problem. (⇤)

Consider an input Y = ((u1 , v1 ), . . . , (un , vn )) of the P CP . First we construct the context-free

grammar Gi = ({Si }, T, Pi , Si ), i = 1, 2, as in the case of reduction of the P CP to the intersection
problem. Afterwards we construct a context-free grammar G = ({S, S1 , S2 }, T, P, S) from it with

P = {S ! S1 , S ! S2 } [ P1 [ P2 .

Because G1 and G2 are unambiguous, the only possible ambiguity of G, while constructing a
derivation tree of G for a string w 2 T ⇤ , is the use of productions S ! S1 or S ! S2 respectively.
Therefore, we have the following result:

G is ambiguous , L(G1 ) \ L(G2 ) 6= ;

, Y has a correspondence.

Thus we have shown the reduction (⇤), and it implies the undecidability of the ambiguity prob-
lem. ⇤
Chapter VI

Complexity

§1 Computational complexity

So far we have considered the computability of problems, i.e. the question whether the given prob-
lems are algorithmically solvable at all. We have also considered a kind of structural complexity,
i.e. determining which type of machine is necessary for solving problems using algorithms. We
have become acquainted with the following hierarchy:

(1) regular languages $ finite automata

(2) context-free languages $ push-down automata

(3) Chomsky-0-languages $ Turing machines

Now we will consider the efficiency or computational complexity, i.e. the question: how much
computing time and how much space (memory) do we need in order to solve the problem algo-
rithmically. We will study time and space depending on the size of the input. There exist two
working directions:

a) Provide most efficient algorithms for concrete problems:

- important for practical problem solving

- Theoretically interesting is the proof of an upper bound for the problem: for exam-
ple, an existing n3 -algorithm proves that the problem is at most n3 “difficult”.

b) Find a lower bound for a problem so that every algorithm solving this problem has at least
this complexity. The size of the input, n, is a trivial lower bound for the time complexity.

Statements about complexity depend on the machine model. In theory we mostly consider
deterministic or also non-deterministic Turing machines with several tapes. By using several

123
124 VI. Complexity

tapes we get more realistic statements than using 1-tape-TM, because the computation time for
merely moving back and forth on the tape of TM can be avoided.

1.1 Definition : Let f : N ! R be a function and ⌧ be a non-deterministic TM with several

tapes and input alphabet ⌃.

(i) ⌧ has the time complexity f (n) if for every string w 2 ⌃⇤ of length n it holds: ⌧ applied
to the input w terminates for every possible computation in at most f (n) steps.

(ii) ⌧ has the space complexity f (n) if for every string w 2 ⌃⇤ of length n it holds: ⌧ applied
to the input w uses for every possible computation on every tape at most f (n) cells.

This definition can be also applied to deterministic TMs with several tapes. For deterministic
TMs there exists exactly one computation for every input w.

If we deal with TMs, we usually represent problems as languages which should be accepted. We
have already used such representation in Chapter “Non-computable functions and undecidable
problems”. For example, the halting problem for TMs was represented as a language

H = {bw⌧ 00u 2 B ⇤ | ⌧ applied to u halts}.

Now we put together problems, i.e. languages, with the same complexity into the so called
complexity classes.

1.2 Definition : Let f : N ! R.

DTIME(f (n)) = {L | there exists a deterministic TM with several tapes

which has time complexity f (n) and accepts L}
NTIME(f (n)) = {L | there is a non-deterministic TM with several tapes
which has time complexity f (n) and accepts L}
DSPACE(f (n)) = {L | there exists a deterministic TM with several tapes
which has space complexity f (n), and accepts L}
NSPACE(f (n)) = {L | there is a non-deterministic TM with several tapes
which has space complexity f (n) and accepts L}
Example : Consider L = {uu | u 2 {a, b}⇤ }. We want to construct a most efficient TM with
several tapes which accepts L.

Solution idea:
- For a given string w 2 {a, b}⇤ first determine the center of w and then
compare both halves.

- In order to determine the center we use the 2nd tape, i.e. we need a
deterministic 2-tape TM.

Operating phases: Consider w 2 {a, b}⇤ with the length n.

 125

t w t t w t
6 `⇤ 6 ` ⇤

" (1) ' % (2)

⌃
6 ⌅ move the OC
lower head on both
n
half as quickly tapes 2
as the upper steps to
head ( n2
bars the left
on tape 2);
also check
whether n is
even

ppp
w
z }| {
u v u t t ··· t
`⇤ ` ⇤

⌃⌅
6 6
(3) #!(4)
⌃ ⌥⇧
⌅
copy the
v
6 2nd half of KA
move the
v from the upper head
1st tape to the right
to the 2nd end of u
tape and
erase it
on the 1st
tape
u u
`⇤
⌦
6 6
(5) "
⌫⇤⌥ ⇧ ⌃ ⌅
v v
compare 6
u and v
backwards;
accept if
u=v

Let us compute time complexity of 5 phases:

⇣n ⌘ ⇣n ⌘ ⇣n ⌘ ⇣n ⌘
(n + 1) + +1 + +1 + +1 + + 1 = 3n + 5
2 2 2 2
Space complexity: n + 2

Thus it holds that: L 2 DTIME (3n + 5),

L 2 DSPACE (n + 2).
We can proceed non-determinstically as follows:

Part 1: Copy character by character from tape 1 to tape 2. Stop this process non-deterministically
at any time.

Part 2: Return to the beginning on tape 2.

Part 3: Now compare character by character starting from the current position and make sure that
the first tape has the same contents as the second one. If it is the case and both heads
point to the blank field, then accept.
126 VI. Complexity

In the best case this process needs

n ⇣n ⌘ ⇣n ⌘ 3n
+ +1 + +1 = + 2 steps.
2 2 2 2
In the worst case, when the non-deterministic decision is made only at the last symbol of the
input string, we need
n + (n + 1) + 1 = 2n + 2 steps.

Thus it holds that: L 2 NTIME(2n + 2)

In complexity theory we compare the asymptotic behavior of time and space complexity, i.e. the
behavior for n which is “large enough”. Thus we can omit constant factors. For this purpose
we use the O-notation from the number theory.

1.3 Definition : Let g : N ! R. Then

O(g(n)) = {f : N ! R | 9n0 , k 2 N 8n n0 : f (n)  k · g(n)}

i.e. O(g(n)) is the class of all functions f which are bounded by a constant multiplied by g for
sufficiently large values of n.

Example : (n 7! 3n + 4), (n 7! n + 2) 2 O(n). Therefore, we also say: the language L is of

time and space complexity O(n) or L is of linear time and space complexity, respectively.

Because a TM can visit at most one new field on its tapes in each computational step, it follows
that:

DSPACE(f (n))
✓ ✓
DTIME(f (n)) NSPACE(f (n))
✓ ✓
NTIME(f (n))
 127

§2 The classes P and NP

The complexity for algorithms which can be applied in practice should be a polynomial p(n) of
the k-th degree, i.e. of the form

p(n) = ak nk + · · · + a1 n + a0

with ai 2 N for i = 0, 1, . . . , k, k 2 N and ak 6= 0.

2.1 Definition (Cobham, 1964):

S
P = DTIME(p(n))
p polynomial in n
S
NP = NTIME(p(n))
p polynomial in n
S
P SP ACE = DSPACE(p(n))
p polynomial in n
S
N P SP ACE = NSPACE(p(n))
p polynomial in n

2.2 Theorem (Savitch, 1970): For all polynomials p in n it holds that:

N SP ACE(p(n)) = DSP ACE(p2 (n))

and thus

N P SP ACE = P SP ACE.

Furthermore, it holds that

P ✓ N P ✓ P SP ACE, (⇤)

because, as we have already mentioned, a TM can visit at most one new cell in every computa-
tional step.

Open problem in computer science: Are the inclusions in (⇤) strict or does the equality
hold ?

The classes P and NP are very important, because they mark the transition from computability
or decidability questions which are of practical interest, to the questions which are purely of
theoretical interest. This transition can be illustrated by the following diagram:
128 VI. Complexity

all problems resp. languages

computable resp. decidable
exponential time
PSPACE

✏
non-det. polyn. time: NP
det. polyn. time: P NPC
6

Practically solvable, i.e. practically computable or decidable are those problems in the class P,
which can be characterized as follows:

P: Construct the right solution deterministically and in polynomial time.

The polynomials which bound the computation time should have small degree, such as n, n2 or
n3 .

However, practically unsolvable are all problems for which it can be proved that the compu-
tation time grows exponentially with the input size n. Between these two extremes there is a
large class of practically important problems for which at the moment we know only exponential
deterministic algorithms. However, these problems can be solved using non-deterministic algo-
rithms in polynomial time. This is the class NP, which in comparison to P can be characterized
as follows:

NP: Guess a solution proposal non-deterministically and then verify / check

deterministically and in polynomial time whether this proposal is right.

In full generality, these problems are also still practically unsolvable. In practice we make use of
so called heuristics, which strongly restricts the non-deterministic search space of the possible
solution proposals. Using these heuristics we try to approximate an “ideal solution”.

Examples of problems from the class NP

(1) Problem of Hamiltonian path

Given: A finite graph with n vertices.
Question: Does the graph contain a Hamiltonian path, i.e. a path which visits each vertex
exactly once?
Consider, for example, the following graph:
r
e2 ⇤C
⇤C
r ⇤ C e3 r
QQ ⇤ C
e1Q
⇤r QCr
⇤ Q C e4
 129

Then the sequence of vertices e1 -e2 -e3 -e4 is a Hamiltonian path. It is easy to see that
the problem of the Hamiltonian path lies in NP: First let us guess a path and then check
whether each vertex is visited exactly once. Because there are n! paths in the graph, this
method could be used deterministically only with exponential complexity.

(2) Traveling salesman problem

Given: A finite graph with n vertices and length 2 N of every edge, as well as a number
k 2 N.
Question: Is there a round-trip for the traveling salesman with the length  k, or more
formally: Is there a cycle in the graph with the length  k which visits each vertex at
least once?
This problem also lies in NP: First let us guess a cycle and then compute its length. The
traveling salesman problem is of practical importance, for example, for designing telephone
networks or integrated circuits.

(3) Satisfiability problem for Boolean expressions (shortly SAT)

Given: A Boolean expression B, i.e. an expression which consists only of variables
x1 , x2 , . . . , xn connected by operators ¬ (not), ^ (and) and _ (or), as well as by brackets.
Question: Is B satisfiable, i.e. is there an assignment of 0 and 1 to the Boolean variables
x1 , x2 , . . . , xn in B such that B evaluates to 1?
For example, B = (x1 ^ x2 ) _ ¬x3 is satisfiable with the values x1 = x2 = 1 or x3 = 0. In
Section 3 we will consider the SAT problem in more detail. ⇤

While considering the question (which still remains open) whether P = NP holds, a subclass of
NP was found, namely the class NPC of the so called NP-complete problems. It holds that:

If one problem of NPC lies in P, then all problems of NP are already in P, i.e. P =
NP holds.

The class NPC was introduced in 1971 by S.A. Cook. Cook was the first to prove that the
SAT problem, which we have just introduced, is NP-complete. Since 1972 R. Karp has proved
that many other problems are also NP-complete. Today we know more than 1000 examples of
problems from the class NPC.

Below we want to define the notion of NP-completeness. For this purpose we need the notion
of polynomial-time reduction, which was introduced by Karp in 1972 as a technique to prove
NP-completeness. Therefore, we tighten the notion of reduction L1  L2 , which was introduced
in Section 2.

2.3 Definition : Let L1 ✓ ⌃⇤1 and L2 ✓ ⌃⇤2 be languages. Then L1 is called polynomially-time
reducible to L2 , shortly
L1  p L2 ,
130 VI. Complexity

if there is a function f : ⌃⇤1 ! ⌃⇤2 which is total and computable with a polynomial time
complexity, such that for all w 2 ⌃⇤1 it holds that:

w 2 L1 , f (w) 2 L2 .

We also say: L1 p L2 using f .

Intuitively, L1 p L2 indicates that L1 is not more complex than L2 . We can easily recognize
that p is a reflexive and transitive relation on languages, because with two polynomials p1 (n)
and p2 (n), p1 (p2 (n)) is also a polynomial.

2.4 Definition (Cook, 1971): A language L0 is called NP-complete if L0 2 N P holds and

8L 2 N P : L p L0 .

2.5 Lemma (Polynomial-time reduction): Let L1 p L2 . Then it holds that:

(i) If L2 2 P holds, then L1 2 P holds as well.

(ii) If L2 2 N P holds, then L1 2 N P holds as well.

(iii) If L1 is NP-complete and L2 2 N P holds, then L2 is also NP-complete.

Proof : for (i) and (ii): Let L1 p L2 using a function f , which is computed by a Turing
machine ⌧1 . Let the polynomial p1 bound the computing time of ⌧1 . Because L2 2 P (or
L2 2 N P , respectively), there is a (non-deterministic) Turing machine ⌧2 which is bounded by
a polynomial p2 and computes the characteristic function L2 .

Similar to normal reduction, the characteristic function L1 for all w 2 ⌃⇤1 can be computed as
follows:
L1 (w) = L2 (f (w)).

We do it by sequentially connecting the Turing machines ⌧1 and ⌧2 . Now let |w| = n. Then
⌧1 computes the string f (w) in p1 (n) steps. This time bound also limits the length of f (w), i.e.
|f (w)|  p1 (n). Therefore, the computation of L2 (f (w)) is carried out in

p1 (n) + p2 (p1 (n))

steps. Thus also L1 2 P (and L1 2 N P , respectively).

for (iii): Let L 2 N P . Because L1 is NP-complete, L p L1 holds. Moreover L1 p L2 holds.

The transitivity of p implies L p L2 , what we wanted to show. ⇤

2.6 Corollary (Karp): Let L0 be an NP-complete language. Then it holds that: L0 2 P ,

P = NP.

Proof : “)”: Statement (i) of the lemma.

“(”: is obvious. ⇤

Example : We show the following polynomial-time reduction:

Hamiltonian path p traveling salesman.

 131

Consider a graph G with n vertices. As a reduction function we consider the following construc-
tion f : G 7! G0 of a new graph G0 as follows:

• G0 has n vertices as G.

• Every edge of G is the edge of G0 of the length 1.

• Every “non-edge” of G is the edge of G0 of the length 2.

Thus G0 is a complete graph, i.e. every two vertices are connected by one edge.

r rp
C ppp
E.g. G: n=5 G0 : and appropriate bound

⇤1C 1 p p2p
⇤C ⇤
pp
⇤C
k =n+1
r r rp 1 ⇤ C p pr
1

pQ p
f
p p 1Q⇤ 2 Cp p p
⇤ C 7 !
Q Q
p
Q ⇤ C
2 pp pp C 1
Q
⇤r QCr p⇤r pp p p p Qp pCr
⇤Q C ⇤Q
2

The construction f takes place in polynomial time (in order to find the non-edges: consider an
incidence matrix, i.e. O(n2 )). Now we show the reduction property of f , i.e.

G has a Hamiltonian path , G0 has a round-trip of the length  n + 1.

Proof of “)”: We need n 1 edges of the length 1 (i.e. “from G”) to connect n di↵erent
vertices. In order to close this Hamiltonian path, we need another edge of the length  2. All
in all the constructed round-trip has the length  n + 1.

Proof of “(”: The round-trip has at least n edges (to connect n vertices) and at most n + 1
edges (due to the edge length 1).

During the round-trip at least 1 vertex is visited twice (start = end). Therefore, we must
definitely remove one edge, in order to get a path with di↵erent vertices. We check whether it
is enough.

Case 1: After removing one edge, the remaining path has the length n 1.

Thus the vertices which can be visited are all di↵erent. Therefore, we have found a Hamiltonian
path.
(Example for case 1: see above)

Case 2: Otherwise, after removing one edge, the remaining path has the length n.

Then this path has n edges and one vertex is visited twice on a remaining cycle. Then we get a
Hamiltonian path after having removed one more edge.
Example for case 2: n = 3
132 VI. Complexity

G0 :
1
⇠ 1
⇠ ⇠ ⇠
r r r 7 ! r r r
⇢ ⇡
⇢ ⇡
1 1
Round-trip of the length 4 Hamiltonian path
 133

§3 The satisfiability problem for Boolean expressions

3.1 Definition SAT:

(i) A Boolean expression is a term over {0,1} and variables with the operations not, and,
or. In particular: Let V = {x1 , x2 , . . .} be an infinite set of variables. Let B = {0, 1} be
the set of logical values (“false” and “true”).

(1) Every xi 2 V is a Boolean expression.

(2) Every a 2 B is a Boolean expression.
(3) If E is a Boolean expression, then so is ¬E (“not E”).
(4) If E1 , E2 are Boolean expressions, then so are (E1 _ E2 ) and (E1 ^ E2 ).
!
If x 2 V , then we mostly write x instead of ¬x. Priorities:
If x 2 V , then we call x and x literals. ¬ over ^ over _

(ii) Conjunctive norman form:

(1) If y1 , y2 , . . . , yk are literals, then (y1 _ y2 _ . . . _ yk ) is called clause (of the order k,
i.e. k literals/alternatives)
(2) If c1 , c2 , . . . , cr are clauses (of the order  k), then c1 ^ c2 ^ . . . ^ cr is called a
Boolean expression in conjunctive normal form (“CNF”) (of the order  k). If at
least one clause contains k literals, then this expression is called a CNF of the order
k. Similarly, we can define “disjoint normal form”.

(iii) An assignment is a function : V ! {0, 1} which can be extended to Boolean expressions

(see (i),(2)-(4)):
(0) = 0, (1) = 1,
(¬E) = 1 (E),
(E1 ^ E2 ) = Min( (E1 ), (E2 )),
(E1 _ E2 ) = Max( (E1 ), (E2 )).

(iv) A Boolean expression E is called satisfiable if there is an assignment with (E) = 1.

(v) The satisfiability problem is described using the language

GSAT = {E | E is Boolean expression, E is satisfiable}.

In this case the alphabet is V [ {0, 1, ¬, _, ^, ), ( }. It is infinite. Thus we encode V using:

xi 7! xj, where j is the binary representation of the number i,

i.e. x1 7! x1, x2 7! x10, x3 7! x11, x4 7! x100 etc. The elements from B can be easily
removed from E using calculation rules, and we can bring E to CNF. Thus we get:
134 VI. Complexity

SAT := {E | E is a Boolean expression (without elements from B) in CNF;

if E contains m di↵erent variables, then these are
the variables x1 , . . . , xm (in encoded form);
E is satisfiable}
⇢ {x, 0, 1, ¬, ^, _, (, ) }⇤ .

SAT(k) := {E | E 2 SAT and E is of the order k}.

3.2 Theorem (Cook, 1971): SAT is NP-complete.

Proof :

• First of all it is easy to see that SAT lies in NP:

Let us guess an assignment : {x1 , . . . , xm } ! {0, 1} for a Boolean expression E in CNF,
which contains exactly m variables. Then to every variable xi we assign the value (xi )
and calculate (E) according to the standard calculation rules (Def. 3.1(iii)). If |E| = n
held, then E would have less than n variables, i.e. m  n; the guessing of an assignment
is carried out in linear time (create a table, m steps), the substitution in E takes |E| · const
steps as well as the evaluation, i.e. SAT 2 NTIME (c1 n + c2 ) for appropriate constants c1
and c2 . In particular SAT 2 NP (think yourself how we can compute (E) in linear time).

• The difficult part is to show that for every L 2 NP it holds that L  SAT. Thus
consider an arbitrary L 2 NP. Then there is a non-deterministic Turing machine ⌧ =
(Q, X, , , q0 , t, F ) with: Q = set of states; X = input alphabet; = tape alphabet,
which contains X; q0 2 Q initial state; t 2 \ X blank symbol; F ⇢ Q set of final states
(compare with the definition in chapter V, supplemented with the final states, because ⌧
must halt for all inputs; only those strings w 2 X ⇤ belong to L which bring ⌧ into a final
state). ⌧ accepts L in polynomial time, i.e. there exists a polynomial p such that ⌧ halts
for every w 2 X ⇤ and every computation sequence after at most p(n) (with n = |w|) steps,
and w 2 L if and only if there is a computation sequence of ⌧ which transfers the initial
configuration q0 w into a final configuration u1 qu2 with q 2 F .
We assume without loss of generality that ⌧ has only one tape. Now for every w 2 X ⇤
we construct a Boolean expression g(w) 2 ⌃⇤ (with ⌃ = {x, 0, 1, ¬, ^, _, (, ) }) in CNF
such that it holds that w 2 L () g(w) 2 SAT. If we have shown that this function
g : X ⇤ ! ⌃⇤ is a polynomial reduction, then it follows that SAT is NP-complete, because
L from NP was arbitrary.
Construction of g(w) from w: (We follow [Mehlhorn, “Data Structures and Algorithms”
2, Section VI.4]) Let Q = {q1 , . . . , qs }, q1 = initial state, F = {qr , qr+1 , . . . , qs }. Let
= {c1 , . . . , c } with c1 = t = blank symbol.
Without loss of generality let : Q ⇥ ! 2Q⇥ ⇥{ L1, +1, 0 } with (q, c) = ; for q 2 F .
R S
✓ 6 I
@
@
“Left” “Right” “Stay”
We make complete by changing every (q, c) = ; into (q, c) = {(q, c, 0)}. Thus the empty
 135

set ; never appears as an image of . Because the end of the computation was determined
by (q, c) = ;, by the modification we define a non-terminating Turing machine ⌧ 0 , which,
however, exactly reflects the operation of ⌧ : If ⌧ halts in state q, then ⌧ 0 after making
p(|w|) steps will reach a configuration u1 qu2 with exactly this state q and ⌧ 0 will stay in
this configuration indefinitely; the opposite direction also holds. Thus:
w2L () ⌧ applied to w is in a configuration u1 qu2 after p(|w|) steps with q 2 F
() ⌧ 0 runs through a sequence K1 , K2 , . . . , Kp(n) of configurations with

(i) K1 = q1 w initial configuration.

(ii) Ki+1 is a successor configuration of Ki for all i 1.
(iii) n = |w| and Kp(n) contains a final state q 2 F .
By we usually mean the subset of Q ⇥ ⇥ Q ⇥ ⇥ { 1, +1, 0}.
Let = {tuple1 , tuple2 , . . . , tuplem } be numbered from 1 to m.
Let w 2 X ⇤ be given, |w| = n, w = cj1 cj2 . . . cjn . The formula g(w) is constructed by the
following variables:

meaning of the variables:

zt,k 1  t  p(n) zt,k = 1 () ⌧ 0 is in state qk at the time point t
1ks

1  t  p(n) at,i,j = 1 () cj is the content of the cell i at the time

at,i,j
p(n)  i  p(n) point t
1j

st,i 1  t  p(n) st,i = 1 () ⌧ 0 is located at the cell i at the time

p(n)  i  p(n) point t

bt,l 1  t  p(n) bt,l = 1 () the l-th tuple of is used to transfer

1lm from the time point t to the time point
t + 1.

Because ⌧ makes at most p(n) steps, we can always assume that |i|  p(n) and t  p(n).
The Boolean expression g(w) should exactly describe the above sequence of configurations
K1 , K2 , . . . , Kp(n) (or all such sequences of configurations, respectively). This requires
fulfilling the following conditions:

(1) Initial configuration: ⌧ 0 in state q1 , tp(n)+1 wtp(n) n is on the tape, the head is at the
cell 1. Time point t = 1.
(2) Final configuration (provided that w is accepted): ⌧ 0 in state qj with r  j  s at
the time point t = p(n).
(3) Transition condition: ⌧ 0 at every time point 1  t  p(n) is in exactly one state, every
cell from p(n) to +p(n) contains exactly one symbol from , the head is exactly at
one of these cells, and exactly one tuple of is used for the transition.
136 VI. Complexity

(4) Successor configuration: The next configuration follows from the previous configura-
tion from the transition determined by the tuple of described in (3).

Let g(w) = A1 ^ A2 ^ A3 ^ A4 with:

!
tp(n)+1
for(1): A1 = a1, p(n),1 ^ a1, p(n)+1,1 ^ . . . ^ a1,0,1
to the left of w

^ a1,1,j1 ^ a1,2,j2 ^ . . . ^ a1,n,jn (w = cj1 . . . cjn )

!
tp(n) n
^ a1,n+1,1 ^ a1,n+2,1 ^ . . . ^ a1,p(n),1
to the right of w
^ z1,1 ^ s1,1
This formula describes exactly the initial configuration with 2 · p(n) + 3 variables.
for(2): A2 = zp(n),r _ zp(n),r+1 _ . . . _ zp(n),s (|F | variables).
for(3): First we describe an auxiliary expression: For variables x1 , . . . , xk let

exactlyone(x1 , . . . , xk ) := atleastone(x1 , . . . , xk ) ^ atmostone(x1 , . . . , xk )

with
atleastone(x1 , . . . , xk ) := (x1 _ x2 _ . . . _ xk )

and
atmostone(x1 , . . . , xk ) := ¬atleasttwo(x1 , . . . , xk )
W !
=¬ (xi ^ xj )
1i<jk apply
V
= (xi _ xj ) de Morgan’s law!
1i<jk

Thus the expression exactlyone(x1 , . . . , xk ) is in CNF; it has k + 12 · k · (k 1) · 2 = k 2

variables. It will be 1 if exactly one xi has the value 1.
Now put ⇣ ⌘
^
A3 = Astate
3 (t) ^ Aplace
3 (t) ^ Acell next
3 (t) ^ A3 (t)
1tp(n)

with
Astate
3 (t) = exactlyone(zt,1 , . . . , zt,s )

Aplace
3 (t) = exactlyone(st, p(n) , st, p(n)+1 , . . . , st,p(n) )

V
Acell
3 (t) = exactlyone(at,i,1 , . . . , at,i, )
p(n)ip(n)

Anext
3 (t) = exactlyone(bt,1 , . . . , bt,m )
Again, this formula is in CNF.
A3 has p(n) · (s2 + (2 · p(n) + 1)2 + (2 · p(n) + 1)2 · 2 + m2 ) variables.
A3 describes exactly the transition condition (3).
 137

V
for(4): A4 = A4 (t)
1t<p(n)
Now we must consider the tuple of in more detail. For l = 1, 2, . . . , m let the l-th
tuple of be given by

tupell = ( qkl , cjl , qk̃l , cj̃l , dl )

2 2 2 2 2
Q Q {+1, 1, 0}

Put: "
V V
A4 (t) = (st,i _ at,i,j _ at+1,i,j )
p(n)ip(n) 1j

V ⇣
^ (st,i _ bt,l _ zt,kl ) ^ (st,i _ bt,l _ at,i,jl )
1lm
^(st,i _ bt,l _ zt+1,k̃l ) ^ (st,i _ bt,l _ at+1,i,j̃l )
⌘
^(st,i _ bt,l _ st+1,i+dl )

Comment: (st,i _ at,i,j _ at+1,i,j ) is 1

() st,i is 1 (i.e. ⌧ 0 is at the time point t at the cell i)
or st,i = 0 and cj is not at the cell i at the time point t
or st,i = 0 and at,i,j = 1 and at+1,i,j = 1
() st,i = 0 and at,i,j = 1 implies that at+1,i,j = 1
(i.e. the cells which are not considered are not changed by ⌧ 0 !)
In a similar way we read the remaining clauses:
(st,i _ bt,l _ zt,kl ) is 1 ()
if ⌧ 0 was at the time point t at the cell i and the l-th tuple was chosen for
transition, then ⌧ 0 must have been in state qkl (at the time point t).
A4 is in CNF. A4 contains p(n) · (2 · p(n) + 1) · (3 · + 15 · m) variables. Now it is
easy to show: (g(w) is in CNF!)
Statement 1: g is polynomial.
Obvious; the above construction gives us the formula with

(2 · p(n) + 3) + (s r + 1)+p(n)·(s2 + (2p(n) + 1)2 · ( 2 + 1) + m2 )

+p(n)·(2 · p(n) + 1) · (3 + 15m) = p0 (n)

variables (for this polynomial p0 ). It is obvious that the generation of g(w) from w
is proportional to p0 (n), i.e. should be computed on a deterministic 1-tape Turing
machine in at most const · (p0 (n))2 steps, i.e. in polynomial time.
Statement 2: g is a reduction, i.e.

8w 2 X ⇤ it holds: (w 2 L () g(w) 2 SAT ).

This expression follows from the construction, where “(” has to be proved in a more
complex way.
Thus g is a polynomial reduction. ) SAT is NP-complete.
138 VI. Complexity

(End of the proof of theorem 3.2) ⇤

3.3 Theorem : SAT(3) is NP-complete.

Proof : Because SAT 2 NP, it also holds that SAT(3) 2 NP. Let us reduce SAT to SAT(3).
Let us replace every clause (x1 _ x2 _ . . . _ xr ) by

(x1 _ y1 ) ^ (y1 _ x2 _ y2 ) ^ (y2 _ x3 _ y3 ) ^ . . . ^ (yr 1 _ x r _ yr ) ^ y r

with new variables y1 , y2 , . . . , yr . It is easy to see that (x1 _ . . . _ xr ) is satisfiable if and only if
the longer formula (of the order 3) is satisfiable. The process is of polynomial size ) SAT can
be polynomially-time reduced to SAT(3). ⇤

For further NP-complete problems: see literature (rucksack problem, shortest round-trip, Hamil-
tonian paths in graphs, clique problem, coloring problem of graphs, integer programming,
timetable problem, allocation problem, graph embedding problem, etc.).