CS 208: Computing Theory

Assoc. Prof. Dr. Brahim Hnich

Faculty of Computer Sciences
Izmir University of Economics
Computability Theory

Decidability
 Motivation

 Turing machines:
 a general purpose computer;
 Formalizes the notion of an algorithm (Church-Turing
thesis)
 We now turn our attention into the power of
algorithms (i.e. Turing machines) to solve
problems
 Try to understand the limitation of computers
 Is studying decidability useful?

 An example of a decidable problem:

 Is a string a member of a context-free language?

 This problem is at the heart of the problem of

recognizing and compiling programs in a
programming language
 Problem vs Language
 Any arbitrary problem can be expressed into a language. For
example:
 1) To find whether a graph is connected or not is a "problem".
Problems are encoded by a TM ‘M’. A TM represents a language
given by a set L = {<G1>, <G2>, …} where all Gi’s are connected
graphs.
 2) And given a particular graph G1, is this graph connected ? is an
instance of the problem.
 3) Instance of problems are encoded into a String ‘w’. That is, w =
<G1> (encoding of G1 in string)
 4) Now the solution is to run the TM M with the input string w, and find
if w∈L.
 Preliminaries
 We generally use the notation <P> to describe an encoding of a problem P in
some way as input to a Turing machine.
 Acceptance problem: ADFA: Does DFA B accept input string w?
 For convenience we use languages to represent various computational
problems
 So, the acceptance problem can be expressed as a language

ADFA = {<B,w>| B is a DFA that accepts input string w}

 Emptiness problem: EDFA: Is the language accepted by some DFA (B) the
empty language (the empty set).
 Equivalence problem: EQDFA: Do two DFAs (A and B) accept the same
language.
 Preliminaries

 ADFA = {<B,w>| B is a DFA that accepts input

string w}
 The problem of testing whether a DFA B accepts a
string w is the same as testing whether <B,w> is a
member of the language ADFA
 Thus, showing that the language is decidable is the
same as showing that the computational problem is
decidable!
 How to simulate any machine

 Our goal is to make TMs that can simulate other

automata, given as input
 TMs can only take strings as input, so we need a
way to encode automata as strings
 We’ll start with the simplest: DFAs
DFAs encoded using {0,1}
DFAs encoded using {0,1}

σ1 σ2
σ2
q1 q2
σ1
Ecoding DFA example: L = {all strings starts with 1}.

00 = separator
 Simulating a DFA

 We have a way to represent a DFA as a string over {0,1}

 Now, we’ll show how to construct a TM that simulates any given DFA
 Given the encoded DFA as input, along with an encoded input
string for it, i.e. given <DFA,w>, simulate DFA on w.
 Decide whether the given DFA accepts the given string
 We’ll use a 3-tape TM
 First tape holds the DFA being simulated
 Second tape holds the DFA’s input string
 Third tape hold the DFA’s current state qi, encoded as 1i as usual.
 Each simulated move performs one state transition and erases one
encoded input symbol
 Simulation strategy for DFA
 Step 1: handle termination:
 If the second tape is not empty, go to step 2
 If it is empty, the DFA is done; search the list of accepting states (tape 1)
for a match with the final state (tape 3)
 If found, halt and accept; if not, halt and reject
 Step 2: look up move:
 Search tape 1 for the move ((q ,a ),q ) = 1i01j01k that applies now, where 1i
i j k
matches the current state (tape 3) and 1j matches the current input
symbol (tape 2)
 Step 3: execute move:
 Replace the 1i on the tape 3 with 1k
 Write B over the 1j (and any subsequent 0) on tape 2
 Go to step 1
Simulating DFA via a TM
Is a problem Decidable ?
Universal TM
 Pseudocode for the simulation strategy
The pseudocode for this function might look something like this:

bool accepts(DFA D, string w)

{
currState = D.startState;
for (char ch: w)
{
currState = D.transitionTable[currState]
[ch];
}
return D.isAcceptingState(currState);
}

Since this is something that we can do with a computer, it's certainly

something that we can do with a TM.
 Examples of Decidable Languages

 ADFA = {<B,w>| B is a DFA that accepts input

string w}
 Theorem: ADFA is a decidable language
 Proof idea: Design a TM M that decides ADFA
 How?
 On input <B,w>
 Simulate B on input w
 If simulation ends in accepting state, accept, otherwise reject
 Examples of Decidable Languages

 ANFA = {<B,w>| B is a NFA that accepts input

string w}
 Theorem: ANFA is a decidable language
 Proof idea: Design a TM M that decides ANFA
 How?
 On input <B,w>
 Convert NFA B into equivalent DFA C
 Run previous TM on input <C,w>
 If that TM accepts, accept, otherwise reject
 Examples of Decidable Languages

 AREX = {<R,w>| R is a regular expression that

generates string w}
 Theorem: AREX is a decidable language
 Proof idea: Design a TM M that decides AREX
 How?
 On input <R,w>
 Convert R into equivalent NFA C
 Run previous TM on input <C,w>
 If that TM accepts, accept, otherwise reject
 Examples of Decidable Languages

 Test if language of a DFA is empty language

 EDFA is decidable
 T = On input <A> where A is a DFA
 Mark the start state of A
 Repeat until no new state is marked
 Mark any state that has a transition coming into it from any state
that is already marked
 If no final state of A is marked, accept; other wise reject
 Examples of Decidable Languages

 EQDFA is decidable
 To test if two DFAs decide the same language we will
rely on several facts
 DFA’s are closed under intersection, union, and complement
 EDFA is decidable (TM T from previous section EDFA)
 If A and B are equal, then the symmetric difference is empty
L( C ) = ( L(A) ∩ L’(B) ) U ( L’(A) ∩ L(B) ) = empty
 F = a TM with input <A,B> where A and B are DFAs
 Construct DFA C, the symmetric difference of A and B
 Run TM T (the one that decides EDFA) on <C>
 If T accepts, then F accepts, if T rejects, then F rejects
 Examples of Decidable Languages

 ACFG = {<G,w>| G is a CFG that generates string

w}
 Theorem: ACFG is a decidable language
 Proof idea: Design a TM M that decides ACFG
 How?
 (Interested students can read the book for proof detail,
we provide the idea here)
 Examples of Decidable Languages
 Theorem: ACFG is a decidable language
 Possibly infinitely many derivations in G, therefore no exhaustive
check
 However, if G is in Chomsky normal form then for any w ∈ L (G), with
|w| = n, n > 0, exactly 2n−1 steps are required for any derivation of w.
 We can then build a TM S that first converts G into Chomsky Normal
Form and then checks every possible derivation of 2n−1 steps
 S =“On input string <G,w>, where G is a CFG and w is a string:
 1. Convert G to an equivalent grammar in Chomsky Normal Form
 2. List all derivations with 2n−1 steps, for |w| = n, if n > 0, otherwise list all
derivations of 1 step.
 3. If any of these derivations generates w, accept, otherwise reject.”
 Examples of Decidable Languages
 ECFG = {<G> | G is a CFG and L (G) = ∅} is a decidable language.
 Proof ideas: check whether string of terminals are derivable from
start. We proceed by solving a more general problem: whether from a
nonterminal a string of terminals is derivable. Algorithm proceeds
bottom up by marking terminals, then all nonterminals with rules
whose rhs consists only of marked symbols
 R =“On input string <G>, where G is a CFG:
 1. Mark all terminal symbols in G
 2. Repeat until no new nonterminal gets marked:
 3. Mark any nonterminal A s.t. G contains a rule A → u , ..., u and each
1 n
symbol ui has already been marked
 4. If the start nonterminal of G is not marked, accept, otherwise reject.”
 What about EQCFG ?

 Consider the language EQCFG = {<G,H> | G and H are

CFGs and L(G) = L(H)}
 Could we proceed to prove its decidability as we did
with EQDFA?
 NO!
 CFGs are not closed under intersection or
complement, therefore we cannot use symmetric
difference.
 We will see later on, EQCFG is indeed undecidable.
Relationships among classes of
languages

Enumerable
Context-free
Or
Turing
regular Recognizable

decidable
 How to simulate any machine, say a TM

 Encoding TM as a binary string:

TM as a binary string
 The Haling Problem
 One of the most philosophically important
theorems of the theory of computation
 Computers (and computation) are limited in a very
fundamental way
 Common, every-day problem are unsolvable
 Does a program sort an array of integers?
 Both program and specification are precise
 But, automating the verification is undecidable
 No computer program can perform the task of checking the
program against the specification!
 Halting Problem

 Halting problem: Does a Turing machine accept a

string?

 ATM = {<M,w>| M is a Turing machine that accepts

string w}

 Theorem: ATM is undecidable

 Halting Problem

 Before proving that ATM is undecidable, note that

ATM is enumerable
 How? Design a Turing Machine U that
recongnizes ATM
 On input <M,w>
 Simulate M on w
 If M ever enters its accept state, accept, and if M ever
enters its reject state reject
 U is called a universal Turing machine
 Diagonalization

 Diagonalization: a very crucial technique that is

useful to prove undecidability of the halting
problem ATM
 Question: what does it mean to say that two
infinite sets are the same size?
 Answered by Georg Cantor in 1873
 How? Pair them off!
 Correspondence
 Recall a correspondence f: A B is a bijection:
 Injective
 Surjective

 Question: what does it mean to say that two

infinite sets are the same size?

 Answer: A and B are the same size if there is a

correspondence from A to B
 Correspondence

 Question: in a crowded room, how can we tell if

there are more people than chairs, or more chairs
than people?
 Correspondence

 Question: in a crowded room, how can we tell if

there are more people than chairs, or more chairs
than people?

 Answer: Establish a correspondence: ask

everyone to sit down!
 Correspondence

 Claim: The set of Natural numbers has the same

size as the set of even numbers!
 Correspondence

 Claim: The set of Natural numbers has the same

size as the set of even numbers!

 Proof: Establish a correspondence

 Let f(i)=2i

Remark: a proper subset of A can be the same size

as A!!!!
 Countable
 Definition: A set A is countable iff
 Either it is finite, or
 It has the same size as N, the set of natural numbers
 We have just seen that the set of even numbers is
countable
 Claim: The set Z of integers is countable
 Proof: Define f:ZN by first rewriting Z as:
 Z={0,1,−1,2,−2,3,−3,…}
Then we can directly see that we can define a mapping
ϕ:Z→N as follows:
 ϕ(x)={2x−1:x>0, and −2x:x≤0}
 Challenge

 In Heaven, there is a hotel with a countable

number of rooms
 One day, the society of Prophets, Oracles, and AI
researchers hold a convention that books every
room in the hotel
 Then one more guest arrives, angrily demanding
a room
 You are the manager. What to do?
 Challenge

 Then one more guest arrives, angrily demanding

a room
 You are the manager. What to do?
 Answer: Ask the guest in room i to move to room
i+1, and put the new comer in room 1!
 Challenge

 Then a countable number of guests arrive, all

angrily demanding a room
 Now What to do?
 Challenge

 Then a countable number of guests (may be

infinite) arrive, all angrily demanding a room
 Now What to do?
 Answer: Ask the guest in room i to move to room
2i, and put the new comers in odd numbered
rooms !!!!

 (Infinity is such an amazing thing!!)

 Rational Numbers

 Let Q ={m/n | m and n are natural numbers}

 Theorem: Q is countable
 What is the correspondence between N and Q?
 Rational Numbers

1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 ……..

2/1 2/2 2/3 2/4 2/5 2/6 2/7 2/8 ……..

3/1 3/2 3/3 3/4 3/5 3/6 3/7 3/8 ……..

4/1 4/2 4/3 4/4 4/5 4/6 4/7 4/8 ……..

. . . . . . . . .
. . . . . . . . .
 Rational Numbers

1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 ……..

2/1 2/2 2/3 2/4 2/5 2/6 2/7 2/8 ……..

3/1 3/2 3/3 3/4 3/5 3/6 3/7 3/8 ……..

4/1 4/2 4/3 4/4 4/5 4/6 4/7 4/8 ……..

Enumerate numbers along Northeast and Southwest diagonals,

and skip duplicates
 Rational Numbers

1/1 1/2 1/3 1/4 1/5 1/6 1/7 1/8 ……..

2/1 2/2 2/3 2/4 2/5 2/6 2/7 2/8 ……..

3/1 3/2 3/3 3/4 3/5 3/6 3/7 3/8 ……..

4/1 4/2 4/3 4/4 4/5 4/6 4/7 4/8 ……..

Does this mean that every infinite set is countable?

 The Real Numbers

 Theorem: R, the set of reals is uncountable

 Cantor introduced the diagonalization method to

prove this theorem!
 The Real Numbers

 Theorem: R, the set of reals is uncountable

 Proof: By contradiction
 Assume there is a correspondence between N and R
 Write it down
n f(n)
1 3.143……
2 55.555….
3 0.12345…
4 0.50000…
 We show now that there a number x not in the list!
 n f(n)
1 3.143……
Diagonalization 2 55.555….
3 0.12345…
4 0.50000…
Proof: By contradiction
 Pick x between 0 and 1, so non-zero digits follow decimal point
 First fractional digit of f(1) is 1
 Pick first fractional digit of x to be different, say 2
 Second fractional digit of f(2) is 5
 Pick second fractional digit of x to be different, say 6
 And so on ….
 X=0.2687….
 Thus x is not the image of any natural which is a contradiction
 So, R is uncountable!
The real numbers are not countable
Proof by contradiction
Consider the real between 0 and 1
 The set of all Turing Machines is
countable
 Recall if Σ is finite, then Σ* is countable
 We can write them all down
 First all of length 0
 Then all of length 1
 Then all of length 2
 Then all of length 3
 Each Turing Machine (M) has an encoding as
<M> which is a string in Σ*
The set of all infinite binary strings is not
countable
 Characteristic functions of languages

 Consider the following function: F

 Given a finite alphabet Σ
 Given a language L over Σ
 L ⊆ Σ*
 Σ* is countable (thus so is L)
 F(i) = 1 if the ith string of Σ* is in L, and 0
otherwise.
 We call F the characteristic function of L
 The set of languages is not countable
 Given a finite alphabet Σ
 Consider the set of all languages, L, over Σ*
 Each language ‘L’ in L has a characteristic function, F, which is an
infinite sequence of 0’s and 1’s (i.e. an infinite binary sequence)
 E.g. consider L = { x | length of x is even }
 F(ε)=1; F(0)=0; F(1)=0; F(11)=1; F(00)=1; F(01)=1; F(10)=1; …
 Thus, there is a correspondence between languages and infinite
binary sequences.
 We know that the set of infinite binary sequences is not countable, so
the set of languages over a finite alphabet Σ*, can’t be countable
either!
TMs can not decide every language
 There are languages not accepted by a
Turing Machine
 There are countable number of TMs
 A Turing Machine describes a language.
 There are uncountable number of languages.
 Thus some languages must not be describable by
a TM.
 The Halting problem

 Until now every problem we have looked at

closely has been decidable.
 One might ask: “is any problem undecidable?”
 There is at least 1 undecidable problem A TM
 Acceptance by Turing Machine
 “Does an arbitrary TM accept an arbitrary input” is
undecidable
 This is an important result, both philosophically
and computationally!
 ATM is Turing Recognizable!

 While not decidable, ATM is Turing Recognizable.

 This depends upon the fact that there is a
universal TM
 The universal Turing Machine takes <tm,input>
and simulates “tm” on “input”.
 Note if “tm” does not halt on “input” neither does
the universal TM halt on <tm,input>
 RTM, Recognizing a TM

 U = On input <M,w>, where M is a TM and w is a

string
 Simulate M on input w
 If M ever enters its accept state, accept; if M ever
enters its reject state, reject

 Note, if we had a way of determining that M would

not halt on w, we could reject, but we don’t.
 ATM is undecidable

 Proof by contradiction
 Assume that ATM is decidable. By a TM called H
 H(<M,w>) = accept if M accepts w, and reject if M does not
accept w (i.e. M either rejects or loops)
 Then if H decides, we can make another machine
D:
 D(<M>) = accept if H(<M>,<M>) rejects, and
rejects if H(<M>,<M>) accepts
 Conclusion: ATM is undecidable

 Since D(<D>) rejects if D(<D>) accepts we have

reached a contradiction.
 So our original assumption that ATM is decidable
must be incorrect.
 Thus, ATM is must be undecidable
 Co-recognizable Language
 A language is Turing co-recognizable if its complement is Turing
recognizable.
 Recall the complement of a language is the language with all the
strings not recognized by the original language.
 Lemma:
 A language, L, is decidable if and only if it is both Turing recognizable and Co-
Turing recognizable.
 Two things to prove
 If L is decidable then it is both Turing and Co-Turing recognizable. This way is easy
 If L is Turing and Co-Turing recognizable, it is decidable
 If M is Turing and Co-Turing recognizable, it is decidable
 P(w) = run M1(w) and M2(w) in parallel.
 P is a decider

 P(w) = run M1(w) and M2(w) in parallel

 If M1 accepts, then P accepts.
 If M2 accepts then P rejects.
 Every string, w, is either in L (M1 halts and
accepts) or it is not (M2 halts and rejects)
 So one of M1(w) or M2(w) must halt.
 P halts when either M1 or M2 halts, so P must
Halt. So P is a decider that accepts all strings in L
and rejects all strings not in L
 Important implications

 Previous theorem has an important application to

the theory of computation
 It shows that some languages are not decidable!
 Or even Turing machine recognizable
 There are languages that are not enumerable
 The set of Turing machines is countable
 The set of languages is uncountable
Important implications
 Important implications

 Theorem: The halting problem is undecidable

 Next slide
 Theorem: A language is decidable if and only if it
is both Turing-recognizable and co-Turing
recognizable
 In other words, a language is decidable if and
only if it and its compliment are Turing-
recognizable (enumerable)
 A non-enumerable language

 Corollory: if L is not decidable, then either L or its

compliment is not enumerable
????

enumerable
Co-enumerable

decidable
 Conclusions

Decidable languages
Halting problem
diagonalization method
undecidable example