Complexity theory

1
 Contents
 Complexity Classes
 Complexity Measures

2
 Introduction
• What is computation?
• From a logical point of view, computation is the process by which to produce an answer to a
question:

• To ensure that all parts of this process (the question, the computation, and the answer) are
finite, it must be possible to present the question in finite time, to do a computation in finite
time, and to read the answer in finite time.

3
• The goal of complexity theory is to provide mechanisms for classifying combinatorial
problems and measuring the computational resources necessary to solve them.

• Complexity theory provides an explanation of why certain problems have no practical

solutions and provides a way of anticipating difficulties involved in solving problems
of certain types.

• The classification is quantitative and is intended to investigate what resources are

necessary (lower bounds) and what resources are sufficient (upper bounds) to solve
various problems.

4
• Computational complexity theory is a branch of the theory of computation in theoretical computer
science that focuses on classifying computational problems according to their inherent difficulty, and
relating those classes to each other.

• The theory formalizes, by introducing mathematical models of computation to study these problems and
quantifying the amount of resources needed to solve them, such as time and storage. Other complexity
measures are also used, such as the amount of communication (used in communication complexity), the
number of gates in a circuit (used in circuit complexity) and the number of processors (used in parallel
computing).

5
 Complexity Classes
• A complexity class is a set of problems of related complexity.

• One central question in computer science is the basic question:

• What functions are computable by a computer?

• Simpler complexity classes are defined by the following factors:

• The type of computational problem: The most commonly used problems are decision
problems. However, complexity classes can be defined based on function problems, counting
problems, optimization problems, promise problems, etc.

• The model of computation: The most common model of computation is the deterministic
Turing machine, but many complexity classes are based on non-deterministic Turing machines

6
 Complexity measures
• For a precise definition of what it means to solve a problem using a given amount of time
and space, a computational model such as the deterministic Turing machine is used.

• Best-case complexity: This is the complexity of solving the problem for the best input of size n.

• Worst-case complexity: This is the complexity of solving the problem for the worst input of size n.

• Average-case complexity: This is the complexity of solving the problem on an average

7
• Usefulness of a Model of Computation
• Expressiveness

• Generality

• Simplicity

• Compile ability

• Verifiability

8
 Chapter one
• Turing Machine (TM)
• Standard TM
• Construction of TMs

9
• The theory of computing provides computer science with concepts, models, and
formalisms for reasoning about both the resources needed to carry out computations
and the efficiency of the computations that use these resources.

• The basic model of computation for our study is the Turing machine, and for
complexity theory, is the multi tape Turing machine.

• Church's thesis states that every computational device can be simulated by a Turing
machine.

• One of the topics in this chapter is the simulation of RAMs by Turing machines.

10
• Church's Thesis
• Church's thesis states that every "effective computation," or "algorithm," can be
programmed to run on a Turing machine.
• Every "computational device" can be simulated by some Turing machine.

11
 Church-Turing Thesis
Every computer algorithm can be
implemented as a Turing machine

Therefore, C, C++, Prolog, Lisp, Small talk, and Java programs

can be simulated in Turing machines

Definition: a programming language is Turing-complete if it is

equivalent to a Turing machine.
Discussion: Are you sure that such a simple model can simulate
my C++ program for computing the Minimum Spanning Tree ? 12
 Background

• Finite Automata accepts regular languages only

For example, {anbn : n = 0, 1, …} is not regular

•Pushdown Automata accepts context-free languages only

For example, {anbncn : w  *} is not context-free

•We can easily write an algorithm (in C) recognizing if a

sequence of characters have the form anbncn or not
13
 Introduction of Turing machine
Introduced by Alan Turing in 1936.

A simple mathematical model of a computer.

It is capable of performing any calculation which can be performed

by any computing machine.

14
 DEFINATION
A Turing machine (TM) is a finite-state machine with an infinite tape and
a tape head that can read or write one tape cell and move left or right.

It normally accepts the input string, or completes its computation, by

entering a final or accepting state.

Tape is use for input and working storage.

15
 The Key Question

•Is it possible to design a formal model of a computational device that

capture the capabilities of any algorithm?

Alan Turing, 1940’s: Yes!

16
 Basic Idea
Turing decomposed operations in the device as follows:

•A pencil will be “observing” symbols from a paper

•The pencil can move left and right
•Computational steps erase a symbol observed
by the pencil and write a new one in its place
•The decision about which symbol should be
written and which will be observed next depend on:
1. The current symbol being observed
2. The “state of mind” of the device
This device is called a Turing Machine
17
• TYPES OF TURING MACHINES
 Two way infinite tape
 Multiple tape Turing Machines
 Nondeterministic Turing machines
 Multidimensional Turing machines
 Multihead Turing Machines
 Off-line Turing machines

18
• The classical model for computation is the Turing Machine, named after Alan Turing,
who proposed it in his seminal paper.

• The goal is to be as simple a model of “computing "as one can come up with, yet very
powerful.

• A Turing Machine consists of a head (processor ) and a tape (memory). The tap e is
infinite and 1-dimensional, and consists of cells . Each cell contains a character σ ∈ Σ
from an alphabet Σ. That alphabet could be {0,1,2}, for instance, where 2 denotes an
empty cell.

19
• The head is always located on one of the cells of the tape. It has a finite set of states Q it
can be in.

• A transition function t tells the head what to do: when being in state q ∈ Q, and reading a
symbol σ ∈ Σ, the transition function specifies : (1) with what to over write the symbol on
the tape, (2) whether to move right or left one cell on the tap e, and (3) which new state q′
to enter . The Turing Machine always starts in a predefined start states, and as soon as it
reaches a predefined finish state f, it terminates

20
21
Description

1. A Turing machine can both write on the tape and read from it.
2. The read-write head can move both to the left and to the right.
3. The tape is infinite.
4. The special states for rejecting and accepting take immediate
effect.

22
23
24
25
 Representation of Turing Machine
Turing Machine is represented by- M=(Q,,τ,δ,q0,B,F),
• Where
• Q is the finite state of states
•  a set of τ not including B, is the set of input symbols,
• τ is the finite state of allowable tape symbols,
• δ is the next move function, a mapping from Q × τ to Q×τ
×{L,R}
• Q0 in Q is the start state,
• B a symbol of τ is the blank,
• F is the set of final states.
26
• As a Turing machine computes, it may halt with ‘accept’ or
‘reject’, or it may never halt!
• During computation, changes occur in
• δ(q , X) = (p ,Y ,R/L)
1. the current state,
2. the current tape contents,
3. Current tape contents written over
4. the current head location.
• The above three items form a “configuration” of the Turing
machine.

27
28
29
 Halting
• The machine halts in a state if there is no transition to follow

30
31
 Turing machine as transducers
• To use a Turing machine as a transducer, treat the entire
nonblank portion of the initial tape as input

• Treat the entire nonblank portion of the tape when the

machine halts as output.

32
33
34
35
36
37
 Exercises
• Design TM for the language anbnc n : n>0

 Construct TM for palindrome strings over alphabet (a,b)

 Construct TM for addition for two positive integers

 Construct TM for addition for two binary number

38
 MULTITAPE TURING MACHINES
 A Turing Machine with several tapes

 Every Tape’s have their Controlled own R/W Head

 For N- tape TM M=(Q,, Γ,δ,q0,B,F)

we define δ : Q X ΓN Q X ΓN X { L , R} N

39
For e.g., if n=2 , with the current configuration

δ( qO ,a ,e) =(q1, x ,y, L, R)

qO q1

a bc def x bc dy f
Tape 1 Tape 2 Tape 1 Tape 2

40
• The multi tape Turing machine is more efficient and easier to program
than single-tape Turing machines.
• For every multi tape Turing machine there is a one-tape Turing machine that
computes the same partial computable function.

41
 NON DETERMINISTIC TURING MACHINES

 It is similar to DTM except that for any input symbol and current state it has a number of choices
A nondeterministic Turing machine allows for the possibility of more than one next move
from a given configuration.
 If there is more than one next move, we do not specify which next move the machine makes,
only that it chooses one such move.
A string is accepted by a NDTM if there is a sequence of moves that leads to a final state
The transaction function δ : Q X Γ 2 QXΓ X { L , R}

Every nondeterministic Turing machine has an equivalent deterministic Turing machine.

42
 MULTIHEAD TURING MACHINE

 Multihead TM has a number of heads instead of one.

 Each head indepently read/ write symbols and move left / right or keep
stationery.

Control unit

a b c d e f g t

43
 Control Unit

Multihead TM
Head 1 Head 2 Head 3 Head 4
.. . a b c d e f g h ….

1st track …. 1 B B B B B B B ..

2nd track …. B B 1 B B B B B ..

B B B B 1 B B B ..
Multi track
3rd track ..
TM
.. B B B B B B 1 B .
4th track
5th track .. a b c d e f g h .
44
 OFF- LINE TURING MACHINE
 An Offline Turing Machine has two tapes

1. One tape is read-only and contains the input

2. The other is read-write and is initially blank.

a b c d Read- Only input

file’s tape

Control
unit

45
f g h i W/R tape
 MULTIDIMENSIONAL TURING MACHINE
A Multidimensional TM has a multidimensional tape.
For example, a two-dimensional Turing machine would read and
write on an infinite plane divided into squares, like a checkerboard.

 For a two- Dimensional Turing Machine transaction function define as:

δ:QXΓ Q X Γ X { L , R,U,D}

46
 Equivalence of TM’s and Computers
• In one sense, a real computer has a finite amount of memory, and thus is weaker than a TM.

• But, we can postulate an infinite supply of tapes, disks, or some peripheral storage device to
simulate an infinite TM tape. Additionally, we can assume there is a human operator to
mount disks, keep them stacked neatly on the sides of the computer, etc.

• Need to show both directions, a TM can simulate a computer and that a computer can
simulate a TM

47
 Computer Simulate a TM
• This direction is fairly easy - Given a computer with a modern programming language,
certainly, we can write a computer program that emulates the finite control of the TM.

• The only issue remains the infinite tape. Our program must map cells in the tape to storage
locations in a disk. When the disk becomes full, we must be able to map to a different disk in
the stack of disks mounted by the human operator.

48
 TM Simulate a Computer
• In this exercise the simulation is performed at the level of stored instructions and
accessing words of main memory.
• TM has one tape that holds all the used memory locations and their contents.

• Other TM tapes hold the instruction counter, memory address, computer input file, and scratch
data.

• The computer’s instruction cycle is simulated by:

1. Find the word indicated by the instruction counter on the memory tape.
2. Examine the instruction code (a finite set of options), and get the contents of any memory words
mentioned in the instruction, using the scratch tape.

3. Perform the instruction, changing any words' values as needed, and adding new address-value pairs to
the memory tape, if needed.

49
 TM/Computer Equivalence
• Anything a computer can do, a TM can do, and vice versa

• TM is much slower than the computer, though

• But the difference in speed is polynomial

• Each step done on the computer can be completed in O(n2) steps on the TM (see book for details of proof).

• While slow, this is key information if we wish to make an analogy to modern computers. Anything that we
can prove using Turing machines translates to modern computers with a polynomial time transformation.

• Whenever we talk about defining algorithms to solve problems, we can equivalently talk about how to
construct a TM to solve the problem. If a TM cannot be built to solve a particular problem, then it means our
modern computer cannot solve the problem either.

50
 Chapter two
Decidability
• Turing Decidable
• Turing Acceptable
• Undecidable Problems

51
 Decidability overview
• Every question about regular language is decidable
• Every question about context free language is decidable but some are not

52
• A solution to a decision problem is an algorithm that answers the question that results
from each instance.

• A decision problem is decidable if a solution exists and is undecidable otherwise.

• We will be concerned with the question of whether certain decision problems are
decidable.

• According to Church's thesis, every computational device can be simulated by some

Turing machine that solves the problem.

• To show that a decision problem is undecidable, we take the point of view that it
suffices to show that no Turing machine solves the problem.

53
The Turing machine that decides (solves) a problem answers YES or NO
for each instance of the problem
Turing Machine
YES
Input
problem NO
instance

54
55
•

56
57
 Computational problems
1. Decision problems
• answered by "yes" or "no“, “True” or “False”
2. Search problems
• In a search problem we not only want to know if a solution exists, but find the actual
solution as well.

3. Optimization problems

• Optimization problems ask for the best possible solution to a problem

• A decision or search problem can have several optimization variants

4. Counting problems
• ask the number of solutions of a given instance
58
• Computable functions
• Functions whose output values can be determined algorithmically from their input values

• Non-computable functions
• Functions that are so complex that there is no well-defined, step-by-step process for
determining their output based on their input values

• The computation of these functions lies beyond the abilities of any algorithmic system

59
 Consider problems with answer YES or NO
Examples:
• Does Machine M have three states ?
w a binary number?
• Is string
• Does DFAM accept any input?

A problem is decidable if some Turing machine

decides (solves) the problem

60
61
Search problems

62
Examples for Search Problems

63
64
65
66
67
68
Introduction - Undecidability

69
70
• In computability theory and computational complexity theory, an undecidable
problem is a decision problem for which it is impossible to construct a single
algorithm that always leads to a correct yes-or-no answer.

• In computability theory, the halting problem is a decision problem which can be stated
as follows:
• Given a description of a program and a finite input, decide whether the program finishes running or will run
forever.

• Alan Turing proved in 1936 that a general algorithm running on a Turing machine that
solves the halting problem for all possible program-input pairs necessarily cannot exist.
Hence, the halting problem is undecidable for Turing machines.

71
72
73
 Undecidability
• Some examples of undecidability problems

74
75
 Halting problem
• In computability theory, the halting problem is the problem of determining, an input,
whether the program will finish running or continue to run forever.

• Alan Turing proved in 1936 that a general algorithm to solve the halting problem
for all possible program-input pairs cannot exist.

• The halting problem is undecidable over Turing machines.

76
 Turing Machine Halting Problem
• Input − A Turing machine and an input string w.

• Problem − Does the Turing machine finish computing of the string w in a finite number of
steps? The answer must be either yes or no.

• Proof − At first, we will assume that such a Turing machine exists to solve this problem and
then we will show it is contradicting itself. We will call this Turing machine as a Halting
machine that produces a ‘yes’ or ‘no’ in a finite amount of time. If the halting machine finishes
in a finite amount of time, the output comes as ‘yes’, otherwise as ‘no’. The following is the
block diagram of a Halting machine −

77
78
79
80
81
Halting Problem- Example

82
83
Halting Problem is Undecidable-proof

84
85
86
• Showing that a problem is decidable, we can construct a Turing machine there is
a decider to decide that language.

• But, to show that a problem is undecidable, we should show that there is no

Turing machine which can decide a corresponding language.

• Therefore, given a problem to show that it is a undecidable it may not be very easy to
show that the problem is undecidable.

• Now, there another way to show that suppose that we have a problem or we have shown
it to be undecidable,

• reducibility

87
• What does undecidability mean for problems real people (not just CS
theorists) care about?
• What do you do when your application’s requirements require “solving”
an undecidable problem?

88
Reducibility

89
90
91
• Proving Undecidability via Reduction
• To prove a problem P is undecidable, we reduce a known undecidable problem Q to it. Proof by
reduction entails:

• Assume there is a program or algorithm DecideP that decides the problem P.

• Next, show that we can use DecideP to decide Q, by translating an instance of the Q problem to an
instance of the P problem and then applying DecideP. The translation must itself halt.

• This yields a contradiction, since Q is known to be undecidable.

• Q is typically the halting problem.

 Another example halting problem
• Fastest compiler
• Want a compiler that find a fastest possible machine code for any given program?
• Java script security
• You have java script with some a variable at high security level and some at low
security level. You want make sure that some attacker can’t get at high security
information.
• Parser
• You have a parser for your programming language. You change it, but you want
to make sure that it still parse all the program it used.

96
• Implications and Applications
• Most interesting questions about programs are undecidable.

• Will this Java program ever throw a NullPointerException?

• Will this program ever access a given object again?

• Will this program ever send sensitive information over the network?

• Will this program divide by 0?

• Will this program ever run out of memory, starting with a given amount available?

• Will this program ever try to treat an integer as an array?

 Chapter three
• Computability
• Recursive Functions
• Recursive Languages
• Recursively Enumerable Languages

98
• Recursive language(REC) – A language ‘L’ is said to be recursive if there exists a
Turing machine which will accept all the strings in ‘L’ and reject all the strings not in ‘L’.

• The Turing machine will halt every time and give an answer(accepted or rejected) for
each and every string input.

• Recursively enumerable language(RE) – A language ‘L’ is said to be a recursively

enumerable language if there exists a Turing machine which will accept (and therefore
halt) for all the input strings which are in ‘L’ but may or may not halt for all input strings
which are not in ‘L’.

• By definition , all REC languages are also RE languages but not all RE languages are
REC languages.

99
100
101
A language is recursively enumerable if some Turing
machine accepts it

102