Computational

Complexity Classes
TAN TIEN PING
Computational Complexity
Classes
Previously, we use big-O to group algorithms to characterize the efficiency of an algorithm e.g.
O(n), O(n2), O(log n), O(n log n), …
This tell us how good an algorithm is. But it does not tell us how difficult a problem is and if a
better algorithm for the problem exist!
In computational complexity theory, computational problems are classified according to their
inherent difficulty. In another word, this tell us the best time complexity or the most efficient
algorithm can be achieved for a particular type of problem.
Turing machine is used in complexity analysis.
Computational Complexity
Classes
Why using Turing machine?
Consider the algorithms analysis that we carried out. The assumption that the addition/multiplication of
2 numbers take constant time may not be true because large number takes more operations to perform.
See https://en.wikipedia.org/wiki/Computational_complexity_of_mathematical_operations

Nevertheless, when it comes to analyze algorithms, we can analyze by using ”computer style code”. Why?
because the operations that matters in different algorithms for a problem is normally the same.
◦ e.g. matrix multiplication: naïve approach, and Strassen approach, they all use multiplication. So, it is safe to do
“apple to apple” comparison.

If the problems are different, the operations that matters are different (e.g. comparison in sorting and
multiplication in matrix multiplication), each operation is in practical do not take constant amount of
time, so we cannot compare “apple to orange”. We have to make all problems using the same type of
operation then only we can compare different problems
Turing Machine
Why Study Turing Machine?
A Turing machine is an automaton with the following properties
◦ More powerful than FSM and PDA. Powerful enough to simulate a computer
◦ Simple

Any algorithms in a modern computer can be modeled using a Turing machine

Help scientist understand the limitation of computing machine
Introduction
A Turing machine M has a writeable tape as storage.
Before M start, the input is written on the tape, one character per square

… □ □ □ a a b b b □ …

Finite State Controller

M’s tape is infinite in both directions
M has a single read/write head, shown with an arrow
Typically when M starts, the read/write head is at the left blank, immediately before the start of
the input
M begins in its start state. At each step, M must
◦ Choose its next state
◦ Write on the current square
◦ Move the read/ write head left or right one square
M can move back and forth on its tape
M will continue to execute until it reaches a special state called a halting state
Formal definition
A Turing machine is 6 tuple (K, , , , s, H):
◦ K is a finite set of states;
◦  is the input alphabet;
◦  is the tape alphabet, including □;
◦ s ∈ K is the initial state;
◦ H  K is a set of halting states;
◦  is the transition function
((q0, a), (q1, b, A)) means
◦ whenever M is in state q0
◦ the character under the read/ write head is a
◦ M will go to state q1
◦ write b
◦ and go left or right one square as specified by A

The transition is a function not a relation. Thus, it is a deterministic system

Example: Turing Machine
M is a Turing machine that takes an input string {aibj: 0<= j <=i} and add b’s to make the number
of b’s equal the number of a’s
Turing Machine: Algorithm
Procedure
i. Move 1 square to the right. If □ then halt, else continue
ii. Loop
◦ Mark off an a with $
◦ Scan rightward to the first b or □
• Mark it off with #
◦ Scan back leftward for a or □
• If a, go back to the top of the loop
• If □, exit loop
◦ Make one last pass through the nonblank area of the tape
• Change $ to a
• Change # to b
… □ □ a a a b b □ □ …

… □ □ $ a a b b □ □ …

… □ □ $ a a # b □ □ …

… □ □ $ a $ # b □ □ …

… □ □ $ a $ # # □ □ …
… □ □ $ $ $ # # □ □ …

… □ □ $ $ $ # # # □ …

… □ □ $ $ $ # # # □ …

… □ □ a a a b b b □ …
Turing Machine: Graphical
Language
1
a/a/→
$/$/→ $/a/→
□/ □/→ a/$/ → $/$/← #/b/→
#/#/→ #/#/←
a/$/→ b/#/ ← □/ □/→
2 3 4 5
□ /#/←
□/ □/← □/ □/←

x/t/a character under the rw head is x, write t, move rw head specified by a

Turing Machine as Language
Recognizer
Turing machine is more powerful than a finite state machine (FSM) and a pushdown automata.
Example: L={w Î anbncn} can be defined using Turing machine, but not FSM or pushdown automata.

a, 2 b, 3
1
a b c
>R 1R 2R 3L□

b, c □, 1, 3, c
□, 2, 3 □, a, 1, 2

2, 3 a, b, c, 1
R n

□ Note: this is a different way to draw

the state transition diagram
y
… □ □ a a b b c c □ …

… □ □ 1 a b b c c □ …

… □ □ 1 a 2 b c c □ …

… □ □ 1 a 2 b 3 c □ …

… □ □ 1 a 2 b 3 c □ …
… □ □ 1 a 2 b 3 c □ …

… □ □ 1 1 2 b 3 c □ …

… □ □ 1 1 2 2 3 c □ …

… □ □ 1 1 2 2 3 3 □ …

… □ □ 1 1 2 2 3 3 □ …

The r/w head will stop at the ‘y’ state, so “aabbcc”  L

Generator vs Recognizer
Example: Given 2 integers A & B, determine the sum.

Generator: Write a program to accept A & B as input then compute the sum A+B

Recognizer: Write a program to accept A & B & C as input then determine if A+B = C

We usually write Generators! But when would an Recognizer be an appropriate solution?

Turing Machine as Language
Recognizer
Using Turing machine as a decider is important, because we can cast any problem in the world
as a language/decision problem using appropriate encoding, and study how difficult the
problem is.

Problems that are already stated as decision problem

◦ Write down the strings which are defined by the language

Problem that are not stated as decision problem

◦ Reformulate the problem as decision problem
◦ Idea: Encode input(s) and results into a single string
◦ E.g. if function f takes in 2 inputs and produce an output, we construct a string of the form i1:i2:r. The string s=x:y:z is in language L
if function f produce the result r given i1 and i2
Turing Machine: Encoding a
Problem into a Decision problem
Example: Verifying addition
◦ <integer1> + <integer2> = <integer3>
◦ L = {w of the form: <integer1> + <integer2> = <integer3> : <integer1>, <integer2> and <integer3> is an
element of {0,1,2,3,4,5,6,7,8,9} and integer3 is the sum of integer1 and integer2}
◦ Examples of strings in L = {2+4=6, 23+47=70}

… □ □ □ 2 + 4 = 6 □ …
Casting Sorting Into a Decision
Problem
Casting sorting as decision:
Problem: Given a list of integers, sort it.
Encoding of the problem: Transform the sorting problem into one of examining a pair of lists.

The language to be decided:

◦ L= {w1 # w2:  n1 (w1 is of the form <int1, int2, … intn>, w2 is of the form <int1, int2, … intn>,
and w2 contains the same objects as w1, and w2 is sorted)}

Examples:
◦ 1,5,3,9,6#1,3,5,6,9  L
◦ 1,5,3,9,6#1,3,5,9,6  L
◦ 1,5,3,9,6#1,2,3,4,5,6,7  L
Turing Machine as Language
Recognizer
Decidable language: Let M be the Turing machine, s is the start state, y and n are 2
halting states. w is an element of ∑*
◦ M accepts w iff (s, □w) ⊢M* (y, w’) for some string w’, where (y, w’) is an accepting configuration
◦ M rejects w iff ((s, □w) ⊢M* (n, w’) for some string w’, where (n, w’) is rejecting configuration

We do not care about the content of tape when it halts, but only requires to know
whether it ends in a accepting state or rejecting state.
Set D/ recursive language is the set of all decidable languages (can be accepted or
rejected), and halt.
There are some languages that Turing machine also unable to decide
(undecidable) – halting problem: Given a program and input, determine whether
the program will halt.
Nondeterministic Turing Machine
A nondeterministic Turing machine allows multiple competing moves from a single
configuration. As long as one of the steps lead to accepting state, the string is
accepted.
A deterministic Turing machine can perform the same task a nondeterminstic
Turing machine can do
But.. It may take exponentially more steps to solve a problem using a deterministic
Turing machine than a nondeterministic one
competing moves
3 5
a/$/→ b/$/ →
□/ □/→
1 2
a/*/→ c/$/ →
4 6
A few Other Properties of Turing
Machine
If a computer requires n operations to perform a task, the Turing machine will require O(n6)
steps.
◦ It can be proved that Turing machine takes at most n6 steps to perform the same task.
◦ This is important, because it means we can use Turing machine to analyze problems.

Turing machine, M with multiple tapes is as powerful as a Turing machine with single tape, M’.
◦ It can be proved that if Turing machine M takes n steps to perform a task, M’ takes O(n2).
Turing Machine for Complexity
Analysis
Turing machine is used for the analysis of problem due to its simplicity. In the analysis.
◦ We will consider Turing machine with any fixed size tape alphabet.
◦ We will allow multiple tapes Turing machine.
◦ We will consider deterministic and nondeterministic Turing machine.

Time complexity counts the maximum number of steps that the Turing machine takes for an
input of size n and halt.
Space complexity is the maximum number of tape squares that M reads on any input of size n.
Turing Machine decides anbncn
Time complexity for Turing machine to decide anbncn

… □ □ a a b b c c □ …

n-1 steps

n steps

n+1 steps

Each time, match one a, one b and one c. It takes about 2n steps.
Takes about n loop to match all a, b and c. So time complexity is O(n 2).
Space required is n+2, so space complexity is O(n).
Computational Complexity
Classes
Problems can be divided based on how difficult it is to solve it.
Two of the complexity classes are P and NP.
For example, all the problems that can be solved quickly are
classified into the class call P, and problems that can be solved
slowly but can be verified the available answer quickly are in the
class call NP.
Complexity zoo is a catalogue of known complexity classes.
There are more than 500 classes, and new one are added from time
to time.
P
P or Polynomial-time class: on input of size n, their worst case running time is O(n k) for some
constant k.
P contains all decision problems which can be solved by a deterministic Turing machine using a
polynomial amount of computation time, or polynomial time.
P contains tractable problems: problems that are solvable in principle and also solvable in
reasonable amount of time.
d
note: polynomial in n of degree d: a n
p ( n) 
i 0
i
i

29
P
In general, all problems with the most efficient algorithms of O(1), O(n), O(log n) O(nlog n), O(nk)
where k is constant are in P.
Example of problems in P are: those problems that can be solved using finite state machine or
regular expressions, push down automata, finding min/max, search a value in an array...
NP
NP or nondeterministic polynomial time class: is the set of decision problems where the "yes"-
instances can be recognized/decided in polynomial time, O(nk) by a non-deterministic Turing
machine, M.
If a solution to the problem is known, then the solution can be verified with a deterministic
Turing machine in polynomial time.
Properties of NP problems
◦ Can be solved by searching through a space of partial solutions. The size of the space that must be
explored grows exponentially with the size of the problem, O(b n)
◦ No better (i.e. not based on search) technique for finding an exact solution is known
◦ But, if a proposed solution were known somehow, it can be verified for correctness very efficiently by
building a deterministic Turing machine, M’ that execute at polynomial time

31
NP
All problems in P is also in NP.
Example of problems in NP, but outside P: Sudoku, crossword puzzle, subset of sum, TSP
Decide...
Is P = NP? No one knows!
NP Problem
In order to explain the verifier-based definition of NP, let us consider the subset sum problem:
Assume that we are given some integers, such as {−7, −3, −2, 5, 8}, and we wish to know
whether some of these integers sum up to zero. In this example, the answer is "yes", since the
subset of integers {−3, −2, 5} corresponds to the sum (−3) + (−2) + 5 = 0. The task of deciding
whether such a subset with sum zero exists is called the subset sum problem.
As the number of integers that we feed into the algorithm becomes larger, the number of
subsets grows exponentially, and in fact the subset sum problem is NP-complete. However,
notice that, if we are given a particular subset (often called a certificate), we can easily check or
verify whether the subset sum is zero, by just summing up the integers of the subset. So if the
sum is indeed zero, that particular subset is the proof or witness for the fact that the answer is
"yes". An algorithm that verifies whether a given subset has sum zero is called verifier. A
problem is said to be in NP if and only if there exists a verifier for the problem that executes in
polynomial time. In case of the subset sum problem, the verifier needs only polynomial time, for
which reason the subset sum problem is in NP.
Material taken from: http://en.wikipedia.org/wiki/NP_%28complexity%29

33
Relation of P and NP
P ⊆ NP
◦ This means that all languages in P is also in NP

One of Seven Millennium Prize Problems, is P = NP?

So far, for NP problems, we cannot find a P solutions for these problems
At the same time, we are not able to prove that P solutions exist

Problems in (NP)c is even harder because even verifying the solution takes non-deterministic
polynomial amount of time.

34
P, NP, NP Hard and NP Complete
Classes

Assumption:
NP Complete
𝖭𝖯-complete problems are the most difficult 𝖭𝖯 problems, if we can solve an 𝖭𝖯-complete
problem efficiently, we can solve all 𝖭𝖯 problems efficiently.
Example of problem in NP Complete: TSPDecide, Sudoku, mine sweeper
 NP Hard
NP-hardness (non-deterministic polynomial-time hard), in computational complexity theory, is a
class of problems that are, informally, "at least as hard as the hardest problems in NP". More
precisely, a problem H is NP-hard when every problem L in NP can be reduced in polynomial
time to H, that is: assuming a solution for H takes 1 unit time, we can use H's solution to
solve L in polynomial time.
Problems in (NP)c is harder than NP because there is not even deterministic polynomial verifier
for it.
Example problem: chess, TSP.

Ref: http://continuations.com/post/77801135368/tech-tuesday-complexity-classes
Relationship of P and NP to other
complexity classes
PSPACE: For any language L, L ∈ PSPACE iff there exists some deterministic Turing machine M
that decides L and spacereq(M) ∈ O(nk) for some k
NPSACE: For any language L, L ∈ NPSACE iff there exists some nondeterministic Turing machine
M that decides L and spacereq(M) ∈ O(nk) for some k
EXPTIME: For any language L, L ∈ EXPTIME iff there exists some deterministic Turing machine M
that decides L and timereq(M) ∈ O(2n^k) for some k
P ⊆ NP ⊆ PSACE ⊆ EXPTIME

38