Computational Complexity

Theory

1
 Computational complexity theory
• Even when a problem is decidable and thus
computationally solvable in principle, it may
not be solvable in practice if the solution
requires an inordinate amount of resources
(time or space)
• And the computational complexity theory
tries to investigate the time, memory, or
other resources required for solving
computational problems.
• But, we limit our discussion to issues of
time – complexity. 2
 Computational complexity theory
Goal: A general theory of the resources
needed to solve computational problems.
What types of resources?
time energy
space
What types of computational problems?
composing a poem optimization

sorting a database decision problem

3
 Decision problems
A decision problem is a computational
problem with a yes or no answer.
Example: Is the number n prime?
Why focus on decision problems?

Decision problems are simple: This makes it

easy to develop a rigorous mathematical
theory.

4
Decision problems are surprisingly general:
Many other problems can be recast in terms
of decision problems that are essentially
equivalent.

5
Recasting other problems as decision problems

Multiplication problem: What is the product

of m and n?
Multiplication decision problem: Is the kth
bit of the product of m and n a one?

Time required to solve one of these problems

is the same (to within a small overhead) as
the time required to solve the other.

6
Factoring problem: What is the smallest non-
trivial factor of n?

Factoring decision problem: Does n have a

non-trivial factor smaller than k?

Time required to solve one of these problems

is the same (to within a small overhead) as
the time required to solve the other.

7
 Efficiency
insoluble insoluble

hard
soluble
soluble soluble
easy

Nomenclature: easy = “tractable” =“efficiently

computable”
hard = “intractable” = “not efficiently
computable”
8
Definition: A problem is easy if there is a
Turing machine to solve the problem that runs
in time polynomial in the size of the problem
input. Otherwise the problem is hard.

A problem is poly-time solvable if T(n) =O(n k)

for some constant k.

This definition is usually applied to both

decision problems and more general problems.
9
 Why polynomial-time = “easy”

It’s a rule of thumb: in practice, problems with

polynomial-time solutions are usually found to
be easily soluble, while problems without are
usually found to be rather difficult.

10
Time Complexity

11
Time Complexity: The number of steps
during a computation

Space Complexity: Space used

during a computation

12
 Measuring Time Complexity
• Model of computation: TM
• timereq(M) is a function of n:
• If M is deterministic
timereq(M) = f(n) = the maximum
number of steps that M executes on any
input of length n.
• If M is nondeterministic
timereq(M) = f(n) = the number of
steps on the longest path that M executes
on any input of length n.
13
 Asymptotic upper bound - O
f(n)  O(g(n)) iff there exists a positive
integer k and a positive constant c such
that:
n  k, (f(n)  c g(n)).
In other words, ignoring some number of
small cases (all those of size less than k),
and ignoring some constant factor c, f(n)
is bounded from above by g(n).
In this case, we’ll say that f is “big-oh” of g
or g asymptotically dominates f or g grows
at least as fast as f does 14
 Asymptotic upper bound - O
● n3  O(n3).

● n3  O(n4).

● 3n3  O(n3).

● n3  O(3n).

● n3  O(n !).

● log n  O(n). 15
 Summarizing O

O(c)  O(loga n)  O(n b)  O(d n)  O(n !)

16
 Algorithmic Gaps
We’d like to show:

1. Upper bound: There exists an algorithm

that decides L and that has complexity C1.
2. Lower bound: Any algorithm that decides
L must have complexity at least C2.
3. C1 = C2.

If C1 = C2, we are done. Often, we’re not

done. 17
 n n
Example: L  {a b : n  0}

Let’s try to analyze the TM algorithm for L.

Algorithm to accept a string w:

1. Repeat if a’s and b’s remain on the tape:

2. Scan across the tape, crossing off a
single a and a single b.
3. If a’s or b’s still remain after all the b’s
or a’s respectively have been crossed off ,
reject. Otherwise accept.
18
•For stages 1 and 2 :
•Each scan uses O(n)
•Crossing of two symbols, at most n/2 scans O(n / 2)

•Total = O(n) O(n / 2)

2
= O(n ) steps
•Stage 3: makes a single scan to decide
whether to accept or reject, so at most O(n)
•Thus the total time M on input length n is:
• O ( n 2
)  O ( n ) or O ( n 2
)

19
 n n
L  {a b : n  0}

Using a multitape Turing machine

Time needed:

•Copy the a on the second tape O (n)

•Compare the a and b O (n)

Total time: O (n)

20
Complexity Classes

21
Complexity classes: are set of languages/
functions that can be decided/ computed
within a given resource.

Now let’s introduce our first complexity

classes

22
Consider a deterministic Turing Machine M
which decides a language L

23
For any string w the computation of M
terminates in a finite amount of transitions


Initial Accept
state or Reject w

24
 Decision Time = #transitions


Initial Accept
state or Reject w

25
Consider now all strings of length n

TM (n ) = maximum time required to decide

any string of length n

26
TIME TM (n )

1 2 3 4  n 
STRING LENGTH

Max time to accept a string of length n

27
Time Complexity Class: TIME (T (n ))

All Languages decidable by a

deterministic Turing Machine
in time O (T (n ))

L1 L2
L3
28
Example: L1  {a b : n  0}
n

This can be decided in O (n ) time

TIME (n )

L1  {a b : n  0}
n

29
Other example problems in the same class
TIME (n )

L1  {a b : n  0}
n

{ab aba : n , k  0 }
n

{b : n is even }
n

{b : n  3k }
n

30
Examples in class:

TIME (n ) 2

{a b : n  0 }
n n

{ww R : w  {a , b }}

{ww : w  {a , b }}

31
Examples in class:
TIME (n ) 3

CYK algorithm
L2  { G ,w : w is generated by
context - free grammar G }

Matrix multiplication
L3  { M1 , M2 , M3 : n  n matrices
and M1  M2  M3 }

32
Polynomial time algorithms: TIME (n )
k

constant k 0

Represents tractable algorithms:

for small k we can decide
the result fast

33
It can be shown: TIME (n k 1 ) TIME (n k )

TIME (n k 1 )

TIME (n ) k

34
The Time Complexity Class P

P   TIME (n k
)
k
0

Represents:

•polynomial time algorithms

• “tractable” problems
35
 Class P
{a b }
n

{a b }
n n
{ww }

CYK-algorithm RELPRIME

Matrix multiplication

36
Our first computational complexity class: “P”
Definition: The set of all decision problems soluble in
polynomial time on a Turing machine is denoted P.

decision problems

P
“easy” problems

Terminology: “Multiplication is in P” “Factoring is thought not to be in P”

means “The multiplication decision means “The factoring decision
problem is in class P”. problem is thought not to be in
class P”.
37
 Which model to use?

Complexity of a problem/language may depend

on the specific model of computation in which
the algorithms that solve the problem are
implemented.

…The same language may have different

time requirements on different models.

38
Doesn’t the definition of P depend upon the
computational model used in the statement
of the definition, namely, the Turing
machine?

39
 The strong Church-Turing thesis

Church-Turing thesis: Any algorithmic process

can be simulated on a Turing machine.

The strong Church-Turing thesis:

Any physically reasonable algorithmic process
can be simulated on a Turing machine, with at
most a polynomial slowdown in the number of
steps required to do the simulation.

40
That is, all reasonable deterministic
computational models are polynomially
equivalent.

The strong Church-Turing thesis implies that

the problems in P are precisely those for
which a polynomial-time solution is the best
possible, in any physically reasonable model of
computation.

41
Many important problems aren’t known to be in P

There are many problems in which a polynomial

time algorithm not known to exist to solve
them.
Example: Factoring.
Example: The traveling salesman problem (TSP).
22 km 23 km

12 km
15 km
19 km 14 km

16 km

Goal: Find the shortest tour through all the cities.

42
 nk
Exponential time algorithms: TIME (2 )

Represent intractable algorithms:

Some problem instances
may take centuries to solve

43
Traveling salesman decision problem: Given a
network and a number, k, is there a tour
through all the cities of length less than k?

It is widely believed that neither of these

problems is in P.

44
 Other Examples
Example: the Hamiltonian Path Problem

s t

Question: is there a Hamiltonian path

from s to t?

45
s t

YES!

46
A solution: search exhaustively all paths

L = {<G,s,t>: there is a Hamiltonian path

in G from s to t}

nk
L TIME (n ! )  TIME (2 )

Exponential time

Intractable problem
47
 The clique problem

Does there exist a clique of size 5?

48
 The clique problem

Does there exist a clique of size 5?

49
Example: The Satisfiability Problem

Boolean expressions in
Conjunctive Normal Form:
t1  t2  t3    tk clauses

ti  x1  x2  x3    x p
Variables
satisfying assignment: a true assignment causing
the output to be 1.
Question: is the expression satisfiable?
50
 Example: ( x1  x2 )  ( x1  x3 )

Satisfiable: x1  0, x2  1, x3  1

( x1  x2 )  ( x1  x3 )  1

51
Example: ( x1  x2 )  x1  x2

Not satisfiable

52
 L  {w : expression w is satisfiable}

nk
L TIME (2 )
exponential

Algorithm:
search exhaustively all the possible
binary values of the variables
53
 Witnesses/Certificates
Example: Factoring decision problem

Determining whether 7747 has a factor less

than 70 is thought to be hard.

Verifying that 61 is a factor of 7747 (= 127

x 61) less than 70 is easy: just check that 61
< 70 and use the efficient long-division
algorithm you learnt in school.

54
The factoring decision problem has witnesses
to yes instances of the problem – informally,
solutions to the factoring problem, which can
be checked in polynomial time.
There are problems hard to determine the
solution but easy to verify the solution once it
is found

TSP seems to be hard, but also has easily-

checked witnesses for yes instances.
55
PATH problem: Given <G,u,v,k>, whether
exists a path from u to v with at most k
edges?

Moreover, also given a path p from u to v,

verify whether the length of p is at most k?

Easy or not?
Of course, very easy.

56
Hamiltonian cycles
– A simple path containing every vertex.
– HAM-CYCLE={<G>: G is a Hamiltonian graph,
i.e. containing Hamiltonian cycle}.
• The naïve algorithm for determining HAM-
CYCLE runs in O(m!)=O(2m) time, where m
is the number of vertices, m n1/2.
• However, given an ordered sequence of m
vertices (called “certificate”), let you
verify whether the sequence is a
Hamiltonian cycle.
• Very easy. In O(n2) time.
57
 Class NP

Definition: The complexity class NP consists

of all decision problems for which yes
instances of the problem have witnesses
checkable in polynomial time on a Turing
machine.

58
Examples: The factoring and TSP decision problems.

Example: The Hamiltonian cycle problem: is

there a tour that goes through each vertex in
the graph exactly once?

Many important optimization problems are in NP – it

is often hard to find a solution, but easy to verify a
solution once it is found.
59
The term NP comes from nondeterministic
polynomial time.
… and is derived from an alternative
characterization by using nondeterministic
polynomial time Turing machines.

60
 Non-Determinism
Language class: NTIME (n)

NTIME (n)
L1 L3
L2

A Non-Deterministic Turing Machine

accepts each string of length n
in time O (n)
61
Example: L  {ww}

Non-Deterministic Algorithm
to accept a string ww :

•Use a two-tape Turing machine

•Guess the middle of the string

and copy w on the second tape

•Compare the two tapes 62

 L  {ww}
Time needed:

•Use a two-tape Turing machine

•Guess the middle of the string O(| w |)

and copy w on the second tape

O(| w |)
•Compare the two tapes
Total time: O(| w |)
63
NTIME (n)

L  {ww}

64
In a similar way we define the class

NTIME (T (n))

for any time function: T (n)

2 3
Examples: NTIME (n ), NTIME (n ),...
65
Non-Deterministic Polynomial time algorithms:

k
L  NTIME (n )

66
 The class NP

NP   NTIME (n k
)
k
0

Non-Deterministic Polynomial time

67
 Example: The satisfiability problem

L  {w : expression w is satisfiable}

Non-Deterministic algorithm:
•Guess an assignment of the variables

•Check if this is a satisfying assignment

68
 L  {w : expression w is satisfiable}

Time for n variables:

•Guess an assignment of the variables O (n)

•Check if this is a satisfying assignment O (n)

Total time: O(n)

69
 L  {w : expression w is satisfiable}

L  NP

The satisfiability problem is an NP - Problem

70
 The P vs. NP Question

P: Languages for which membership can be

decided quickly
• Solvable by a DTM in poly-time

NP: Languages for which membership can be

verified quickly (i.e. can be tested in poly-time )
• Solvable by a NDTM in poly-time

71
Set L is in P if membership in L can be
decided in poly-time.
Set L is in NP if each x in L has a short
“proof of membership” that can be verified
in poly-time.

Fact: P  NP
Question: Does NP  P ?

72
The relationship of P to NP

decision problems

NP P

73
 Observation:
P  NP

Deterministic Non-Deterministic
Polynomial Polynomial

NP includes all problems in P and some problems

possibly outside P
Yes instances for problems in P can be
verified in poly- time, even without a witness.
74
 Open Problem: P  NP ?

WE DO NOT KNOW THE ANSWER

This is the most famous unsolved problem in

computer science and mathematics.

75
Open Problem: P  NP ?
Example: Does the Satisfiability problem
have a polynomial time
deterministic algorithm?

If these classes were equal, then any

polynomally verifiable problem would be
polynomially decidable.

WE DO NOT KNOW THE ANSWER 76

 Why proving P  NP is harder than it looks

Intuition: There is no better way of finding a

solution than searching through most possible
witness.

77
Example: The Hamiltonian cycle problem: is there a
tour that goes through each vertex in the graph
exactly once?

Intuition: To determine whether

there is a Hamiltonian cycle, the
best way is to search through all
possible cycles.

78
 Why Care?
NP Contains Lots of Problems We Don’t Know
to be in P.
Some of them are:
Hamiltonian cycle
Clique
Classroom Scheduling
Packing objects into bins
Scheduling jobs on machines
Finding cheap tours visiting a subset of cities
Allocating variables to registers
Finding good packet routings in networks
Decryption
… 79
To reason about the P=NP problem?

That is how can we prove that NP  P?

I would have to show that every set in NP

has a polynomial time algorithm…

How do I do that?
It may take a long time!
Also, what if I forgot one of the sets in NP?

80
We can describe just one problem L in NP,
such that if this problem L is in P,
then NP  P.

It is a problem that can capture all other

problems in NP.

81