Theory of computation

Lecture 10: Complexity Theory

Adnane Saoud

Tuesday, December 26, 2023

1
 Summary of lecture 7

Lecture 9
• The Reducibility Method to prove undecidability and T-
unrecognizability.
• Mapping reducibility as a type of reducibility.

Lecture 10
• Introduction to Complexity Theory
• Complexity classes; the Class P; the Class NP

2
Time complexity

3
 Intro to Complexity Theory

Computability theory (1930s - 1950s):

Is A decidable?
Complexity theory (1960s - present):
Is A decidable with restricted resources?
(time/memory/…)

Example: Let 𝐴 = a! b! 𝑘 ≥ 0 .
Q: How many steps are needed to decide 𝐴?
Depends on the input.
We give an upper bound for all inputs of length 𝑛.
Called “worst-case complexity”.

4
 Number of steps to decide
𝐴 = a! b! 𝑘 ≥ 0
Theorem: A 1-tape TM 𝑀 can decide 𝐴 where, on inputs of length 𝑛,
𝑀 uses at most 𝑐𝑛" steps, for some fixed constant 𝑐.
Terminology: 𝑀 uses 𝑂(𝑛") steps. Analysis:
Proof: 𝑀 = “On input 𝑤 𝑂 𝑛 steps
1. Scan input to check if 𝑤 ∈ a∗b∗, reject if not. +𝑂(𝑛) iterations
2. Repeat until all crossed off. ×𝑂(𝑛) steps
Scan tape, crossing off one a and one b. ---------------------------
Reject if only a’s or only b’s remain. 𝑂 𝑛 + 𝑂(𝑛" ) steps
3. Accept if all crossed off. ” = 𝑂(𝑛" ) steps

Big 𝑶 and little 𝒐

Defn: 𝑓(𝑛) is 𝑂 𝑔 𝑛 if 𝑓 𝑛 ≤ 𝑐𝑔(𝑛)
𝑀 a a a a b b b b ˽ for some fixed 𝑐 independent of 𝑛.
Defn: 𝑓(𝑛) is 𝑜 𝑔 𝑛 if 𝑓 𝑛 ≤ 𝜖𝑔(𝑛)
for all 𝜖 > 0 and large 𝑛.

5
Number of steps to decide
𝐴 = a! b! 𝑘 ≥ 0

Exercise:
How much improvement is possible
in the bound for this theorem about
1-tape TMs deciding 𝐴?
(a) 𝑂(𝑛" ) is best possible.
(b) 𝑂(𝑛 log 𝑛) is possible.
(c) 𝑂(𝑛) is possible.

6
 Deciding 𝐴 = a! b! 𝑘 ≥ 0 faster

Theorem: A 1-tape TM 𝑀 can decide 𝐴 by using 𝑂(𝑛 log 𝑛) steps.

Proof:
𝑀 = “On input 𝑤 Analysis:
1. Scan tape to check if 𝑤 ∈ a∗b∗. Reject if not. 𝑂 𝑛 steps
2. Repeat until all crossed off. +𝑂(log 𝑛) iterations
Scan tape, crossing off every other a and b. ×𝑂(𝑛) steps
Reject if even/odd parities disagree. ---------------------------------
3. Accept if all crossed off. ” 𝑂 𝑛 + 𝑂(𝑛 log 𝑛) steps
= 𝑂(𝑛 log 𝑛) steps

𝑀 a a a a a a b b b b b b ˽

Stage 2 crosses off at least half the 0s and 1s Parities

each time it is executed, so at most log ! 𝑛 a’s even (6) odd (3) odd (1)
iterations of the repeat loop occur before all
get crossed off b’s even (6) odd (3) odd (1)

Further improvement? Not possible. 7

 Deciding 𝐴 = a! b! 𝑘 ≥ 0 even faster

Theorem: A multi-tape TM 𝑀 can decide 𝐴 using 𝑂(𝑛) steps.

𝑀 = “On input 𝑤 Analysis:
1. Scan input to check if 𝑤 ∈ a∗b∗, reject if not. 𝑂 𝑛 steps
2. Copy a’s to second tape. +𝑂(𝑛) steps
3. Match b’s with a’s on second tape. +𝑂(𝑛) steps
4. Accept if match, else reject. ” ------------------
= 𝑂(𝑛) steps

𝑀 a a a a a a b b b b b b ˽

a˽ ˽a a˽ ˽a ˽a ˽a ˽ ˽ ˽ ˽ ˽ ˽ ˽

8
 Model Dependence

Number of steps to decide 𝐴 = a! b! 𝑘 ≥ 0 depends on the

model.
• 1-tape TM: 𝑂(𝑛 log 𝑛)
• Multi-tape TM: 𝑂(𝑛)
Computability theory: model independence (Church-Turing Thesis)
Therefore model choice doesn’t matter. Mathematically nice.
Complexity Theory: model dependence
But dependence is low (polynomial) for reasonable deterministic
models.
We will continue to use the 1-tape TM as the basic model for
complexity.

9
 TIME Complexity Classes

Definition: Let 𝑡: ℕ → ℕ. Say TM 𝑀 runs in time 𝑡(𝑛) if

𝑀 always halts within 𝑡(𝑛) steps on all inputs of length 𝑛.
Definition: TIME 𝑡 𝑛 = {𝐵| some deterministic 1-tape TM 𝑀
decides 𝐵 and 𝑀 runs in time 𝑂 𝑡 𝑛 }
Example:
𝐴 = a! b! 𝑘 ≥ 0 ∈ TIME 𝑛 log 𝑛

# ...
TIME 2
TIME 𝑛" ...
TIME 𝑛!
TIME 𝑛 log 𝑛
Regular 𝐴
languages

10
TIME Complexity Classes

Exercice:
Let 𝐵 = 𝑤𝑤 ℛ 𝑤 ∈ a, b ∗ }.
What is the smallest function 𝑡
such that 𝐵 ∈ TIME 𝑡 𝑛 ?
(a) 𝑂(𝑛)
(b) 𝑂 𝑛 log 𝑛
(c) 𝑂(𝑛" )
(d) 𝑂 𝑛$
 Multi-tape vs 1-tape time
Theorem: Let 𝑡 𝑛 ≥ 𝑛.
If a multi-tape TM decides 𝐵 in time 𝑡(𝑛), then 𝐵 ∈ TIME 𝑡 " 𝑛 .
Proof: Analyze conversion of multi-tape to 1-tape TMs.
a a b b a ˽ ˽ ...
𝑀 1 0 1 ˽ ...
...

multi-tape c c c a ˽ ...

𝑆 a a b b a # 1 0 1 # … # c c c a ˽ ˽

1-tape 𝑂 𝑡 𝑛

To simulate 1 step of 𝑀’s computation, 𝑆 uses 𝑂 𝑡 𝑛 steps.

So total simulation time is 𝑂 𝑡 𝑛 × 𝑡 𝑛 = 𝑂 𝑡 " 𝑛 .
Similar results can be shown for other reasonable deterministic models.
12
 Relationships among models

Informal Definition: Two models of computation are polynomially

related if each can simulate the other with a polynomial overhead:
So 𝑡 𝑛 time → 𝑡 ! (𝑛) time on the other model, for some 𝑘.

All reasonable deterministic models are polynomially related.

• 1-tape TMs
• multi-tape TMs
• multi-dimensional TMs
• random access machine (RAM)
• cellular automata

13
 The Class P

Definition: P = ⋃! TIME(𝑛! )
= polynomial time decidable languages
• Invariant for all reasonable deterministic models
• Corresponds roughly to realistically solvable problems

𝑃𝐴𝑇𝐻 = 𝐺, 𝑠, 𝑡 𝐺 is a directed graph with a path from 𝑠 to 𝑡 }

𝐺
𝑠 𝑡

14
 The Class P

Example: 𝑃𝐴𝑇𝐻 = 𝐺, 𝑠, 𝑡 𝐺 is a directed graph with a path from 𝑠 to 𝑡 }

Theorem: 𝑃𝐴𝑇𝐻 ∈ P
Proof: 𝑀 = “On input 〈𝐺, 𝑠, 𝑡〉
1. Mark 𝑠
2. Repeat until nothing new is marked: ≤ 𝑛 iterations
For each marked node 𝑥: × ≤ 𝑛 iterations
Scan 𝐺 to mark all 𝑦 where 𝑥, 𝑦 is an edge × 𝑂 𝑛" steps
3. Accept if 𝑡 is marked. Reject if not. -------------------
𝑂(𝑛% ) steps

To show polynomial time:

Each stage should be clearly
polynomial and the total
number of stages polynomial.

15
 𝑃𝐴𝑇𝐻 and 𝐻𝐴𝑀𝑃𝐴𝑇𝐻

Example: 𝐻𝐴𝑀𝑃𝐴𝑇𝐻 = 𝐺, 𝑠, 𝑡 𝐺 is a directed graph with a path from 𝑠 to 𝑡

and the path goes through every node of 𝐺 }
Recall Theorem: 𝑃𝐴𝑇𝐻 ∈ P Called a Hamiltonian path
Question: 𝐻𝐴𝑀𝑃𝐴𝑇𝐻 ∈ P ?
“On input 𝐺, 𝑠, 𝑡
1. Let 𝑚 be the number of nodes in 𝐺. 𝐺
2. For each path of length 𝑚 in 𝐺: 𝑠 𝑡
test if 𝑚 is a Hamiltonian path from 𝑠 to 𝑡.
Accept if yes.
3. Reject if all paths fail.”
May be 𝑚! > 2& paths of length 𝑚
so algorithm is exponential time
not polynomial time.

16
 𝑃𝐴𝑇𝐻 and 𝐻𝐴𝑀𝑃𝐴𝑇𝐻

Exercise:
Is 𝐻𝐴𝑀𝑃𝐴𝑇𝐻 ∈ P ?
(a) Definitely Yes. You have a polynomial-time algorithm.
(b) Probably Yes. It should be similar to showing 𝑃𝐴𝑇𝐻 ∈ P.
(c) Toss up.
(d) Probably No. Hard to beat the exponential algorithm.
(e) Definitely No. You can prove it!

17
 Nondeterministic Complexity

In a nondeterministic TM (NTM) decider, all branches halt on all inputs.

Definition: An NTM runs in time 𝑡(𝑛) if all branches halt within 𝑡(𝑛)
steps on all inputs of length 𝑛.

Definition: NTIME 𝑡 𝑛 = {𝐵| some 1-tape NTM decides 𝐵

and runs in time 𝑂 𝑡 𝑛 }
Definition: NP = Computation tree
for NTM on input 𝑤.
⋃! NTIME(𝑛! )
= nondeterministic
polynomial time decidable
𝑡 𝑛
languages

...
all branches halt
18
within 𝑡(𝑛) steps
 𝐻𝐴𝑀𝑃𝐴𝑇𝐻 ∈ NP
Computation of
Theorem: 𝐻𝐴𝑀𝑃𝐴𝑇𝐻 ∈ NP M on 〈𝐺, 𝑠, 𝑡〉

Proof:
“On input 〈𝐺, 𝑠, 𝑡〉 (Say 𝐺 has 𝑚 nodes.) Guess
1. Nondeterministically write a sequence bits of 𝑣$
⋮
𝑣@, 𝑣", … , 𝑣A of 𝑚 nodes.
2. Accept if 𝑣@ = 𝑠 Guess
𝑣A = 𝑡 ⋮ bits of 𝑣!
each (𝑣B , 𝑣BC@) is an edge ⋮
and no 𝑣B repeats.
3. Reject if any condition fails.” Guess
bits of 𝑣%
⋮

⋯ Check
𝑣$ , 𝑣! , … , 𝑣%
c / rej c / rej works
ac ac
 𝐶𝑂𝑀𝑃𝑂𝑆𝐼𝑇𝐸𝑆 ∈ NP

Definition:
𝐶𝑂𝑀𝑃𝑂𝑆𝐼𝑇𝐸𝑆 = 𝑥 𝑥 is not prime and 𝑥 is written}
= 𝑥 𝑥 = 𝑦𝑧 for integers 𝑦, 𝑧 > 1}
Theorem: 𝐶𝑂𝑀𝑃𝑂𝑆𝐼𝑇𝐸𝑆 ∈ NP
Proof: “On input 𝑥
1. Nondeterministically write 𝑦 where 1 < 𝑦 < 𝑥.
2. Accept if 𝑦 divides 𝑥 with remainder 0.
Reject if not.”

Theorem (2002): 𝐶𝑂𝑀𝑃𝑂𝑆𝐼𝑇𝐸𝑆 ∈ P

We won’t cover this proof.

5
 Intuition for P and NP

NP = All languages where can verify membership quickly

Examples of quickly verifying membership:
- 𝐻𝐴𝑀𝑃𝐴𝑇𝐻: Give the Hamiltonian path.
- 𝐶𝑂𝑀𝑃𝑂𝑆𝐼𝑇𝐸𝑆: Give the factor.
The Hamiltonian path and the factor are called short certificates of membership.
P ⊆ NP
Question: P = NP? Famous unsolved problem (Cook 1971).
Conjecture: P ≠ NP. Some problems are NP and not in P.
Hard to prove the conjecture because polynomial-time algorithms are powerful.
 Summary

1. Introduction to Complexity Theory

2. Which model to use? 1-tape-TMs
3. TIME 𝑡 𝑛 complexity classes
4. The class P
5. 𝑃𝐴𝑇𝐻 ∈ P
6. HAM𝑃𝐴𝑇𝐻 ∈ NP

22