Bonga University

College of Engineering and Technology

Department of Computer Science
CoSc4132 – Automata & Complexity theory
Chapter 1 Handouts – Introduction
 Introduction
• Central areas of the theory of computation are:
automata, computability, and complexity.
• They are linked by the question:
“What are the fundamental capabilities and limitations
of computers?”
• This question goes back to the 1930s when
mathematical logicians first began to explore the
meaning of computation.
• Technological advances since that time have greatly
increased our ability to compute and have brought
this question out of the realm of theory into the
world of practical concern.
 Cont…..
Theory of computation: is the branch of computer
science how effectively deal with the problem using
algorithm.
Writing affective algorithm that run on computing
machine
Computational problem considers
• What can/can‟t computed
• Speed of computation
• Amount of memory used for computation
Three ranch of theory of computation
 Automata theory
 Complexity theory
 Computability theory
 cont.….
Automata theory: deals with definition and properties
of mathematical model of computation
Automaton is a machine that operates singularly on
input and follows a defined pattern or configuration to
produce the desired output.
 COMPEXITY THEORY
Computer problems come in different varieties;
some are easy and some are hard.
For example, the sorting problem is an easy one.
• Even a small computer can sort a million numbers
rather quickly. Compare that to a scheduling
problem.
• The scheduling problem seems to be much harder
than the sorting problem. If you have just a
thousand classes,
• finding the best schedule may require centuries,
even with a supercomputer.
 Mathematical preliminaries and
notations
• Set: is a collection of elements, without any structure
other than membership.
If m is an element of the set S, we write m ε S. A set is
specified by enclosing some description of its elements in curly
braces. E.g. S={a,b,c}
Universal set U of all possible elements is, if U
is specified, then S={ x: x ε U, x ɆS}
The set with no element is called the empty set
or null set. It is denoted by Ø.
 Con’t

Operation on sets
• Let A and B be two sets and
 A c B iff, for all x ε A, x ε B
 A=B iff, A c B and B c A
 A c B iff, Ac B but A ≠ B
 A u B={x: x ε A or x ε B}
 A ∩ B={x: x ε A and x ɆB}
 A X B={ (x, y), x ε A and y ε B}
 Formal Language

• A formal language is an abstraction of the general

characteristics of programming languages.
• A formal language consists of a set of symbols and some
rules of formation by which these symbols can be combined
into entities called sentences.
 Con’t
Formal Language Natural language

• Developed with strict rules. • Rules come after the

• Predefined syntax and language
semantics. • Evolve and develop
• Precise • Highly flexible
• Unambiguous • Quite powerful
• No special learning
Disadvantage: effort needed
• Unfamiliar natation Disadvantage:
• Initial learning effort • Vague
E.g. C++, Pascal,…. • Imprecise and ambiguous
• User and context dependent
E.g. Amharic, English, ….
 Alphabet and Strings
An alphabet is a finite set of symbols which are used to
form words in a language. An example of an alphabet
might be a set like
{a, b}.
• Example: Σ = {a, b, c, d} is an alphabet set
where „a‟, „b‟, „c‟, and „d‟ are alphabets.
 Con’t

• String:
– string over Σ is some number of elements of Σ (possibly
none) placed in order. So if Σ ={ a ,b} then strings over Σ
can be a, ab, bbaa, abab and so on.
– Length of a String
It is the number of symbols present in a string. (Denoted
by |S|). E.g.
• If S=„cabcad‟, |S|= 6
• If |S|= 0, it is called an empty string
(Denoted by λ or ε)
 Con’t

Kleene Star
 Definition: The set Σ* is the infinite set of all possible
strings of
all possible lengths over Σ including λ.
 Representation: Σ* = Σ0 U Σ1 U Σ2 U…….
 Example: If Σ = {a, b}, Σ*= {λ, a, b, aa, ab, ba,
bb,………..}
Kleene Closure / Plus
– Definition: The set Σ+ is the infinite set of all possible
strings of all possible lengths over Σ excluding λ.
– Representation: Σ+ = Σ0 U Σ1 U Σ2
U……. Σ+ = Σ* − { λ }
– Example: If Σ = { a, b } , Σ+ ={ a, b, aa, ab, ba,
bb,………..}
 Con’t
Transpose Operation
• for any string x ε Σ* and a ε Σ, (xa)T = a(x)T.
• A palindrome of even length can be obtained by the
concatenation of a string and its transpose.
e.g. Let Σ={0,1}
Y= (01001)T
(xa)T=(0100 1)T
=1(0100)T
=10(010)T
=100(01)T
=10010
 Language

• Definition : A language is a subset of Σ* for some

alphabet Σ. It can be finite or infinite.
• Language is a set of string generated by a grammar
• Example : If the language takes all possible strings of
length 2 over Σ = {a, b}, then L = { ab, bb, ba, bb}
 Grammars
• Grammars denote syntactical rules for conversation in
natural languages. Linguistics have attempted to define
grammars since the inception of natural languages like
English, Amharic.
• Strings may be derived from other strings using the
productions in a grammar.
• Set of rule used to describe the string of a language
• The set of all strings that can be derived from a grammar
is said to be the language generated from that grammar.
 Cont…
A grammar G is defined as a quadruple (represent in four
tuples)
G =(V, T, S, P),
where V is a finite set of objects called variables,
T is a finite set of objects called terminal symbols,
S ∈ V is a special symbol called the start variable,
P is a finite set of productions
The production rules are the heart of a grammar; they specify
how the grammar transforms one
string into another, and through this they define a language
associated with the grammar.
 Cont.…
Consider the grammar
Grammar G
(({S, A}, {a, b}, S,{S → aAb, aA → aaAb, A → ε } )
Here,
S and A are Non-terminal symbols.
a and b are Terminal symbols.
ε is an empty string.
S is the Start symbol, S ∈ N
Production P : S → aAb, aA → aaAb, A → ε
 Cont.…
Strings may be derived from other strings using the
productions in a grammar.
Some of the strings that can be derived are −
S ⇒ aAb using production S → aAb
⇒ aaAbb using production aA → aAb
⇒ aaaAbbb using production aA → aaAb
⇒ aaabbb using production A → ε
aaabbb is the string that generated from the above grammar
We can also drive another string
S ⇒ aAb using production S → aA
⇒ aaAbb using production aA → aaAb
⇒ aabb using production A → ε
The string “aabbb” generated from the grammar
 Cont.…
Normally, a given language has many grammars that generate
it. Even though these grammars are
different, they are equivalent in some sense. We say that two
grammars G1 and G2 are equivalent if
they generate the same language, that is, if
L(G1)= L(G2)
E.g..
G1 (({S,}, {a, b}, S,{S → aSb, S → ε} )
and
G2 (({S,A}, {a, b}, S,{S → aSb, s → ε, S → aAb, A → ε} )
G1 and G2 are equivalent hence they can generate the same
language
 Exercise

1. construct a grammar for a language

alphabet Σ = {a, b}:

(a) All strings in the language L : {anbm |n>0, m ≥ 0}

(b) All strings in the language L : {anbm |m>0, n ≥ 0}
(c) All strings in the language Contains at least two a’s
2. Check the following grammers are equivelent or not
(a) G1:({S, A, B}, {a, b}, S, {S → AB, A → a, B → b}) and
G2:({S, A}, {a, b}, S, {S → A, A → a, A→ b})
(b) G1:({S, A, B,C}, {a, b,c}, S, {S → ABC, A → a, B → b, C→ c})
and
G1:({S, A, B,C}, {a, b,c}, S, {S → ABC, A → aA, B → bB, C→ cC}
 Finite Automata (FA)

• The term "Automata" is derived from the Greek word

"αὐτόματα" which means "self- acting". An automaton
(Automata in plural) is an abstract self-propelled computing
device which follows a predetermined sequence of
operations automatically.
• An automaton with a finite number of states is called a
Finite Automaton (FA) or Finite State Machine (FSM).
 Con’t
• Finite Automata (Finite state machine)
•We denote a FA by the 5-tuple(Ǫ, E,q0,δ,A)
where
Q: Finite set of states.
•∑: Set of input symbols.
•q: Initial state.
•F: Set of final states.
•δ: Transition function
• Finite Automaton can be classified into two types:
– Deterministic Finite Automata (DFA)
– Non-deterministic Finite Automata (NDFA)
 Con’t

Deterministic Finite Automaton (DFA)

• In DFA, for each input symbol, one can determine the state
to which the machine will move. Hence, it is called
Deterministic Automaton.
• As it has a finite number of states, the machine is called
Deterministic Finite Machine or Deterministic Finite
Automaton.
 Con’t

Non-deterministic Finite Automaton (NDFA)

• In NDFA, for a particular input symbol, the machine can
move to any combination of the states in the machine. In
other words, the exact state to which the machine moves
cannot be determined.
• Hence, it is called Non-deterministic Automaton. As it has
finite number of states, the machine is called Non-
deterministic Finite Machine or Non-deterministic Finite
Automaton.
 Deterministic Finite Automaton
(DFA)
Formal Definition of a DFA
A DFA can be represented by a 5-tuple (Q, Σ, δ, q0, F) where:
• Q is a finite set of states.
• Σ is a finite set of symbols called the input alphabet.
• δ is the transition function where δ: Q × Σ → Q
• q0 is the initial state from where any input is processed
(q0 ∈ Q).
• F is a set of final state/states of Q (F ⊆ Q).
 Con’t

Graphical Representation of a DFA

A DFA is represented by digraphs called state diagram:
• The vertices represent the states.
• The arcs labeled with an input alphabet show the
transitions.
• The initial state is denoted by an empty single
incoming arc.
• The final state is indicated by double circles.
 Con’t

Example
Let a deterministic finite automaton be
• Q = {a, b, c},
• Σ = {0, 1},
• q0={a},
• F={c}, and
 Con’t
• Transition function δ as shown by the
following table:
 Con’t

• Its graphical representation would be as follows:

 Con’t
•Find the result of δ(a, 0101) Note
Solution: The language accepted by a
δ( a, o1o1) DFA M = (Q, Σ,δ, q0,F) is the
= δ(δ( a,0), 101) set of all strings on Σ
accepted by M. In M iff
=δ (a, 101)
δ(q0,w) =qo for some state
=δ (δ(a,1), 01) qo in F.
=δ(b, 01)
=δ(δ(b,0),1) Let M = (Q, Σ, δ,q0,F) be a
=δ(c,1) DFA. Then the language
accepted by M is the set:
=c L(M) = {w | w is in Σ* and
c is Ꜫ final sate hence the δ(q0,w) is in F}
string “o1o1” accepted by
DFA
Non-deterministic Finite Automaton (NFA)
NFA means a choice of moves for an automaton. Rather than
prescribing a unique move in each situation, we allow a set of
possible moves
Formal Definition of an NDFA
An NDFA can be represented by a 5-tuple (Q, Σ, δ, q0, F)
where:
• Q is a finite set of internal states.
• Σ is a finite set of symbols called the input alphabets.
•δ is the transition function where δ: Q × {Σ U ε} → 2Q
(Here the power set of Q (2Q) has been taken because in case
of NDFA, from a state, transition can occur to any
combination of Q states)
• q0 is the initial state from where any input is processed (q0
∈ Q).
• F is a set of final state/states of Q (F ⊆ Q).
 Con’t

Graphical Representation of an NDFA: (same as DFA)

An NDFA is represented by digraphs called state diagram.
• The vertices represent the states.
• The arcs labeled with an input alphabet show the transitions.
• The initial state is denoted by an empty single incoming
arc.
• The final state is indicated by double circles.
 Con’t
Example
• Let a non-deterministic finite automaton be
NFA M = (Q,Σ,δ, q0,F)
• Q = {a, b, c} Σ = {0, 1} q0 = {a} F={c}

A string is accepted by an NFA if there is some sequence of

possible moves that will put the machine in a final state at
the end of the string. i.e. the language consists of all strings
w for which there is a walk labelled w from the initial
vertex of the transition graph to some final vertex

A string is rejected (that is, not accepted) only if there

is no possible sequence of moves by which a final state can
be reached.
 Con’t
The transition function δ as shown below:

• Its graphical representation would be as follows:

 Converting an NDFA to an
Equivalent DFA
• Let X = (Qx, Σ, δx, q0, Fx) be an NDFA which accepts the
language L(X). We have to design an equivalent DFA Y =
(Qy, Σ, δy, q0, Fy) such that L(Y) = L(X).
 Con’t

The following procedure converts the NDFA to its equivalent

DFA:
Algorithm:
Input: An NDFA
Output: An equivalent DFA
• Step 1 Create state table from the given NDFA.
• Step 2 Create a blank state table under possible input
alphabets for the equivalent DFA.
• Step 3 Mark the start state of the DFA by q0 (Same as
the NDFA).
 Con’t

• Step 4 Find out the combination of States {Q0, Q1,... , Qn} for
each possible input alphabet.
• Step 5 Each time we generate a new DFA state under the
input alphabet columns, we have to apply step 4 again,
otherwise go to step 6.
• Step 6 The states which contain any of the final states of the
NDFA are the final states of the equivalent DFA.
 Con’t

Example 1
• Let us consider the NDFA shown in the figure below.
Con’t
Con’t
 The difference between DFA & NFA
DFA NFA
For each symbolic representation of No need to specify how does the
the alphabet, there is only one state NFA react according to some
transition in DFA. symbol.

DFA cannot use Empty String NFA can use Empty String transition
transition.
Time needed for executing an input Time needed for executing an input
string is less string is more

All DFA are NFA All NFA are DFA

DFA requires more space NFA requires less space than DFA
DFA requires more space
δ: QxΣ -> Q i.e. next possible state
belongs to Q. δ: QxΣ -> 2^Q i.e. next possible state
belongs to power set of Q.
 Con’t
Exercise
1. Find a deterministic finite accepter that recognizes the set
of all strings on Σ= {a,b} starting with the prefix ab.
2. Construct grammar for L(M) = {x | x is a string of a’s and
b’s and |x| >= 2}
3. construct NFA all the strings with exactly two a‟s and
more than two b‟s.
4. .Construct an equivalent DFA for a given NFA
Formal Language and
Automata Theory
Chapter Two

Regular expression and regular

language
 Regular Expressions
• A language is regular if there exists accepter
for it.
• The notation of regular expressions involves a
combination of strings of symbols from some
alphabet Σ, parentheses, and the operators +, .,
and *.
 Formal definition of RE
• Let Σ be a, given alphabet. Then
1. φ,λ and a ε Σ are all regular expressions. These
are called primitive regular expressions.
2. If r1 and r2 are regular expressions, so are r1+r2,
r1*r2, r1.r2, r1*, and (r1).
3. A string is a regular expression if and only if it
can be derived from the primitive regular
expressions by a finite number of applications of
the rules in (2).
 Languages associated with RE
A Regular Expression can be recursively
defined as follows:
1. ε is a Regular Expression indicates the
language containing an empty string. (L (ε) =
{ε})
2. φ is a Regular Expression denoting an empty
language. (L (φ) = { })
3. x is a Regular Expression where L={x}
 Con’t
4.If X is a Regular Expression denoting the
language L(X) and Y is a Regular Expression
denoting the language L(Y), then
a. X + Y is a Regular Expression corresponding to the
language L(X) U L(Y) where L(X+Y) = L(X) U
L(Y).
b. X . Y is a Regular Expression corresponding to the
language L(X) . L(Y) where L(X.Y)= L(X) . L(Y)
c. R* is a Regular Expression corresponding to the
language L(R*) where L(R*) = (L(R))*
5.If we apply any of the rules several times from 1
to 5, they are Regular Expressions.
 Con’t
Example:
• For Σ={a,b}, the expression
r = (a+ b )* ( a + bb )
• is regular. It denotes the language
L ( r ) : { a, bb, aa, abb, ba, bbb, .. . . }
• We can see this by considering the various parts
of r. The first part, (a + b)*, stands for any string
of a's and b's. The second part, (a + bb) represents
either an a or a double b. Consequently, L (r) is
the set of all strings on {a,b}, terminated by either
an a or a bb.
Con’t
Con’t
 Regular Sets
• Any set that represents the value of the Regular Expression is called
a Regular Set.
Properties of Regular Sets
Property 1. The union of two regular set is regular.
Proof:
Let us take two regular expressions
RE1 = a(aa)* and RE2 = (aa)*
So, L1= {a, aaa, aaaaa,.....} (Strings of odd length excluding Null)
and L2={ ε, aa, aaaa, aaaaaa,.......} (Strings of even length including Null)
L1 ∪L2 = { ε,a,aa, aaa, aaaa, aaaaa, aaaaaa,.......}
(Strings of all possible lengths excluding Null) AutomataTheory
21
RE (L1 ∪L2) = a* (which is a regular expression itself)
 Con’t
Property 2. The intersection of two regular set is regular.
Proof:
Let us take two regular expressions
RE1 = a(a*) and RE2 = (aa)*
So, L1 = { a,aa, aaa, aaaa, ....} (Strings of all possible lengths
excluding Null)
L2 ={ ε, aa, aaaa, aaaaaa,.......} (Strings of even length
including Null)
L1 ∩ L2 = { aa, aaaa, aaaaaa,.......} (Strings of even length
excluding Null)
RE (L1 ∩ L2) = aa(aa)* which is a regular expression itself.
 Con’t
Property 3. The complement of a regular set is regular.
Proof:
Let us take a regular expression:
RE = (aa)*
So, L = {ε, aa, aaaa, aaaaaa, .......} (Strings of even length
including Null)
Complement of L is all the strings that is not in L.
So, L’ = {a, aaa, aaaaa, .....} (Strings of odd length
excluding Null)
RE (L’) = a(aa)* which is a regular expression itself.
 Con’t
Property 4. The difference of two regular set is regular.
Proof:
Let us take two regular expressions:
RE1 = a (a*) and RE2 = (aa)*
So, L1= {a,aa, aaa, aaaa, ....} (Strings of all possible
lengths excluding Null)
L2 = { ε, aa, aaaa, aaaaaa,.......} (Strings of even length
including Null)
L1 – L2 = {a, aaa, aaaaa, aaaaaaa, ....}
(Strings of all odd lengths excluding Null)
RE (L1 – L2) = a (aa)* which is a regular expression.
 Con’t
Property 5. The reversal of a regular set is regular.
Proof:
We have to prove L(R) is also regular if L is a
regular set.
Let, L= {01, 10, 11, 10}
RE (L)= 01 + 10 + 11 + 10
L(R)= {10, 01, 11, 01}
RE (LR)= 01+ 10+ 11+10 which is regular
 Con’t
Property 6. The closure of a regular set is regular.
Proof:
If L = {a, aaa, aaaaa, .......} (Strings of odd length
excluding Null)
i.e., RE (L) = a (aa)*
L*= {a, aa, aaa, aaaa , aaaaa,……………} (Strings
of all lengths excluding Null)
RE (L*) = a (a)*
 Con’t
Property 7. The concatenation of two regular sets is regular.
Proof:
Let RE1 = (0+1)*0 and RE2 = 01(0+1)*
Here, L1 = {0, 00, 10, 000, 010, ......} (Set of strings ending in
0)
and L2 = {01, 010,011,.....} (Set of strings beginning with 01)
Then, L1 L2 =
{001,0010,0011,0001,00010,00011,1001,10010,.............}
Set of strings containing 001 as a substring which can be
represented by an RE: (0+1)*001(0+1)*
 Identities Related to Regular Expressions
• Given R, P, L, Q as regular expressions, the following
identities hold:
1. Ø* = ε
2.ε* =ε
3.R+ = RR* =
R*R 4. R*R* =
R*
5. (R*)* = R*
6. RR* = R*R
7. (PQ)*P =P(QP)*
8. (a+b)* = (a*b*)* = (a*+b*)* = (a+b*)* = a*(ba*)*
9. R + Ø = Ø + R = R (The identity for union)
10. Rε = εR = R (The identity for concatenation)
11.ØL = LØ = Ø (The annihilator for concatenation)
12.R + R = R (Idempotent law)
13.L (M + N) = LM + LN (Left distributive law)
14.(M + N) L = LM + LN (Right distributive law)
15.ε + RR* = ε + R*R = R*
 Construction of an RE from an FA
Problem
• Construct a regular expression corresponding to the
automata given below:
 Con’t
Solution:
Here the initial state is q1 and the final state
is q2
Now we write down the
equations: q1 = q10 + є
q2 = q11 + q20
q3 = q21 + q30 + q31
Now, we will solve these three
equations: q1 = є0* *As, εR = R+
So, q1 = 0* s
q2 = 0*1 +
q20
So, q2 = 0*1(0)* *By Arden’s theorem+
Hence, the regular expression is 0*10*.
 Construction of an FA from an RE

• We can use Thompson's Construction to find

out a Finite Automaton from a Regular
Expression.
• We will reduce the regular expression into
smallest regular expressions and converting
these to NFA and finally to DFA.
 Con’t
• Case 1: For a regular expression ‘a’, we can
construct the following FA:
 Con’t
• Case 2: For a regular expression ‘ab’, we can construct the
following FA:

• Case 3: For a regular expression (a+b), we can construct the

following FA:
 Con’t
• Case 4: For a regular expression
(a+b)*, we can construct the following
FA:
 Con’t
Method:
• Step 1 Construct an NFA with Null moves
from the given regular expression.
• Step 2 Remove Null transition from the NFA
and convert it into its equivalent DFA.
 Con’t
Problem
Convert the following RA into its equivalent DFA: 1 (0 + 1)* 0
Solution:
We will concatenate three expressions "1", "(0 + 1)*" and "0"
 Con’t
• Now we will remove the є transitions. After we remove the є
transitions from the NDFA, we get the following:

• If you want to convert it into a DFA, simply apply the method

of converting NDFA to DFA discussed in Chapter 2.
 Pumping Lemma for Regular
Languages
Theorem
Let L be a regular language. Then there exists a
constant ‘c’ such that for every string w in L:
|w| ≥ c
We can break w into three strings, w = xyz, such
that:
1. |y| > 0
2. |xy| ≤ c
3. For all k ≥ 0, the string xykz is also in L.
 Con’t
Applications of Pumping Lemma
Pumping Lemma is to be applied to show that
certain languages are not regular. It should never
be used to show a language is regular.
1. If L is regular, it satisfies Pumping Lemma.
2.If L is non-regular, it does not satisfy Pumping
Lemma.
 Con’t
Method to prove that a language L is not regular:
1. At first, we have to assume that L is regular.
2. So, the pumping lemma should hold for L.
3. Use the pumping lemma to obtain a contradiction:
(a) Select w such that |w| ≥ c
(b) Select y such that |y| ≥ 1
(c) Select x such that |xy| ≤ c
(d) Assign the remaining string to z.
(e) Select k such that the resulting string is not in L.
 Con’t
Hence L is not regular.
Problem
Prove that L = {aibi | i ≥ 0} is not regular.
Solution:
1. At first, we sassume that L is regular and n is the number of states.
2. Let w = anbn. Thus |w| = 2n ≥ n.
3. By pumping lemma, let w = xyz, where |xy|≤ n.
4. Let x = ap, y = aq, and z = arbn, where p + q + r = n.p ≠ 0, q ≠ 0, r ≠
0. Thus |y|≠ 0
5. Let k = 2. Then xy2z = apa2qarbn.
6. Number of as = (p + 2q + r) = (p + q + r) + q = n + q
7. Hence, xy2z = an+q bn. Since q ≠ 0, xy2z is not of the form anbn.
8. Thus, xy2z is not in L. Hence L is not regular.
 Exercise
1. Find all strings in L((a + b) b (a + ab)*) of length less than four

2. Find a regular expression for the set {anbm: n ≥ 3,m is even}.

3. Find a regular expression for
L = {w∈{0,1}* : w has exactly one pair of consecutive zeros}
4. Find an NFA that accepts the language L (aa* (a + b)).
5. Show that the language L={a2n : n>=0} over Ꜫ={a} is not
regular using pumping lemma
 Chapter Three
Context Free Grammer

Simachew A FLTA 1
Chomesky language hierarchy

According to Chomsky hierarchy, grammars is divided into 4 types:

Type 0 known as unrestricted grammar.
Type 1 known as context sensitive grammar.
Type 2 known as context free grammar.
Type 3 Regular Grammar.
Type 0: Unrestricted Grammar: The productions can be in the form
of α → β where α is a string of terminals and nonterminals with at least
one non-terminal and α cannot be null. β is a string of terminals and
non-terminals.
Example
S → ACaB
Bc → acB
CB → DB
aD → Db

Simachew A FLTA 2
Chomesky language hierarchy

Type - 1 Grammar
Type-1 grammars generate context-sensitive languages. The
productions must be in the form
αAβ→αγβ
where A ∈ N (Non-terminal)
and α, β, γ ∈ (T ∪ N)* (Strings of terminals and non-terminals)
The strings α and β may be empty, but γ must be non-empty.
The rule S → ε is allowed if S does not appear on the right side of any
rule. The languages generated by these grammars are recognized by a
linear bounded automaton.
AB → AbBc
A → bcA
B→b

Simachew A FLTA 3
Chomesky language hierarchy

Type - 2 Grammar
Type-2 grammars generate context-free languages.
The productions must be in the form A → γ
where A ∈ N (Non terminal)
and γ ∈ (T ∪ N)* (String of terminals and non-terminals).
These languages generated by these grammars are be recognized by a
non-deterministic pushdown automaton
S→Xa
X→a
X → aX
X → abc
X→ε

Simachew A FLTA 4
Chomesky language hierarchy

Type - 3 Grammar
Type-3 grammars generate regular languages. Type-3 grammars
must have a single non-terminal on the left-hand side and a right-hand
side consisting of a single terminal or single terminal followed by a
single non-terminal.
The productions must be in the form X → a or X → aY
where X, Y ∈ N (Non terminal)
and a ∈ T (Terminal)
The rule S → ε is allowed if S does not appear on the right side of any
rule.
Example
X→ε
X → a | aY
Y→b

Simachew A FLTA 5
Chomesky language hierarchy

Simachew A FLTA 6
Context Free Langauges

Definition
A grammar G = (V, T , S, P) is said to be context-free if all
productions of p are in the form

A → x,

wehre A ∈ V and x ∈ (v ∪ T ) ∗ .

A language L is said to be context-free iff there is a context free

grammar G such that L = L(G)

Every regular grammar is contex-free, so regular language is

also a context free language.

Simachew A FLTA 7
Context Free Language

Example: The grammar G = ({S}, {a, b}, S, P) with produc-

tions:
S → aSa,
S → bSb,
S→λ
L(G) = { ww R : w ∈ { a, b} ∗ }

Simachew A FLTA 8
Leftmost and Rightmost Derivations

In context-free grammars that are not linear, a derivation may

involve sentential forms with more than one variable.
Definition
A derivation is said to be leftmost if in each step the leftmost
variable in the sentential form is replaced. If in each step the
rightmost variable is replaced, we call the derivation rightmost.

Example Consider the grammar with productions

S → aAB,

A → bBB,
B → A|λ

Simachew A FLTA 9
Leftmost and Rightmost Derivations

Then
S ⇒ aAB ⇒ abBbB ⇒ abAbB ⇒ abbBbbB ⇒ abbbbB ⇒
abbbb .. leftmost derivation
S ⇒ aAB ⇒ aA ⇒ abBb ⇒ abAb ⇒ abbBbb ⇒ abbbb ...
rightmost derivation

Simachew A FLTA 10
Derivation Trees
A second way of showing a derivation, independent of the order
in which productions are used, is by derivation tree.
A derivation tree is an ordered tree in which nodes are labeled
with the left sides of productions and in which the children of a
node represent its corresponding right sides.
A → abABc.

Simachew A FLTA 11
Ordered Tree

Definition
Let G = (V, T , S, P) be a context-free grammar. An order tree
is a derivation tree for G iff it has the following properties.
The root is labeled S
Every leaf has a label from T ∪ { λ}
Every interior vertex has a label from V.
If a vetex has label A ∈ V , and its children are labeled
a1, a2, ..., an them P must contain a production of the form

A → a1a2...an

A leas labeled λ has no siblings, that is, a vertex which a

child labeled λ can have no other children

Simachew A FLTA 12
Dervation Tree

A tree that has properties:

Every leaf has a label from V ∪ T ∪ { λ } is said to be a partial
derivation tree.

The string of symbol obtained by reading the leaves of the tree

from left to right, omitting λ J ... called yield of the tree.

Example: Consider the grammar G, with productions

S → aAB,

A → bBb
B → A|λ

Simachew A FLTA 13
Derivation Tree

Figure: Partial tree

Figure: Der ivation

Simachew A FLTA
tree 14
Exercise

Find context-free grammars for the following language with n ≥

0, m ≥ 0
L = { w ∈ { a, b} ∗ : na (w ) /= nb (w )}

Consider the grammar with productions

S → aaB

A → bBb|λ
B → Aa
Show that the string aabbabba is not in the language generated
by this grammar.

Simachew A FLTA 15
Parsing and Ambiguity

Parsing describes a sequence of productions by which a w ∈

L(G) is derived.

Example: Consider the grammar

S → SS|aSb|bSa|λ

and the string w = aabb

Round 1.
S ⇒ SS
S ⇒ aSb
S ⇒ bSa
S⇒λ

Round 2. ...
Simachew A FLTA 16
Simple Grammar
Definition
A context-free grammar G = (V, T , S, P) is said to be a simple
grammar or s-grammar if all its productions are of the form

A → ax,

Where A ∈ V, a ∈ T , x ∈ V ∗ , and any pair (A, a) occurs at most

once in P.

Example: The grammar

S → aS|bSS|c
is an s-grammar while the grammar
S → aS|bSS|aSS|c
is not an s-grammar because the pair (S,a) occurs in the two
productions S → aS and S → aSS.
Simachew A FLTA 17
Ambiguity in Grammars and Languages

if a number of different derivation trees exist for w ∈ L(G), the

situation is referred to as ambiguity.

Definition
A context-free grammar G is said to be ambiguous if there
exists some w ∈ L(G) that has at least two distinct derivation
trees. Alternatively, ambiguity implies the existence of two or
more leftmost or rightmost derivations.

Example: The grammar with productions

S → aSb|SS|λ

is ambiguous since the sentence aabb has two derivation trees.

Simachew A FLTA 18
Ambiguity in Grammars and Languages

Simachew A FLTA 19
Context Free Grammar and Programming Language

One of the most important uses of the theory of formal language

is int he definition of programming languages and in the con-
struction of interpreters and compilers for them.

Backus-Naur form is used for writing on programming languages

by specifying grammars.

Example: the grammar

E → T |E + T ,

F → F |T ∗F ,
F → I|(E ),
I → a|b|c.

Simachew A FLTA 20
Context Free Grammar and Programming Language

The symbols + and * are terminals. The symbol | is used as an

alternator, while := used instead of →

Many part of pascal-like programming language are susceptible

to definition by restricted form of context-free grammars.

Aspect of programming language which can be modelled by a

context-free grammar are usually referred to as its syntax.

Simachew A FLTA 21
Exercise

Simachew A FLTA 22
Simplification of Context Free Grammars

Empty string plays singular role in many theorems, i. e., to give

special attention.

There is no difference between context free language that in-

clude λ and those that do not.

Our focus will be only on λ − free languages, L(G) − { λ }

simplification is the removal of certain types of undesirable pro-

ductions that does not necessarily result in an actual reduction
of the number of rules.

Simachew A FLTA 23
A Useful Substitution Rule

A production A → x1Bx2 can be eliminated from a grammar if B

is replaced by a set of productions in which all strings is derived
in one step.

Example: Consider G = ({A, B}, {a, b, c}, A, P) with productions

A → a|aaA|abBc,

B → abbA|b.

Using the suggested substitution for B, we get a grammar with

productions
A → a|aaA|ababbAc|abbc,
B → abbA|b.

Simachew A FLTA 24
A Useful Substitution Rule

The new grammar Gˆ is equivalent to G, for instance, the string

aaabbc has the derivation

A ⇒ aaA ⇒ aaabBc ⇒ aaabbc

in G, and the corresponding derivation

A ⇒ aaA ⇒ aaabbc

in Gˆ

Note that the variable B is no longer play a part in any derivation.

Simachew A FLTA 25
Removing Useless Productions

Productions that never take part in the grammar have to be re-

moved.

Example:
S → aSb|λ|A,
A → aA

The production S → A plays no role. A cannot transformed into

a terminal string. Removing this production does not affect the
language.

Simachew A FLTA 26
Removing Useless Productions
Definition
Let G = (V, T , S, P) be a context-free grammar. A variable
A ∈ V is said to be useful if and only if there exist at least one
w ∈ L(G) such that
S →−∗ xAy →−∗ w,
with x, y ∈ (v ∪ T ) ∗

A variable is useful if and only if it occurs in at least one deriva-

tion and a production is useless if it involves any useless vari-
able.

Example:
S → A,
A → aA|λ
B → bA.
Simachew A FLTA 27
Removing Useless Productions

Example:
S → A,
A → aA|λ
B → bA.

the variable B derives a terminal string but not reachable from

S, thus, it is useless.

Two reasons why a variable is useless:

not reachable from the start symbol.
cannot derive a terminal string.

Simachew A FLTA 28
Removing Useless Productions

Example: Eliminate useless symbols and productions from G =

(V, T , S, P) where V = {S, A, B, C } and T = {a, b}, with P con-
sisting of
A → aS|A|C
A → a,
B → aa,
C → aCb,
First identify terminating strings, A → a and B → aa, A and B
belong to this set while C is useless. Removing C led to the
grammar G1 with variable V1 = {S, A, B} terminals T = {a},
production
S → aS|A
A → a,
B → aa.
Simachew A FLTA 29
Removing Useless Productions
Then eliminate non-readable variables by drawing dependency
graph. The dependency graph has its vertices labeled with vari-
ables, with an edge between vertices C and D iff there is a pro-
duction form
C → xDy

a variable is useful if there is a path from the vertex labeled by

the initial variable to that variable.
B is useless, removing B we led to Gˆ = (Vˆ, Tˆ,S, Pˆ)with Vˆ =
{ S, A} , T̂ = { a} and productions
S → aS|A
A → a.
Simachew A FLTA 30
Removing λ − Productions

Definition
Any productions of a contex-free grammar of the form

A→λ

is called a λ-production. Any variable A for which the derivation

∗
A→
− λ

is called nullable.

Example: Consider the grammar

S → aS1b,

S1 → aS1b|λ

Simachew A FLTA 31
Removing λ − Productions

This grammar generates λ-free language {a n b n : n ≥ 1}, thus,

S1 → λ

can be removed by adding new productions by substituting λ for

S1
S → aS1b|bb,
S1 → aS1b|bb.

Theorem 6.3: Let G be any context-free grammar with λ not in

L(G). Then there exists an equivalent grammar Gˆ having no λ-
productions.

Simachew A FLTA 32
Removing λ − Productions

First find the set VN of nullable variables of G, using the following

steps.
For all productions A → λ, put A into VN
Repeat the following step until no further variables are
added to VN . For all productions

B → A1A2...An.

where A1, A2 , ..., An are in VN , put B into VN

Once the set VN has been found, we are ready to construct
P̂. To do so, we look at all productions in P of the form

A → x1x2 ...xm , m ≥ 1

Simachew A FLTA 33
Removing λ − Productions

Example: Find a context-free grammar without λ-productions

equivalent to the grammar defined by
S → ABaC,

A → BC,
B → b|λ
C → D|λ
D → d.

the nullable variables are A, B, C.

Simachew A FLTA 34
Removing λ − Productions

Then following the second step we get

S → ABaC|BaC|AaC|ABa|aC|Aa|Ba|a,

A → B|C|BC,
B → b,
C → D,
D → d.

Simachew A FLTA 35
Removing Unit-Productions

Definition
Any productions of a contex-free grammar of the form

A→B

where A, B ∈ V is called a unit-production.

Simachew A FLTA 36
 Bonga University
College of Enginering and Technology
Department of Computer Science
CoSc3011 – Automata and complexity Theory
Chapter 4 – Push Down Automata (PDA

Pushdown Automata (PDA)

• Informally:
– A PDA is an NFA-ε with a stack.
– Transitions are modified to accommodate stack operations.
• Questions:
– What is a stack?
– How does a stack help?

A DFA can “remember” only a finite amount of information, whereas a PDA can “remember” an
infinite amount of (certain types of) information, in one memory-stack.

 Just as a DFA is a way to implement a regular expression, a pushdown automata is a way

to implement a context free grammar
PDA equivalent in power to a CFG

Can choose the representation most useful to our particular problem

 Essentially identical to a regular automata except for the addition of a stack
Stack is of infinite size
Stack allows us to recognize some of the non-regular languages
Can visualize a PDA with the schematic

1
The tape is divided into finitely many cells. Each cell contains a symbol in an alphabet Σ.

The stack head always scans the top symbol of the stack. It performs two basic operations:

Push: add a new symbol at the top.

Pop: read and remove the top symbol.

Alphabet of stack symbols: Γ

The head scans at a cell on the tape and can read a symbol on the cell. In each move, the head
can move to the right cell.

The finite control has finitely many states which form a set Q. For each move, the state is
changed according to the evaluation of a transition function

δ : Q x (Σ U {ε}) x (Γ U {ε}) → 2

2
• (p, u)  δ(q, a, v) means that if the tape head reads a, the stack head read v, and the
finite control is in the state q, then one of possible moves is that the next state is p, v is
replaced by u at stack, and the tape head moves one cell to the right.
• (p, u)  δ(q, ε, v) means that this a ε-move.
• (p, u)  δ(q, a, ε) means that a push operation performs at stack.
• (p, ε)  δ(q, a, v) means that a pop operation performs at stack
• There are some special states: an initial state s and a final set F of final states.
• Initially, the PDA is in the initial state s and the head scans the leftmost cell. The tape
holds an input string. The stack is empty.
• When the head gets off the tape, the PDA stops. An input string x is accepted by the
PDA, when All the input is consumed AND the last state is an accepting state. we do not
care about the stack contents at the end of the accepting computation.
• If the automaton attempts to pop from empty stack then it halts and rejects input.

The States

3
The PDA can be represented by

Where Σ is the alphabet of input symbols and Γ is the alphabet of stack symbols.

• The set of all strings accepted by a PDA M is denoted by L(M). We also say that the
language L(M) is accepted by M.

Types of PDA
1. NDPDA
2. DPDA

Deterministic PDA: DPDA

Allowed transitions:

4
 (Deterministic choices)

Not allowed:

(Non deterministic choices)

DPDA example

5
Example of Non-DPDA

Equivalence of PDA and CFG

• A context-free grammar and pushdown automata are equivalent in power.

• Theorem: Given a CFG grammar G, then some pushdown automata P recognizes L(G).

– To prove this, we must show that we can take any CFG and express it as a PDA.
Then we must take a PDA and show we can construct an equivalent CFG.
– We’ll show the CFGPDA process, but skip the PDACFG.

Proof for CFG to PDA

• Proof idea:
– The PDA P will work by accepting its input w, if G generates that input, by
determining whether there is a derivation for w.
– Design rules for P such that the transitions match the production rules in the
grammar
• But the PDA can access only the top symbol on the stack and that might
be a terminal
– But if our PDA makes transitions only for variables, we we won’t
know to do
• To get around this problem, we’ll use the non-determinism of the PDA to
match terminal symbols on the stack with symbols in the input string
before the first variable
– This “chomps” any leading terminals until we can process a
variable

6
 Example
• Consider the grammar
S aTb | b Start qstart
T  Ta | ε
ε, ε  S
ε, SaTb
ε, TTa
Given the string “aab” derived by qloop ε, Sb
S  aTb  aTab  aab ε, Tε
We have the corresponding moves: a, aε
(qs, aab, $) ├ (qloop, aab, S$) ├ b, bε
ε, $ε
(qloop, aab, aTb$) ├ (qloop, ab, Tb$) ├
(qloop, ab, Tab$) ├ (qloop, ab, ab$) ├
qaccept
(qloop, b, b$) ├ (qloop, ε, $) ├
(qaccept, ε,ε)

7
 • A DPDA is simply a pushdown automaton without non-determinism.

– i.e. no epsilon transitions or transitions to multiple states on same input

– Only one state at a time

• DPDA not as powerful as non-deterministic PDA

– This machine accepts a class of languages somewhere between regular languages and
context-free languages.
– For this reason, the DPDA is often skipped as a topic
– In practice the DPDA can be useful since determinism is much easier to implement.
• Parsers in programs such as YACC are actually implemented using a DPDA.

Finite automata and PDA schematics

Differences between PDA’s and NFA’s

• Transitions read a symbol of the string and push a symbol onto or pop a symbol off of the stack
• Stack alphabet is not necessarily the same as the alphabet for the language
– e.g., $ marks bottom of stack in previous (0n1n) example

8
Simplification of CFG

 Remove Useless Variables

Another grammar:

Not reachable from S

Remove Nullable Variables

Example Grammar:

9
Unit-Productions

Observation:

Example Grammar:

S  aA
A  a
A  B
B  A
B  bb

10
Normal forms

 Every context-free grammar can be transformed into an equivalent one in Chomsky

normal form or Greibach normal form. "Equivalent" here means that the two grammars
generate the same language.
 Because of the especially simple form of production rules in Chomsky Normal Form
grammars, this normal form has both theoretical and practical implications.
 For instance, given a context-free grammar, one can use the Chomsky Normal Form to
construct a polynomial-time algorithm which decides whether a given string is in the
language represented by that grammar or not (the CYK algorithm).
Chomsky Normal Form (CNF)

• A Grammar G is in CNF if all of its productions are of the form:

– A  BC or
– A  a or
– S  ,

where A, B, C  V, aT and S is the starting symbol

• It has been proven that all CFG could be convert into CNF

Steps

• Step 1: get rid of useless symbols and remove all -productions and unit productions
– Example:
S  ASB | AB

11
 A  aAS | aA | a
B  SbS | bS | Sb | b | aAS | aA | a | bb

• Step 2: get rid of production whose bodies are mixes of terminals and variables, or consist of
more than one terminal
S  ASB | AB
A  CAS | CA | a
B  SDS | DS | SD | b | CAS | CA | a | DD
Ca
Db

• Step 3: break up production bodies longer than 2

S  AE | AB
A  CF | CA | a
B  SG | DS | SD | b | CF | CA | a | DD
Ca
Db
E  SB
F  AS
G  DS

A CFG is in Chomsky Normal Form if all its productions are of the form:

A  BC or
Aa

where A, B, C  V and a  T. Also, S   may be one of the productions.

Examples of CNF

Example 1: S  AB
A  BC | CC | a
B  CB | b
Cc

12
Example 2: S  AB | BC | AC | 
A  BC | a
B  AC | b
C  AB | c

Is that all Contexts Free Grammars can be expressed in Chomsky Normal Form?

Consider the following simple grammar:

A  cA | a
B  ABC | b
Cc

How to convert this grammar to CNF?

Convert the following CFG into Chomsky Normal Form:

S
S  ABBA
B  bCb
Aa
Cc
Greibach Normal Form (GN F)

A CFG is in Greibach normal form if each rule has one these forms:

i. A  aA1A2…An
ii. Aa
iii. S 

where a   and Ai  V – {S} for i = 1, 2,…, n

Applications of CFLs & CFGs

 Compilers use parsers for syntactic checking

 Parsers can be expressed as CFGs
1. Balancing paranthesis:
 B ==> BB | (B) | Statement
 Statement ==> …
2. If-then-else:
 S ==> SS | if Condition then Statement else Statement | if Condition then
Statement | Statement
 Condition ==> …
 Statement ==> …
3. C paranthesis matching { … }
4. Pascal begin-end matching
5. YACC (Yet Another Compiler-Compiler)

13
CFG conventions

 Terminal symbols <== a, b, c…+,-,*,(,…,0,1,2,….

 Non-terminal symbols <== A,B,C, …
 Terminal or non-terminal symbols <== X,Y,Z
 Terminal strings <== w, x, y, z
 Arbitrary strings of terminals and non-terminals <== , , , ..
Example

Work out:

1. S  S1 | S2
S1  S1b | Ab
A  aAb | ab | ε
S2  S2a | Ba
B  bBa | ba| ε

14
 Bonga University
College of Enginering and Technology
Department of Computer Science
CoSc3101 – Automata and complexity Theory
Chapter 5 – Turing Machines

Introduction

Background

• Automata accepts regular languages only

For example, {anbn : n = 0, 1, …} is not regular

• Pushdown Automata accepts context-free languages only

For example, {anbncn : w  *} is not context-free

• We can easily write an algorithm (in C) recognizing if a sequence of characters have the
form anbncn or not.

Context Sensitive Grammar

Def: A type 1 grammar G= (V,, S,P) is context-sensitive if for every production rule    in P,
||≤||.

Lemma: Every context-free language which does not contain ɛ is context-sensitive.

Lemma: A language L is context-sensitive iff there exists a linear-bounded automaton accepting
L.

The language {a n b n c n } is context-sensitive:

S  abc | aAbc
Ab  bA
Ac  Bbcc
bB  Bb
aB  aa | aaA

Theorem:
A language is context sensitive if and only if is accepted by a Linear-Bounded Automaton

Page 1
Linear Bounded Automaton (LBA)

LBA’s have more power than NPDA’s

LBA’s have also less power than Turing Machines

Linear Bounded Automata (LBAs) are the same as Turing Machines with one difference:

The input string tape space is the only tape space allowed to use

Grammars Languages & Accepting Machines

Grammars Languages Accepting Machines

Type 3 grammars: Regular languages DFA, NDFA

Regular grammars,

Type 2 grammars: Context-free PDA

Context-free grammars languages

Type 1 grammars: Contest-sensitive Linear-bounded

Context-sensitive grammars, languages Automata

Page 2
Type 0 grammars: Recursively DTM
Phrase-structure grammars, enumerable, NDTM
Unrestricted grammars Unrestricted

The Key Question

• Is it possible to design a formal model of a computational device that captures the

capabilities of any algorithm?

Alan Turing, 1940’s: Yes!

Turing Machine

Turing Machine is COOOOOL, why?

Understanding it makes us more like CS guys
But, probably there are few graduate students who really understand it.
Because, most of us forgot the details very soon after we learnt it. 
Anyway, it is cool, instinctively.

Why it is cool, seriously

Anything a real computer can compute, a TM can also compute.
Despite its simplicity, TM can be adapted to simulate the logic of any
computer that could possibly be constructed.
Therefore, TM is the foundation of computational complexity theory.
A basis to analyze algorithms

Alan Turing 1912-1954

Great mathematician, the father of the modern computer

Basic Idea

Turing decomposed operations in the device as follows:

• A pencil will be “observing” symbols from a paper

• The pencil can move left and right
• Computational steps erase a symbol observed by the pencil and write a new one in its
place
• The decision about which symbol should be written and which will be observed next
depend on:
1. The current symbol being observed
2. The “state of mind” of the device

This device is called a Turing Machine

Page 3
Church-Turing Thesis

Every computer algorithm can be implemented as a Turing machine

Therefore, C, C++, Prolog, Lisp, Small talk, and Java programs can be simulated in Turing
machines

Definition: a programming language is Turing-complete if it is equivalent to a Turing machine.

The Language Hierarchy

Actions of a Turing Machine

Page 4
Structure of Turing Machine

Page 5
States & Transitions

Initially, the input string (finite-length string of symbols) is placed on the tape
All other tape cells, extending infinitely to left and right, hold blanks
Blank is a tape symbol, but not an input symbol
Initially, tape head points to the beginning of the input string
: the transition function
(q,X): a state q and a tape symbol X
(q,X) = (p,Y,D) where:
p is next state in Q
Y is the symbol written in the cell being scanned
D is a direction, either L or R
Tape head: always positioned at one of tape cells
A move (or say ‘a step’) may:
Read an input symbol
Change machine state
Write a tape symbol in the cell scanned
Move the tape head left or right
So simple, right? But…
The computational capabilities of all other known computational models (e.g.
any automata) are less than or equivalent to TM
Their speeds may not be as same as that of the TM’s
Their computational capabilities are less than or equivalent to TM, i.e., no
‘more’ mathematical functions can be calculated

Page 6
 • If a language L is accepted by a Turing machine M then we say that L is:
Turing Recognizable

Other names used:

• Turing Acceptable
• Recursively Enumerable

Deterministic Turing Machine (DTM)

No epsilon transitions allowed

Page 7
Example 1

Accepts the language: a*

Page 8
Infinite Loop Example

Page 9
Because of the infinite loop:

• The accepting state cannot be reached

• The machine never halts

Page 10
The input string is rejected

Example 2

Turing machine for the language{a n b n } n>=1

Basic Idea:
Match a’s with b’s:
Repeat:
replace leftmost a with x
find leftmost b and replace it with y
Until there are no more a’s or b’s
If there is a remaining a or b reject

Example 3 Balanced Brackets

For a TM for balanced brackets, one idea is to find the innermost matching pair of brackets, erase them,
and repeat the process. We use x to indicate an erased bracket.

Page 11
Example 4: Even-length Palindromes

For even-length palindromes, we match first and last symbols and erase; then repeat. If reach ɛ
without mismatch, then string was palindrome

Summary

A Turing Machine (TM) is like an FA, but it has an infinite tape. The input starts on the tape
surrounded by blank cells denoted ∆. The programof a TM is represented as a diagram:
depending on the symbol under the head and the state, the machine writes a symbol, moves
left or right or stays in place, and/or changes state.Once a TM enters the accept state it stops.

Page 12
Chapter 6

2015 1
· Decision Problem: computational problem
with intended output of “yes” or “no”, 1 or 0
· Optimization Problem: computational
problem where we try to maximize or
minimize some value
· Introduce parameter k and ask if the optimal
value for the problem is a most or at least k.
Turn optimization into decision

2015 2
· Deterministic in nature
· Solved by conventional computers in
polynomial time
◦ O(1) Constant
◦ O(log n) Sub-linear
◦ O(n) Linear
◦ O(n log n) Nearly Linear
◦ O(n2) Quadratic
· Polynomial upper and lower bounds

2015 3
· Some problems are intractable:
as they grow large, we are unable to solve them
in reasonable time
· What constitutes reasonable time?
◦ Standard working definition: polynomial time
◦ On an input of size n the worst-case running time is
O(nk) for some constant k
◦ O(n2), O(n3), O(1), O(n lg n), O(2n), O(nn), O(n!)
◦ Polynomial time: O(n2), O(n3), O(1), O(n lg n)
◦ Not in polynomial time: O(2n), O(nn), O(n!)

2015 4
· Are some problems solvable in polynomial time?
◦ Of course: many algorithms we’ve studied provide
polynomial-time solutions to some problems
· Are all problems solvable in polynomial time?
◦ No: Turing’s “Halting Problem” is not solvable by any
computer, no matter how much time is given
· Most problems that do not yield polynomial-
time algorithms are either optimization or
decision problems.

2015 5
· Optimization Problems
◦ An optimization problem is one which asks, “What is the
optimal solution to problem X?”
◦ Examples:
🞄0-1 Knapsack
🞄Fractional Knapsack
🞄Minimum Spanning Tree
· Decision Problems
◦ An decision problem is one with yes/no answer
◦ Examples:
🞄Does a graph G have a MST of weight  W?

2015 6
· An optimization problem tries to find an optimal
solution
· A decision problem tries to answer a yes/no question
· Many problems will have decision and optimization
versions
◦ Eg: Traveling salesman problem
🞄optimization: find hamiltonian cycle of minimum weight
🞄decision: is there a hamiltonian cycle of weight  k
· Some problems are decidable, but intractable:
as they grow large, we are unable to solve them in
reasonable time
◦ Is there a polynomial-time algorithm that solves the problem?

2015 7
P: the class of decision problems that have
polynomial-time deterministic algorithms.
◦ That is, they are solvable in O(p(n)), where p(n) is a
polynomial on n
◦ A deterministic algorithm is (essentially) one that always
computes the correct answer

Why polynomial?
◦ if not, very inefficient
◦ nice closure properties
🞄 the sum and composition of two polynomials are always polynomials
too

2015 8
· MST
· Sorting
· Others?

2015 9
NP: the class of decision problems that are solvable
in polynomial time on a nondeterministic machine
(or with a nondeterministic algorithm)
· (A determinstic computer is what we know)
· A nondeterministic computer is one that can “guess”
the right answer or solution
◦ Think of a nondeterministic computer as a parallel machine
that can freely spawn an infinite number of processes
· Thus NP can also be thought of as the class of
problems
◦ whose solutions can be verified in polynomial time
· Note that NP stands for “Nondeterministic
Polynomial-time”

2015 10
· MST
· Sorting
· Others?
◦ Hamiltonian Cycle (Traveling Salesman)
◦ Graph Coloring
◦ Satisfiability (SAT)
🞄the problem of deciding whether a given
Boolean formula is satisfiable

2015 11
 i = 23

· For each two cities, an integer cost is given to

travel from one of the two cities to the other. The
salesperson wants to make a minimum cost circuit
visiting each city exactly once.
2015 12
 Logic Gates 0 1 0
1 1
NOT
1 1
0
OR 1
0 0 1
1

AND

· Take a Boolean circuit with a single output

node and ask whether there is an
assignment of values to the circuit’s inputs
so that the output is “1”
2015 13
 5 6
2
3
1 4 7

L
s

· Given s and w can we translate a subset of

rectangles to have their bottom edges on L so
that the total area of the rectangles touching
L is at least w?

2015 14
· Polynomial-Time Approximation Schemes
· Much faster, but not guaranteed to find the
best solution
· Come as close to the optimum value as
possible in a reasonable amount of time
· Take advantage of rescalability property of
some hard problems

2015 15
· P = set of problems that can be solved in
polynomial time
◦ Examples: Fractional Knapsack, …
· NP = set of problems for which a solution can be
verified in polynomial time
◦ Examples: Fractional Knapsack,…, Hamiltonian Cycle, CNF
SAT, 3-CNF SAT
· Clearly P  NP
· Open question: Does P = NP?
◦ Most suspect not
◦ An August 2010 claim of proof that P ≠ NP, by Vinay
Deolalikar, researcher at HP Labs, Palo Alto, has flaws

2015 16
· What if a problem has:
◦ An exponential upper bound
◦ A polynomial lower bound
· We have only found exponential algorithms,
so it appears to be intractable.
· But... we can ’ t prove that an exponential
solution is needed, we can ’ t prove that a
polynomial algorithm cannot be developed,
so we can’t say the problem is intractable...

2015 17
· The upper bound suggests the
problem is intractable
· The lower bound suggests the
problem is tractable
· The lower bound is linear: O(N)
· They are all reducible to each other
◦ If we find a reasonable algorithm
(or prove intractability) for one,
then we can do it for all of them!

2015 18
· With N teachers with certain hour
restrictions M classes to be scheduled,
can we:
◦ Schedule all the clas ses
◦ Make sure that no two teachers teach
the s am e clas s at the s am e time
◦ No teacher is scheduled to teach two
classes at once

2015 19
· With N students and K projects, where N
is even, can we:
◦ Assign pairs of students to each project
◦ Every s tudent works on every project
◦ No student has the same partner more
than once
· Is this an NP-complete
problem?

2015 20
· Graph isomorphism is NP-hard; is it NP-
complete?

2015 21
· What is N P?
· NP is the set of all decision problems (question
with yes-or-no answer) for which the 'yes'-answers
can be verified in polynomial time (O(n^k) where n
is the problem size, and k is a constant) by a
deterministic Turing machine. Polynomial time is
sometimes used as the definition of fast or quickly.
· What is P?
· P is the set of all decision problems which can be
solved in polynomial time by a deterministic Turing
machine. Since it can solve in polynomial time, it
can also be verified in polynomial time. Therefore P
is a subset of NP.

2015 22
· What is NP-Complete?
· A problem x that is in NP is also in NP-
Complete if and only if every other problem in
NP can be quickly (ie. in polynomial time)
transformed into x. In other words:
· x is in NP, and
· Every problem in NP is reducible to x
· So what makes NP-Complete so interesting is
that if any one of the NP-Complete problems
was to be solved quickly then all NP problems
can be solved quickly

2015 23
· What is NP-Hard?
· NP-Hard are problems that are at least as hard as
the hardest problems in NP. Note that NP-
Complete problems are also NP-hard. However not
all NP-hard problems are NP (or even a decision
problem), despite having 'NP' as a prefix. That is
the NP in NP-hard does not mean 'non-
deterministic polynomial time'. Yes this is
confusing but its usage is entrenched and unlikely
to change.

2015 24
· Nondeterministic algorithms produce an
answer by a series of “correct guesses”

· Deterministic algorithms (like those that a

computer executes) make decisions based
on information.

2015 25
“NP-Complete” comes from:
◦ Nondeterministic Polynomial
◦ Complete - “Solve one, Solve them all”

There are more NP-Complete problems than

provably intractable problems.

2015 26
· Show that the problem is in N P. (i.e. Show that a
certificate can be verified in polynomial time.)
· As sume it is not N P complete
· Show how to convert an existing NPC problem into
the problem that we are trying to show is NP
C omplete (in polynomial time).

· If we can do it we’ ve done the proof!

· Why?
· If we can turn an exisiting NP-complete problem into
our problem in polynomial time... 

2015 27
· Poly time algorithm: input size n (in some
encoding), worst case running time – O(nc) for
some constant c.
· Three classes of problems
◦ P: problems solvable in poly time.
◦ NP: problems verifiable in poly time.
◦ NPC: problems in NP and as hard as any problem in
NP.

2015 28
· If a problem is proved to be NPC, a good
evidence for its intractability (hardness).
· Not waste time on trying to find efficient
algorithm for it
· Instead, focus on design approximate
algorithm or a solution for a special case of
the problem
· Some problems looks very easy on the
surface, but in fact, is hard (NPC).

2015 29
· A decision problem D is NP-complete iff
1. D  NP
2. every problem in NP is polynomial-time reducible
to D

· Cook’s theorem (1971): CNF-sat is NP-

complete

2015 30
· A problem R can be reduced to another problem Q
if any instance of R can be rephrased to an
instance of Q, the solution to which provides a
solution to the instance of R
◦ This rephrasing is called a transformation
· Intuitively: If R reduces in polynomial time to Q, R
is “no harder to solve” than Q
· Example: lcm(m, n) = m * n / gcd(m, n),
lcm(m,n) problem is reduced to gcd(m, n)
problem

2015 31
View of Theoretical Computer Scientists on P, NP, NPC

NP NPC

P  NP, NPC  NP, P  NPC = 

2015 32