Formal Language & Automata Theory

Module 1

Introduction:
Language is a fundamental aspect of human communication and is central to
our daily lives. It allows us to convey thoughts, emotions, and ideas. When
studying languages, linguists and computer scientists have developed various
frameworks and models to understand their structure and organization. In this
context, we will explore the concepts of alphabets, languages, grammars,
productions, derivations, and the Chomsky hierarchy of languages.

Alphabets:
An alphabet is a set of symbols or characters that form the building blocks of a
language. It typically consists of individual letters or symbols that represent
sounds or meaningful units. Alphabets can vary in size, ranging from a few
symbols in constructed languages to several thousand in complex writing
systems like Chinese or Japanese.

Languages and Grammars:

A language is a system of communication that enables individuals to express
their thoughts and ideas. It comprises a set of words, phrases, and sentences that
conform to specific rules and patterns. Grammars, in the context of languages,
are formal systems that describe the structure and rules of a particular language.
They define how sentences and phrases are formed, what words and structures
are allowed, and how they are combined.

Productions and Derivation:

In the study of grammars, productions are rules that specify how symbols in a
language can be transformed or combined to create larger linguistic structures.
Productions consist of a left-hand side (LHS) and a right-hand side (RHS). The
LHS represents a symbol that can be replaced or expanded using the RHS.
Derivation refers to the step-by-step application of productions to generate valid
sentences or structures in a language. It involves repeatedly replacing symbols
according to the rules of the grammar until a desired structure is achieved.

Chomsky Hierarchy of Languages:

The Chomsky hierarchy is a classification of formal languages, proposed by the
linguist Noam Chomsky in the 1950s. It categorizes languages into four levels
or types based on their generative power and the complexity of their grammars.
The four types, from least to most complex, are:

© SOUMYAJIT BAG
1. Type 3 (Regular Languages): Regular languages can be described by regular
grammars, which use simple productions and finite-state automata (such as
regular expressions) for their representation. Regular languages have finite
memory and can be recognized by deterministic or non-deterministic finite
automata.

2. Type 2 (Context-Free Languages): Context-free languages are defined by

context-free grammars, where productions allow for the replacement of non-
terminal symbols with any combination of symbols. Context-free languages can
be recognized by pushdown automata, which have a stack to store and retrieve
symbols.

3. Type 1 (Context-Sensitive Languages): Context-sensitive languages are

described by context-sensitive grammars, in which productions allow for more
complex transformations based on the context of symbols. The rules can modify
strings of symbols while considering the surrounding context. These languages
require more powerful computational models like linear-bounded automata.

4. Type 0 (Unrestricted Languages): Unrestricted languages, also known as

recursively enumerable languages, have the most expressive power. They can be
described by unrestricted grammars, which have no restrictions on the
production rules. These languages can be recognized by Turing machines,
which are theoretical computational devices with infinite memory.

Understanding the Chomsky hierarchy helps us categorize languages and

analyze their computational complexity. It provides insights into the different
levels of linguistic complexity and the computational models required to
generate or recognize those languages.

Module 2

Regular Languages and Finite Automata:

Regular Languages:
Regular languages are a type of language in the Chomsky hierarchy that can be
described by regular grammars and recognized by finite automata. These
languages have simple and regular patterns and can be efficiently processed by
computers. Regular languages are closed under various operations such as
union, concatenation, and Kleene closure.

Regular Expressions and Languages:

Regular expressions are formal notations used to describe patterns in strings.
They provide a concise and expressive way to represent regular languages.
Regular expressions consist of symbols from an alphabet, along with operators
such as concatenation, union, and closure. Examples of regular expressions

© SOUMYAJIT BAG
include "ab" (denotes the string "ab"), "a|b" (denotes either "a" or "b"), and "a*"
(denotes zero or more occurrences of "a").

Deterministic Finite Automata (DFA) and Equivalence with Regular

Expressions:
A deterministic finite automaton (DFA) is a computational model used to
recognize and accept strings that belong to a regular language. A DFA consists
of a finite set of states, an input alphabet, a transition function that determines
the next state based on the current state and input symbol, a start state, and a set
of accepting (or final) states. DFAs can be constructed directly from regular
expressions, and they recognize the same language as the regular expression.

Nondeterministic Finite Automata (NFA) and Equivalence with DFA:

A nondeterministic finite automaton (NFA) is another computational model
used to recognize regular languages. NFAs differ from DFAs in that they allow
multiple possible transitions for a given state and input symbol. NFAs are more
expressive but can be converted into equivalent DFAs using techniques like the
subset construction. NFAs and DFAs recognize the same class of regular
languages.

Regular Grammars and Equivalence with Finite Automata:

Regular grammars are a formalism used to describe regular languages. They
consist of production rules with a single non-terminal symbol on the left-hand
side and either a terminal symbol or a terminal symbol followed by a non-
terminal symbol on the right-hand side. Regular grammars can be converted
into equivalent finite automata, and vice versa, through well-defined algorithms.
The resulting automata and grammars recognize the same regular language.

Properties of Regular Languages:

Regular languages possess several important properties:

1. Closure under Union, Concatenation, and Kleene Closure: If L1 and L2 are

regular languages, then their union, concatenation, and Kleene closure
(repetitions) are also regular languages.

2. Closure under Intersection and Complement: Regular languages are closed

under intersection and complementation. If L1 and L2 are regular languages,
then their intersection and complement are also regular languages.

3. Closure under Homomorphism and Reversal: Regular languages are closed

under homomorphism (applying a function to each symbol in the language) and
reversal (reversing the order of symbols in each string).

Pumping Lemma for Regular Languages:

The pumping lemma for regular languages is a tool used to prove that a given
language is not regular. It states that if a language L is regular, then there exists
© SOUMYAJIT BAG
a pumping length (a positive integer) such that any string s in L, where |s| ≥ the
pumping length, can be divided into several parts (x, y, z) satisfying certain
conditions. By demonstrating a violation of these conditions, it can be shown
that the language is not regular.

Minimization of Finite Automata:

Minimization refers to reducing the number of states in a finite automaton while
preserving its language recognition capability. The process of minimizing a
DFA involves finding an equivalent DFA with the fewest possible states.
Techniques such as the table-filling algorithm or the Hopcroft's algorithm can
be used to minimize DFAs. The minimized automaton is unique up to
isomorphism and represents the same regular language as the original DFA.

Module 3

Context-Free Languages and Pushdown Automata:

Context-Free Grammars and Languages (CFG and CFL):

Context-free grammars (CFGs) are formal systems used to describe context-free
languages (CFLs), which are languages that can be generated by production
rules that replace non-terminal symbols with strings of terminals and/or non-
terminals. CFGs consist of a set of production rules, a start symbol, and a set of
terminal and non-terminal symbols. CFGs are more expressive than regular
grammars and can describe languages with nested structures.

Chomsky and Greibach Normal Forms:

Chomsky normal form and Greibach normal form are two standard forms for
CFGs that simplify their structure and aid in analysis and processing.

1. Chomsky Normal Form (CNF): In CNF, all production rules are in one of two
forms: either A -> BC (where A, B, and C are non-terminals) or A -> a (where A
is a non-terminal and a is a terminal symbol). CNF allows for easy parsing and
analysis of the language.

2. Greibach Normal Form (GNF): GNF is a more restricted form where the
right-hand side of each production rule consists of a single terminal symbol
followed by zero or more non-terminals. GNF is less commonly used than CNF
but still useful for certain parsing algorithms.

Nondeterministic Pushdown Automata (PDA) and Equivalence with CFG:

A pushdown automaton (PDA) is a computational model that extends the
capabilities of finite automata by incorporating a stack. PDAs can recognize
context-free languages. They have a finite set of states, an input alphabet, a
stack alphabet, a transition function that considers the current state, input
symbol, and top symbol of the stack, a start state, and a set of accepting states.

© SOUMYAJIT BAG
PDAs are more powerful than finite automata because they can track and
manipulate nested structures using the stack.

PDAs and CFGs are equivalent in terms of language recognition, meaning that
for every CFG, there exists an equivalent PDA and vice versa. The PDA can
simulate the derivation process of the CFG, using the stack to keep track of the
non-terminals being expanded.

Parse Trees:
Parse trees are hierarchical structures that represent the syntactic structure of a
sentence according to a CFG. They provide a graphical representation of how
the production rules of the CFG are applied to generate the sentence. In a parse
tree, the non-terminals are represented as internal nodes, and the terminals are
represented as leaves. Each node in the parse tree corresponds to a step in the
derivation process.

Ambiguity in CFG:
Ambiguity in a context-free grammar refers to situations where a given string
can have multiple valid parse trees or interpretations according to the grammar.
It means that the grammar allows for more than one derivation for a specific
sentence. Ambiguity can lead to difficulties in understanding and processing
languages and can be undesirable in certain applications.

Pumping Lemma for Context-Free Languages:

The pumping lemma for context-free languages is a property used to prove that
a given language is not context-free. Similar to the pumping lemma for regular
languages, it states that if a language L is context-free, then there exists a
pumping length (a positive integer) such that any string s in L, where |s| ≥ the
pumping length, can be divided into several parts (uvxyz) satisfying certain
conditions. By demonstrating a violation of these conditions, it can be shown
that the language is not context-free.

Deterministic Pushdown Automata:

Deterministic pushdown automata (DPDAs) are pushdown automata that have a
deterministic transition function. In DPDAs, for every state and input symbol,
there is at most one possible transition. DPDAs

can recognize a subset of context-free languages called deterministic context-

free languages (DCFLs). However, not all context-free languages can be
recognized by DPDAs.

Closure Properties of CFLs:

Context-free languages possess several important closure properties:

1. Union: If L1 and L2 are context-free languages, their union L1 ∪ L2 is also

context-free.
© SOUMYAJIT BAG
2. Concatenation: If L1 and L2 are context-free languages, their concatenation
L1 ∘ L2 is also context-free.

3. Kleene Closure: If L is a context-free language, its Kleene closure L*

(including all possible repetitions of strings in L) is also context-free.

4. Homomorphism and Reversal: Context-free languages are closed under

homomorphism (applying a function to each symbol in the language) and
reversal (reversing the order of symbols in each string).

Understanding the properties and limitations of context-free languages and their

recognition by pushdown automata helps in designing parsers and analyzing the
syntax of programming languages and natural languages.

Module 4

Context-Sensitive Languages:

Context-Sensitive Grammars and Languages (CSG):

Context-sensitive grammars (CSGs) are formal systems used to describe
context-sensitive languages (CSLs), which are languages that can be generated
by production rules that transform strings based on the surrounding context. In
CSGs, the left-hand side of the production rules can be a string of symbols,
rather than just a single non-terminal as in CFGs. CSGs are more expressive
than CFGs and can describe languages with complex structural constraints.

In a CSG, each production rule specifies a transformation of a substring of the

input based on the context of the surrounding symbols. The context is defined
by the presence of certain symbols or patterns in the input. The rules are applied
iteratively to generate valid strings in the language.

Linear Bounded Automata (LBA) and Equivalence with CSG:

Linear bounded automata (LBA) are computational devices that recognize
context-sensitive languages. LBAs are similar to Turing machines but with the
restriction that the tape head cannot move beyond the portion of the tape
containing the input string. This restriction ensures that the computation is
limited to a linear space, which is why LBAs are also known as linear space-
bounded Turing machines.

LBAs have a finite set of states, an input alphabet, a tape alphabet, a transition
function that considers the current state, the symbol under the tape head, and the
adjacent symbols, a start state, and a set of accepting states. The tape size of an
LBA is determined by the length of the input string. LBAs can recognize and
accept languages that can be generated by context-sensitive grammars.

© SOUMYAJIT BAG
The equivalence between CSGs and LBAs means that for every context-
sensitive grammar, there exists an equivalent LBA, and vice versa. The LBA
simulates the production and derivation process of the CSG by using the linear
tape to keep track of the symbols and apply the production rules based on the
context.

LBAs provide a computational model that captures the computational power

required to recognize and generate context-sensitive languages. They are more
powerful than pushdown automata (PDAs) and finite automata but less
powerful than Turing machines.

Understanding the properties and recognition capabilities of context-sensitive

languages and linear bounded automata is crucial in studying the computational
complexity of languages and designing advanced parsing and processing
algorithms for natural language understanding and programming languages.

Module 5

Turing Machines:

The Basic Model for Turing Machines (TM):

A Turing machine (TM) is a theoretical computational model that consists of an
infinite tape divided into cells, a tape head that can read and write symbols on
the tape, a finite control unit that determines the machine's behavior, and a set of
states that the machine can transition between. TMs can perform computations
by reading symbols from the tape, transitioning between states based on the
current symbol and the current state, and modifying the tape content.

Turing Recognizable (Recursively Enumerable) Languages:

A language is Turing recognizable, also known as recursively enumerable, if
there exists a Turing machine that, when provided with an input string
belonging to the language, halts and accepts the string. However, if the input
string does not belong to the language, the Turing machine may either halt and
reject, or it may run indefinitely without halting. Turing recognizable languages
are those that can be generated by Turing machines with no restriction on their
behavior when given non-member inputs.

Turing-Decidable (Recursive) Languages:

A language is Turing-decidable, also known as recursive, if there exists a Turing
machine that, when provided with an input string, always halts and either
accepts or rejects the string. In other words, the Turing machine makes a
definitive decision for every input string, indicating whether it belongs to the
language or not. Turing-decidable languages are a subset of Turing recognizable
languages.

Closure Properties of Turing Recognizable and Turing-Decidable Languages:

© SOUMYAJIT BAG
Turing recognizable and Turing-decidable languages possess several important
closure properties:

1. Union: If L1 and L2 are Turing recognizable or Turing-decidable languages,

their union L1 ∪ L2 is also Turing recognizable or Turing-decidable.

2. Concatenation: If L1 and L2 are Turing recognizable or Turing-decidable

languages, their concatenation L1 ∘ L2 is also Turing recognizable or Turing-
decidable.

3. Kleene Closure: If L is a Turing recognizable or Turing-decidable language,

its Kleene closure L* is also Turing recognizable or Turing-decidable.

Variants of Turing Machines:

Several variants of Turing machines exist, which introduce additional features
or constraints:

1. Nondeterministic Turing Machines (NTMs): NTMs have multiple possible

next moves for a given configuration. They can transition to multiple states or
write multiple symbols simultaneously. NTMs recognize the same class of
languages as deterministic Turing machines (DTMs) but may require fewer
steps in some cases.

2. Multitape Turing Machines: These machines have multiple tapes, each with
its own tape head. The tapes can be used for different purposes, such as input,
output, or auxiliary storage. Multitape Turing machines are more efficient than
single-tape machines for certain computations.

3. Oracle Turing Machines: These machines have an additional tape called the
oracle tape, which allows them to query an oracle for answers to specific
questions. Oracle Turing machines are used in theoretical discussions and
complexity theory to analyze the limitations of algorithms and computations.

Unrestricted Grammars and Equivalence with Turing Machines:

Unrestricted grammars, also known as Type-0 grammars, are formal grammars
with no restrictions on their production rules. They can generate languages that
can be recognized by Turing machines. The equivalence between unrestricted
grammars and Turing machines means that for every Turing machine, there
exists an equivalent unrestricted grammar, and vice versa. The grammar can
generate strings by applying production rules that simulate the computation of
the Turing machine.

Turing Machines as Enumerators:

Turing machines can be used as enumerators to list or generate the members of
a language. An enumerator is a Turing machine that systematically generates all
strings in a language, one after another, in a specific order. By running an
© SOUMYAJIT BAG
enumerator, one can list all the strings belonging to a given language.
Enumerators are used to analyze and understand the properties and structures of
languages.

Understanding Turing machines and their variants, along with the recognition
capabilities of Turing recognizable and Turing-decidable languages, plays a
fundamental role in the theory of computation and computational complexity.

Module 6

Undecidability:

Church-Turing Thesis:
The Church-Turing thesis is an important hypothesis in the theory of
computation. It states that any function that can be effectively computed can be
computed by a Turing machine. In other words, the Church-Turing thesis
suggests that Turing machines capture the notion of an algorithm or a
mechanical procedure.

Universal Turing Machine:

A universal Turing machine (UTM) is a Turing machine that can simulate the
behavior of any other Turing machine, given an appropriate encoding of the
machine's description and input. The UTM is capable of reading the description
of another Turing machine, taking its input, and executing the same steps as the
simulated machine. The UTM plays a fundamental role in the theory of
computation as it demonstrates the universality of Turing machines.

Universal and Diagonalization Languages:

The universal language, also known as the language of all encodings of Turing
machines that halt on a given input, is the set of strings representing Turing
machines that halt on a specific input. This language is not decidable, meaning
that there is no Turing machine that can determine for every input whether a
given Turing machine halts on it.

The diagonalization language, also known as the halting problem language, is

the set of strings representing Turing machines that do not halt on their own
encodings as input. It is a language that cannot be decided by any Turing
machine. This undecidable language is a crucial example used to prove the
existence of undecidable problems.

Reduction Between Languages and Rice's Theorem:

In computability theory, reduction is a technique used to establish the
undecidability or decidability of a language by mapping it to another language
whose decidability is already known. If a language L1 can be reduced to
language L2, and L2 is undecidable, then L1 must also be undecidable.

© SOUMYAJIT BAG
Rice's theorem, formulated by Henry Gordon Rice, states that for any non-
trivial property of the behavior of Turing machines, there is no general
algorithm that can decide whether a given Turing machine has that property. In
other words, any property of the language recognized by a Turing machine that
depends solely on the language itself and not on the specific implementation is
undecidable.

Undecidable Problems about Languages:

There exist several undecidable problems in the theory of languages and
computation. Some notable examples include:

1. The Halting Problem: Given a Turing machine M and an input string w,

determining whether M halts on w is undecidable.

2. Post's Correspondence Problem: Given a finite set of string pairs, determining

whether there exists a sequence that can be formed by concatenating strings
from the pairs in the same order from both the top and bottom is undecidable.

3. The Emptiness Problem: Given a Turing machine M, determining whether the

language recognized by M is empty is undecidable.

These undecidable problems demonstrate the existence of fundamental limits in

computation, highlighting the importance of understanding the boundaries and
complexity of different computational tasks.

© SOUMYAJIT BAG