COMP11212 Fundamentals of Computation

Part 1: Formal Languages

Andrea Schalk
[email protected]
January 24, 2013

Organizational issues
When, what, where
This course will be taught as follows.
Lectures: Will take place Mondays at 9.00 and Thursdays at 10.00 in
Kilburn 1.1.
Examples classes: Will take place from Week 2 as follows.
Groups
Z
B+X
Y
M+W

Time
Tuesdays 10.00
Mondays 2.00
Mondays 3.00
Mondays 1.00

Location
LF15
IT407
IT407
IT407

Two parts
This course consists of two distinct parts, taught by two different members
of staff, Andrea Schalk and David Lester.
Lectures
1
211
1221
22

From
To
28/01
28/01
31/01
04/03
07/03
29/04
To be announced

Content
Intro
Part 1
Part 2
Revision

Examples classes in Weeks

26
711

Taught by
Andrea & David
Andrea
David
David & Andrea

belongs to
Part 1
Part 2

Assessment
There is assessed coursework in the form of exercises you are expected to
prepare before each examples class, see below for more detail. The
coursework is worth 25% of the overall mark for the course unit.
The exam consists of four question, with two questions from each part
of the course. Each question in the exam is worth twenty marks. Students
have to answer three out of the four questions. This means your exam mark
will be out of sixty. It will be worth 75% of your mark for this course unit.
2

Coursework
You are expected to prepare exercises before each examples class, see
the exercise sheets at the end of the notes. During the examples class these
will be marked, and you will also get help with the questions you could not
do, potentially allowing you to get some additional marks if you can catch
up before the end of the class.
Each exercise is marked out of two, where the marks mean the following.
0: You did not make a serious attempt to solve the question.
1: You made a serious attempt but did not get close to finishing the
question.
2: You did work your way through the exercise and made no serious
mistakes.
There are five marked exercises for each examples class in this part,
allowing a mark out of ten each week. Note that not all exercise are
equally difficult, or require an equal amount of work. If you are seriously
struggling with one exercise, move on to the next one (they dont usually
build on each other). If you are struggling for time, try to do as many as you
can, leaving out the more labour-intensive ones, but try to do something.
Be aware that if you exclusively intend to work during the examples classes
youll likely get no more than four or five marks each week. Later you will
still have to learn the material for the exam.
Note that the marker may ask you how you solved the various exercises
before assigning you a mark for each of them.
If you cannot answer the question we assume you have plagiarized the
solution and you will get a mark of 0 for all your work that
week.
If this happens twice we will put a note in your file that you
have been caught plagiarizing work.
Weeks in which you have done this will not count towards the seven
examples class threshold (see below).
However, note that your marks here only count if you have something
recorded for at least seven out of the ten examples classes, or if there
are mitigating circumstances.

Changes
Note that the course unit was taught very differently until (and including)
2011/2012. Here are the most important changes.
There used to be three parts. The old second part (Part B in old exam
papers) is no longer taught. Part 1 has been very slightly, and Part 2
(the old Part 3) substantially, expanded since then.
In the old exams questions used to be worth ten marks each, with the
exam being marked out of fifty.
There used to be coursework on top of the exercises prepared for the
examples classes. This no longer exists, but now the examples class
prep is being marked.
3

About Part 1 of the course

As detailed on the previous page, Part 1 of this course unit is concerned with
formal languages and is taught by Andrea Schalk. The following concerns
organizational issues just for this first part.

Learning the material

The most important aims of this part of the course are to teach you
a few concepts (such as formal language, regular expression, finite state
automaton, context-free grammar) and
how to solve a number of typical problems,
with an emphasis on the latter.
Exercises. The best way to learn how to solve problems is to do it,
which is why the notes contain a great number of exercises. Those not
covered in the exercise sheets provide further opportunities to practise, for
example when revising for the exams. The exercises are part of the examinable material, although I wont ask you to recall any particular exercise
in the exam. Some exercises have a *, which means that they are not part
of the examinable material. Note that they are not necessarily difficult,
although some of them are.
Lectures. I use the lectures to introduce the main ideas, and to explain
the important algorithms, as well as suggesting some ways for solving typical
problems.
Self-study. The course has only three contact hours a week, and you
are expected to spend a substantial amount of time each week working on
the material by studying the notes yourself, connecting the written material
with what was introduced in the lectures, and solving exercises.
Notes. The notes are fairly substantial because they aim to cover the
material in detail. Typically, for each new idea or algorithm, there is a
formal description as well as one or two examples. For most people it is
worthwhile trying to understand the example before tackling the general
case, although in the notes the general case is sometimes described before
the example (because it can be difficult to explain the idea using just a
special case). Note that there is a glossary that you can use to quickly find
the meaning of the various technical terms used.
Mathematical notation. The language used to describe much of theoretical computer science is that of mathematics, because that is the only
way of describing with precision what we are talking about. In the main
part of the notes I have tried to remain as informal as possible in my descriptions, although all the definitions given are formal. You will not be
required to repeat these definitions in the exam, you only need to understand informally what they mean. There is an appendix which spells out in
more detail how the language of mathematics is used to make rigorous the
4

concepts and operations we use. If you only understand the informal side of
the course you can pass the exam (and even get a very good mark) but to
master the material (and get an excellent result in the exam) you will have
to get to grips with the mathematical notation as well. A maximum of 10%
of the final mark depends on the material in the Appendix.
Examples classes. The examples classes give you the opportunity to
get help with any problems you have, either with solving exercises or with
understanding the notes. There are four examples classes associated with
this part of the course, in Weeks 25. For each of these you are expected
to prepare by solving the key exercises on the appropriate sheet. It also
suggests exercises to do if you are working through the Appendix, and there
are suggestions for additional exercises you might want to do if you find the
set work easy. The sheets can be found from pages 83 of the notes. We will
check in each examples classes whether you have done the preparation, and
the data will be entered into Arcade. Solutions for each exercise sheet will
be made available on the webpage for this part of the course after the last
associated examples class has taken place.
Marking criteria. The marking criteria for the assessed coursework are
stricter than those for the exam. In particular, I ask you to follow various
algorithms as described in the notes, whereas in the exam Im happy if you
can solve the problem at hand, and I dont mind how exactly you do that.
In all cases it is important to show your workif you only give an answer
you may lose a substantial number of marks.
Revision. For revision purposes I suggest going over all the exercises
again. Some of the exercises will probably be new to you since they are not
part of the set preparation work for the examples classes. Also, you can turn
all the NFAs you encounter along the way into DFAs (removing -transitions
as required). Lastly, there are exams from previous years to practice on. If
you can do the exercises you will do well in the exam. You will not be asked
to repeat definitions, but knowing about properties of regular and contextfree languages may be advantageous. You should also be aware of the few
theorems, although you will not be asked to recite them.
Webpage for the course. http://www.cs.manchester.ac.uk/ugt/
2012/COMP11212/

Reading
For this part of the course these notes cover all the examinable material.
However, sometimes it can be useful to have an additional source to see the
same material introduced and explained in a slightly different way. Also,
there are new examples and exercises to be looked at. For this purpose you
may find one of the following useful.
M. Sipser. Introduction to the Theory of Computation. PWS Publishing Company, 1997. ISBN 0-534-94728-X.
This is a fairly mathematical book that nonetheless tries to keep mathematical notation at a minimum. It contains many illustrated examples and
aims to explain ideas rather than going through proofs mechanically. A 2005
edition is also available. Around 50this book is very well thought of and
even used copies are quite expensive. Relevant to this course: Chapters 0
(what hasnt yet been covered by COMP10020), 1 and 2.1

J.E. Hopcroft, R. Motwani, and J.D. Ullman. Introduction to Automata

Theory, Languages, and Computation. Addison Wesley, second edition, 2001. ISBN 0-201-44124-1.
This account is derived from the classical textbook on the subject. It
is more verbose than Sipser, and it aims to include up-to-date examples
and applications, and it develops the material in much detail. A number of
illustrated examples are also included. I have heard that this book is very
popular among students. 2006/7 editions are also available. About 50.
Relevant for this course are Chapters 1 to 5.

Get in touch
Feedback. Im always keen to get feedback regarding my teaching. Although the course has been taught a few times the notes probably still
contain some errors. Id like to hear about any errors that you may find
so that I can correct them. I will make those available at http://www.cs.
manchester.ac.uk/ugt/2010/COMP11212/ and fix them for future years.
You can talk to me after lectures, or send me email (use the address on the
title page).
Reward! If you can find a substantial mistake in the lecture notes (that
is, more than just a typo, or a minor language mistake) you get a chocolate
bar.
Wanted. Im also on the lookout for good examples to use, either in the
notes or in the lectures. These should be interesting, touch important applications of the material, or be fun. For a really good reasonably substantial
such example the reward is a chocolate bar.
Acknowledgements. I would like to thank Howard Barringer, Pete
Jinks, Djihed Afifi, Francisco Lobo, Andy Ellyard, Ian Worthington, Peter
Sutton, James Bedford, Matt Kelly, Cong Jiang, Mohammed Sabbar, Jonas
Lorenz, Tomas Markevicius and Joe Razavi for taking the time to help me
improve these notes.

Contents
Organization

1 What it is all about

2 Describing languages to a computer

2.1 Terminology . . . . . . . . . . . . . . . . . . . .
2.2 Defining new languages from old ones . . . . .
2.3 Describing languages through patterns . . . . .
2.4 Regular expressions . . . . . . . . . . . . . . . .
2.5 Matching a regular expression . . . . . . . . . .
2.6 The language described by a regular expression
2.7 Regular languages . . . . . . . . . . . . . . . .
2.8 Summary . . . . . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

11
11
12
13
15
16
17
19
19

3 How do we come up with patterns?

3.1 Using pictures . . . . . . . . . . . . . .
3.2 Following a word . . . . . . . . . . . .
3.3 Finite state automata . . . . . . . . .
3.4 Non-deterministic automata . . . . . .
3.5 Deterministic versus non-deterministic
3.6 From automata to patterns . . . . . .
3.7 From patterns to automata . . . . . .
3.8 Properties of regular languages . . . .
3.9 Equivalence of Automata . . . . . . .
3.10 Limitations of regular languages . . .
3.11 Summary . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.
.

20
20
22
23
25
28
34
42
50
53
55
56

4 Describing more complicated languages

4.1 Generating words . . . . . . . . . . . . .
4.2 Context-free grammars . . . . . . . . . .
4.3 Regular languages are context-free . . .
4.4 Parsing and ambiguity . . . . . . . . . .
4.5 A programming language . . . . . . . .
4.6 The Backus-Naur form . . . . . . . . . .
4.7 Properties of context-free languages . .
4.8 Limitations of context-free languages . .
4.9 Summary . . . . . . . . . . . . . . . . .

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.

58
58
59
63
64
67
68
69
70
70

Glossary

72

A In the language of mathematics

A.1 The basic concepts . . . . . . .
A.2 Regular expressions . . . . . . .
A.3 Finite state automata . . . . .
A.4 Grammars . . . . . . . . . . . .

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

.
.
.
.

75
75
78
79
82

Exercise Sheet 1

83

Exercise Sheet 2

84

Exercise Sheet 3

85

Exercise Sheet 4

86

Exercise Sheet 5

87

Chapter 1

What it is all about

Computers need to be able to interpret input from a keyboard. They have
to find commands and then carry out appropriate actions. With a command
line interface this may appear a fairly simple problem, but, of course, the
keyboard is not the only way of feeding a computer with instructions. Computers also have to be able to read and interpret files, for example when it
comes to compiling a program from a language such as Java, C or ML and
they then have to be able to run the resulting code. In order to do this
computers have to be able to split the input into strings, parse these strings,
and turn those into instructions (such as Java bytecode) that the machine
can carry out.
How does a computer organize potentially quite large input? How does
it decide what variables to create, and what values to give these? In this
part of Fundamentals of Computation we look at some techniques used for
this purpose.
In order to organize (or parse) the given text-like input a computer has
to be able to recognize specific strings, for example simple commands (such
as if or else). The ability to find certain strings is useful in other contexts
as well. When a search engine such as Google looks for a string or phrase
entered by a user it certainly has to be capable of telling when it has found
the string in question. However, most search engines do their job in a
much more clever way than that: They will also recognize plurals of strings
entered, and if such a string should be a verb of the English language it will
typically recognize different forms, such as type, typing and typed. How can
we come up with clever ways of searching for something more than just a
given fixed string? Again this is an issue that this course covers.
How does the web server for the Intranet in the School of Computer
Science check whether you are accessing it from within the University?
How do online shopping assistants work? They find a given product at
various shops and tell you what it costs. How do they extract the information, in particular since some of the pages describing the product are created
on demand only?
Imagine you had to write a piece of code that accepts a string as an
input and checks whether this string could be a valid email address.
Imagine you had to write a program that goes through a number of
emails in a folder and returns all the subject lines for those (every email
client has a way of doing this, of course). How would you do that?
Imagine you had to write a program that goes through a bunch of documentation files and checks whether they contain any double typed words
(such as the the). This is a very common mistake, and a professional or-

ganization would want to remove these before delivering the software to a

customer.
What if
Your program had to work across lines (and an arbitrary number of
spaces)?
You had to take into account that the first doubled word might be
capitalized?
The two doubled words might be separated by html tags?
Imagine you had to write code for a vending machine that takes coins
and in return delivers variously priced goods. How does the machine keep
track of how much money the user has put in so far, and how to produce
correct change from the coins it holds? Its your job to keep track of all the
cases that may arise.
Imagine you had to write a piece of code that takes as input a file containing a Java program and checks whether the curly brackets { } are all
properly balanced.
This course is going to help you with understanding how such tasks can
be carried out. However, we wont spend much time writing code. The tools
we describe here could be used when using the Unix grep command, or when
programming in languages such as Perl, Python, Tcl, or even when using
the GNU Emacs editor. It is more important to me that you understand
the ideas, and how to express them in a reasonably formal way (so that
programming them then becomes easy).
The material can be split into three parts, dealing with the following
issues.
How do we describe to a computer what strings to look for?
How can we come up with the right description to solve our problem?
How can we write code that checks whether some other piece of code
is correctly formed, whether the opening and closing brackets match,
or whether a webpage is written in valid html?

10

Chapter 2

Describing languages to a
computer
In order to solve the kinds of problems that are mentioned in Chapter 1 we
need to be able to describe to a computer what it is looking for.
Only in very simple cases will this consist of just one or two stringsin
general, we want the computer to look for a much larger collection of words.
This could be the set of all possible IP addresses within the University, or
it could be the collection of all strings of the form Subject:. . . (to pick
out emails on a particular topic), or it could be the set of all strings of the
form s s (in the simplest case of finding all doubled wordsthis one wont
take care of doubled words spread over two lines, nor of the first word being
capitalized).

2.1

Terminology

In order to talk about how we do this in practice we have to introduce a

few terms so that we can talk about the various concepts involved. You can
find formal definitions of these Appendix A.
A symbol is a building block for a string. Symbols cannot be subdivided, they are the atoms of everything we build. In the theory of
formal languages they are usually called letters. Examples: a, A, 0,
1, %, @. We use letters from the end of the Roman alphabet, such as
x, y, z to refer to an arbitrary symbol.
An alphabet is a collection of letters. We think of it as a set. At
any given time we are only interested in those words we can build
from a previously specified alphabet. Examples: {0, 1}, {0, 1, . . . , 9},
{a, b, . . . , z}, {a, b, . . . , z, A, B, . . . , Z},. We use capital Greek letters,
typically , to refer to an arbitrary alphabet.
A string is something we build from 0 or more symbols. In the theory
of formal languages we usually speak of words. Examples: ababab, 0,
1001. Note that every letter can be viewed as a one-letter word. We
use letters s, t, u to refer to an arbitrary word.
The empty word (which consists of 0 letters) is a bit difficult to
denote. We use for this purpose.
Concatenation is the operation we perform on words (or letters) to
obtain longer words. When concatenating a with b we get ab, and we
11

can concatenate that with the word ba to obtain abba. If we concatenate 0 letters we get the word . When we concatenate any word with
the word we obtain the same word.
We use the notation of powers for concatenation as follows: If s is a
word then (s)n is the word we get by concatenating n copies of s. For
example, (010)3 = 010010010, 12 = 11, b0 = and c1 = c.
A language is a collection of words, which we think of as a set. Examples are {}, , {ab, abc, aba} and {an | n N}. We use letters such
as L, L1 and L to refer to an arbitrary language.
In these notes we are interested in how we can describe languages in
various ways.
If a language is finite then we can describe it quite easily: We just have
to list all its elements. However, this method fails when the language is
infinite, and even when the language in question is very large. If we want to
communicate to a computer that it is to find all words of such a language
we have to find a concise description.

2.2

Defining new languages from old ones

One way of describing languages is using set-theoretic notation (see Appendix A for more detail). In the main development here we try to avoid
being overly mathematical, but there are some operations on languages we
need to consider. There are seven of them:
Union. Since languages are just sets we can form their unions.
Intersection. Since languages are merely sets we can form their intersections.
Set difference. If L1 and L2 are languages we can form
L1 \ L2 = {s L1 | s
/ L2 }.
Complement. If L is the language of all words over some alphabet
, and L is a subset of L then the complement of L in L is L \ L ,
the set of all words over which are not contained in L .
Concatenation. Because we can concatenate words we can use the
concatenation operation to define new languages from existing ones by
extending this operation to apply to languages as follows. Let L1 and
L2 be languages over some alphabet . Then
L1 L2 = {s t | s L1 and t L2 }.
n-ary Concatenation. If we apply concatenation to the same language by forming L L there is no reason to stop after just one concatenation. For an arbitrary language L we define
Ln = {s1 s2 sn | si L for all 1 i n}.
We look at the special case
L0 = {s1 sn | si L for all 1 i n} = {},
since is what we get when concatenating 0 times.
12

 Kleene star. Sometimes we want to allow any (finite) number of

concatenations of words from some language. The operation that does
this is known as the Kleene star and written as () . The definition
is:
L = {s1 s2 sn | n N, s1 , s2 , . . . , sn L}
[
=
Ln .
nN

Note that
=

n = {} = L0

nN

for all L, which may be a bit unexpected.

Exercise 1. To gain a better understanding of the operation of concatenation on languages, carry out the following tasks.
(a) Calculate {a, aaa, aaaaaa} {, bb, bbb}.
(b) Calculate {, a, aa} {aa, aaa}.
(c) Calculate {a, a3 , a6 } {b0 , b2 , b3 }.
(d) Describe the language {0m 1n | m, n N} as the concatenation of two
languages.
(e) Calculate {0, 01, 001}2 .
Exercise 2. To gain a better understanding of the Kleene star operation
carry out the following tasks. (You may use any way you like to describe the
infinite languages involved: Just using plain English is the easiest option. I
would prefer not to see . . . in the description.)
(a) Calculate {a, b} , {a} {b} , {a} {b} , {aa} \ {aaaa} and the
complement of {a} in the language of all words over the alphabet {a, b}.
(b) Calculate {02n | n N} .
(c) Describe the set of all words built from a given alphabet using the
Kleene star operation.

2.3

Describing languages through patterns

By definition a languages is a set. While sets are something mathematicians

can deal with very well, and something that computer scientists have to
be able to cope with, computers arent at all well suited to make sense of
expressions such as
{(01)n | n N}.
There are better ways of showing a computer what all the words are that
we mean in a particular case. We have already seen that we can express the
language L in a different way, namely as
{01} .
If there is only one word in a language, here {01}, we might as well
leave out the curly brackets and write1 (01) . That is actually something a
1

Can you see why we need brackets here?

13

machine can cope with, although we would have to use a slightly different
alphabet, say (01)^*. All a computer now has to do is to compare: Does
the first character2 of my string equal 0? Does the next one3 equal 1? And
so on.
What we have done here is to create a pattern. It consists of various characters of the alphabet, concatenated. We are allowed to apply the Kleene
star to any part of the string so created, and we use brackets ( ) to indicate
which part should be affected by the star. A computer can then match this
pattern.
Are these all the patterns we need? Not quite. How, for example, would
we describe the language
{0n | n N} {1n | n N} = {xn | x = 0 or x = 1}?
We cant use either of 0 1 or (arguably worse) (01) because both of
these include words that contain 0 as well as 1, whereas any word in our
target language consists entirely of 0s or entirely of 1s. We need to have a
way of saying that either of two possibilities might hold. For this, we use
the symbol |. Then we can use 0 |1 to describe the language above.
Exercise 3. Which of the following words match the given patterns?
Pattern
(ab)
a b
(a|b)
(a|b)
ab|b|a

ab

aba

abab

aab

aabb

aa

Exercise 4. Describe all the words matching the following patterns. For
finite languages just list the elements, for infinite ones, try to describe the
words in question using English. If you want to practise using set-theoretic
notation, add a description in that format.
(a) (0|1|2)
(b) (0|1)(0|2)2
(c) (01|10)
(d) 0 1,
(e) (01) 0,
(f) 0 1 ,
(g) (010) ,
(h) (01) 0 ,
(i) (01) (01) .
(j) (0|1)
(k) 0 |1 |2
(l) 0|1 |2
2
3

What should the computer do if the string is empty?

What if there isnt a next symbol?

14

2.4

Regular expressions

So far we have been using the idea of a pattern intuitivelywe have not
said how exactly we can form patterns, nor have we properly defined when
a word matches a pattern. It is time to become rigorous about these issues.
For reason of completeness we will need two patterns which seem a
bit weird, namely (the pattern which is matched precisely by the empty
word ) and (the pattern which is matched by no word at all).
Definition 1. Let be an alphabet. A pattern or regular expression
over is any word over
pat = {, , |, , (, )}
generated by the following inductive definition.
Empty pattern The character is a pattern;
Empty word the character is a pattern;
Letters every letter from is a pattern;
Concatenation if p1 and p2 are patterns then so is (p1 p2 );
Alternative if p1 and p2 are patterns then so is (p1 |p2 );
Kleene star if p is a pattern then so is (p).
In other words we have defined a language4 , namely the language of all
regular expressions, or patterns. Note that while we are interested in words
over the alphabet we need additional symbols to create our patterns. That
is why we have to extend the alphabet to pat .
In practice we will often leave out some of the brackets that appear in
the formal definitionbut only those brackets that can be uniquely reconstructed. Otherwise we would have to write ((0|1) 0) instead of the simpler
(0|1) 0. In order to be able to do that we have to define how to put the
brackets back into such an expression. We first put brackets around any occurrence of with the sub-pattern immediately to its left, then around any
occurrence of concatenation, and lastly around the alternative operator |.
Note that every regular expression with all its brackets has precisely one
way of building it from the ruleswe say that patterns are uniquely parsed.
This isnt quite true once we have removed the brackets: (0|(1|2)) turns into
0|1|2 as does ((0|1)|2). However, given the way we use regular expressions
this does not cause any problems.
Note that many computer languages that use regular expressions have
additional operators for these (see Exercise 7). However, these exist only
for the convenience of the programmer and dont actually make these regular expressions more powerful. We say that they have the same power of
expressivity. Whenever I ask you to create a regular expression or pattern
it is Definition 1 I expect you to follow.
4

In Chapter 4 we look at how to describe a language like thatit cannot be done using
a pattern.

15

2.5

Matching a regular expression

This language is recursively defined: There are base cases consisting of

the , and all the letters of the underlying alphabet, and three ways of
constructing new patterns from old ones. If we have to define something for
all patterns we may now do so by doing the following:
Say how the definition works for each of the base cases;
assuming we have defined the concept for p, p1 and p2 , say how to
obtain a definition for
p1 p2 ,
p1 |p2 and
p .
This is an example of a recursive definition, and you will see more of
these in COMP10020.5
Note that in the definition of a pattern no meaning is attached to any of
the operators, or even the symbols that appear in the base cases. We only
find out what the intended meaning of these is when we define when a word
matches a pattern. We do this by giving a recursive definition as outlined
above.
Definition 2. Let p be a pattern over an alphabet and let s be a word
over . We say that s matches p if one of the following cases holds:
Empty word The empty word matches the pattern ;
Base case the pattern p = x for a character x from and s = x;
Concatenation the pattern p is a concatenation p = (p1 p2 ) and there are
words s1 and s2 such that s1 matches p1 , s2 matches p2
and s is the concatenation of s1 and s2 ;
Alternative the pattern p is an alternative p = (p1 |p2 ) and s matches
p1 or p2 (it is allowed to match both);
Kleene star the pattern p is of the form p = (q) and s can be written
as a finite concatenation s = s1 s2 sn such that s1 , s2 ,
. . . , sn all match q; this includes the case where s is empty
(and thus an empty concatenation, with n = 0).
Note that there is no word matching the pattern .
Exercise 5. Calculate all the words matching the patterns and a (for
a ) respectively.
Exercise 6. For the given pattern, and the given word, employ the recursive
Definition 2 to demonstrate that the word does indeed match the pattern:
(a) the pattern (ab) a and the word aba.
(b) the pattern (0|1) 10 and the word 10010,
(c) the pattern (0 |1 )10 and the word 0010,
5

Students who are on the joint honours CS and Maths programme dont take
COMP10020, but they should be able to grasp these ideas without problems.

16

(d) the pattern (abc) a and the word a.

Exercise 7. Heres a hint of the usage of regular expressions in the real
world.
(a) Print out the manual page for the Unix command grep (type man grep
to get that page). Now argue that for every regular expression understood
by grep there is a regular expression according to Definition 1 that has
precisely the same words matching it.6
(b) Give a command-line instruction which will show all the subjects and
authors of mails contained in some directory. Make sure to test your suggested answer on a computer running Linux. Hint: Try using egrep.
(c) Give a regular expression that matches all IP addresses owned by the
University of Manchester. You probably want to use some kind of shortcut
notation.

2.6

The language described by a regular expression

Given a pattern we can now define a language based on it.

Definition 3. Let p be a regular expression over an alphabet . The language defined by p, L(p) is the set of all words over that match p. In
other words
L(p) = {s | s matches p}.
Note that different patterns may define the same language, for example
L(0 ) = L(|00 ).
Note that we can also define the language given by a pattern in a different
way, namely recursively.
L() = ;
L() = {};
L(x) = {x} for x ;
and for the operations
L(p1 p2 ) = L(p1 ) L(p2 );
L(p1 |p2 ) = L(p1 ) L(p2 );
L(p ) = (L(p)) .
We can use this description to calculate the language defined by a pattern
as in the following example.
L((0|1) ) = (L(0|1))
= (L(0) L(1))
= ({0} {1})
= {0, 1}
This is the language of all words over the alphabet {0, 1}.
6

Theoreticians want their patterns with as few cases as possible so as to have fewer
cases for proofs by induction. Practicians want lots of pre-defined shortcuts for ease of
use. This exercise shows that it doesnt really matter which version you use.

17

Exercise 8. Use this recursive definition to find the languages defined by

the following patterns, that is the languages L(p) for the following p.
(a) (0 |1 )
(b) (01) 0
(c) (00)
(d) ((0|1)(0|1)) .
Can you describe these languages in English?
In order to tell a computer that we want it to look for words belonging
to a particular language we have to find a pattern that describes it. The
following exercise lets you practise this skill.
Exercise 9. Find7 a regular expression p over the alphabet {0, 1} such that
the language defined by p is the one given. Hint: For some of the exercises it
may help not to think of a pattern that is somehow formed like the strings you
want to capture, but to realize that as long as theres one way of matching
the pattern for such a string, thats good enough.
(a) All words that begin with 0 and end with 1.
(b) All words that contain at least two 0s.
(c) All words that contain at least one 0 and at least one 1.
(d) All words that have length at least 2 and whose last but one symbol
is 0.
(e) All words which contain the string 11, that is, two consecutive 1s.
(f) All words whose length is at least 3.
(g) All words whose length is at most 4.
(h) All words that start with 0 and have odd length.
(i) All words for which every letter at an even position is 0.
(j) All words that arent equal to the empty word.
(k) All words that contain at least two 0s and at most one 1.
(l) All words that arent equal to 11 or 111.
(m) All words that contain an even number of 0.
(n) All words whose number of 0s is divisible by 3.
(o) All words that do not contain the string 11.
(p) All words that do not contain the string 10.
(q) All words that do not contain the string 101.
(r)* All words that contain an even number of 0s and whose number of 1s
is divisible by 3.
7

If you find the last few of these really hard then skip them for now. The tools of the
next chapter should help you finish them.

18

Exercise 10. Find a regular expression p over the alphabet {a, b, c} such
that the language defined by p is the one given.
(a) All the words that dont contain the letter c.
(b) All the words where every a is immediately followed by b.
(c) All the words that do not contain the string ab.
(d) All the words that do not contain the string aba.

2.7

Regular languages

We have therefore defined a certain category of languages, namely those that

can be define using a regular expression. It turns out that these languages
are quite important so we give them a name.
Definition 4. A language L is regular if it is the set of all words matching
some regular expression, that is, if there is a pattern p such that L = L(p).
Regular languages are not rare, and there are ways of building them.
Nonetheless in Chapter 4 we see that not all languages of interest are regular.
Proposition 2.1. Assume that L, L1 , and L2 are regular languages over
an alphabet . Then the following also are regular languages.
(a) L1 L2
(b) L1 L2
(c) Ln
(d) L
Regular expressions are very handy when it comes to communicating to
a computer quite complicated collections of words that we might be looking
for. However, coming up with a regular expression that describes precisely
those words we have in mind isnt always easy.
In the next chapter we look at different ways of describing languages
ways which make it easier for human beings to ensure that they are describing the language they had in mind. There is then an algorithm that
translates one of these descriptions into a pattern.

2.8

Summary

In order to describe a language to a computer we can use regular

expressions, also known as patterns.
For regular expressions we have the notion of it being matched by some
particular word.
Each regular expression defines a language, namely the set of all words
that match it.
We say that a language is regular if there is a pattern that defines it.

19

Chapter 3

How do we come up with

patterns?
As you should have noticed by now (if you have done the last two exercises
from the previous section) coming up with a pattern that describes a particular language can be quite difficult. One has to develop an intuition about
how to think of the words in the desired language, and turn that into a
characteristic for which a pattern can be created. While regular expressions
give us a format that computers understand well, human beings do much
better with other ways of addressing these issues.

3.1

Using pictures

Imagine somebody asked you to check a word of an unknown length to see

whether it contains an even number of 0s. The word is produced one letter
at a time. How would you go about that?
In all likelihood you would not bother to count the number of 0s, but
instead you would want to remember whether you have so far seen an odd,
or an even, number of 0s. Like flicking a switch, when you see the first 0,
youd remember odd, when you see the second, youd switch back (since
thats the state youd start with) to even, and so forth.
If one wanted to produce a description of this idea it might look like this:

0
Even

Odd
0

Every time we see 0 we switch from the even state to the odd state
and vice versa.
If we wanted to give this as a description to somebody else then maybe
we should also say what we are doing when we see a letter other than 0,
namely stay in whatever state were in. Lets assume we are talking about
words consisting of 0s and 1s. Also, wed like to use circles for our states
because they look nicer, so well abbreviate their names.

0
E

O
0

20

So now somebody else using our picture would know what to do if the
next letter is 0, and what to do if it is 1. But how would somebody else
know where to begin?

0
E

O
0

We give them a little arrow that points at the state one should start
in. However, they would still only know whether they finished in the state
called E or the one called O, which wouldnt tell them whether this was the
desired outcome or not. Hence we mark the state we want to be in when
the word comes to an end and now we do have a complete description of our
task.

0
E

O
0

We use a double circle for a state to indicate that if we end up there

then the word we looked at satisfied our criteria.
Lets try this idea on a different problem. Lets assume were interested
in whether our word has 0 in every position that is a multiple of three, that
is, the third, sixth, ninth, etc, letters, if they exist, are 0.
So we start in some state, and we dont care whether the first letter is 0
or 1, but we must remember that weve seen one letter so that we can tell
when we have reached the first letter, so wed draw something like this:

0, 1

Similarly for the second letter.

0, 1

0, 1

Now the third. This time something happens: If we see 0, were still
okay, but if we see 1 then we need to reject the word.

4
0

0, 1

0, 1

1
2

So what now? Well, if we reach the state labelled 4 then no matter

what we see next, we will never accept the word in question of satisfying
our condition. So we simply think of this state as one that means we wont
accept a word that ends there. All the other states are fineif we stop in
any of them when reading a word it does satisfy our condition. So we mark
all the other states as good ones to end in, and add the transitions that keep
us in state 4.
21

0, 1
4
0

0, 1

0, 1

1
2

But what if the word is okay until state 3? Then we have to start all
over again, not caring about the next letter or the one after, but requiring
the third one to be 0. In other words, were in the same position as at the
startso the easiest thing to do is not to create state 3, but instead to have
that edge go back to state 0.

0, 1
4

0
0

0, 1

0, 1

1
2

Exercise 11. For the languages described in parts (b) and (c) of Exercise 9,
draw a picture as in the examples just given.

3.2

Following a word

Consider the following picture.

b
a

a, b

a, b
b

b
a

What happens when we try to follow a word, say abaa?

To begin with we are in the start
state.

We see a, so we follow the edge labelled a. abaa.

b
a

a, b
b

a, b

a, b
b

b
a

a, b

b
a

Having followed the first letter, we

can drop it.
22

Now we see the letter b, so we go

once round the loop in our current
state. baa.

Now we have aa so we follow the

edge labelled a
b

b
a

a
a

a, b

a, b

a, b
b

a, b

b
a
a

Lastly we have the word a, so we

follow the edge to the right, labelled a, b.
b
a

a, b
b

a, b

b
a

We end up in a non-accepting state so this wasnt one of the words we

were looking for.
Exercise 12. Follow these words through the above automaton. If you
can, try to describe the words which end up in the one desirable state (the
double-ringed one).
(a) abba
(b) baba
(c) baab.

3.3

Finite state automata

So far we have been very informal with our pictures. It is impossible to

define an automaton without becoming fairly mathematical. For any of
these pictures we must know which alphabet we are considering.
Every automaton consists of
a number of states,
one state that is initial,
a number (possibly 0) of states that are accepting
for every state and every letter x from the alphabet precisely one transition from that
state to another labelled with x.

Formally we may consider the set of all the states in the automaton,
say Q. One of these states is the one we want to start in, called the start
state or the initial state. Some of the states are the ones that tell us if we

23

end up there we have found the kind of word we were looking for. We call
these accepting states. They form a subset, say F , of Q.
The edges in the graph are a nice way of visualizing the transitions.
Formally what we need is a function that
takes as its input
a state and
a letter from
and returns
a state.
We call this the transition function, . It takes as inputs a state and a letter,
so the input is a pair (q, x), where q Q and x . That means that the
input comes from the set
Q = {(q, x) | q Q, x }.
Its output is a state, that is an element of Q. So we have that
:Q

- Q.

The formal definition then is as follows.

Definition 5. Let be a finite alphabet of symbols. A (deterministic)
finite1 automaton, short DFA, over consists of the following:
A finite non-empty set Q of states;
a particular element of Q called the start state (which we often denote
with q );
a subset F of Q consisting of the accepting states;
a transition function which for every state q Q and every symbol
x returns the next state (q, x) Q, so is a function from Q
to Q. When (q, x) = q we often write
q

- q.

We sometimes put these four items together in a quadruple and speak of the
DFA (Q, q , F, ).
Sometimes people also refer to a finite state machine.
Note that for every particular word there is precisely one path through
the automaton: We start in the start state, and then read off the letters
one by one. The transition function makes sure that we will have precisely
one edge to follow for each letter. When we have followed the last letter
of the word we can read off whether we want to accept it (if we are in an
accepting state) or not (otherwise). Thats why these automata are called
deterministic; we see non-deterministic automata below.
For every word x1 x2 xn we have a uniquely determined sequence of
states q , q1 , . . . , qn such that
(q = q0 )

x1

- q1

x2

- q2

xn

- qn .

We accept the word if and only if the last state reached, qn is an accepting
state. Formally it is easiest to define this condition as follows.
1

All our automata have a finite number of states and we often drop the word finite
when referring to them in these notes.

24

Definition 6. A word s = x1 xn over is accepted by the deterministic finite automaton (Q, q , F, ) if

(q , x1 ) = q1 ,
(q1 , x2 ) = q2 ,
...,
(qn1 , xn ) = qn and
qn F , that is, qn is an accepting state.
In particular, the empty word is accepted if and only if the start state is
an accepting state.
Just as was the case for regular expression we can view a DFA as defining
a language.
Definition 7. We say that a language is recognized by a finite automaton if it is the set of all words accepted by the automaton.
Exercise 13. Design automata that accept precisely those words given by
the languages described in the various parts of Exercise 9 (you should already
have taken care of (b) and (c)). Note that (r) is no longer marked with
here.
Exercise 14. Do the same for the languages described in Exercise 10.

3.4

Non-deterministic automata

Sometimes it can be easier to draw an automaton that is non-deterministic.

What this means is that from some state, there may be several edges labelled
with the same letter. As a result, there is then no longer a unique path when
we follow a particular word through the automaton. Hence the procedure
of following a word through an automaton becomes more complicatedwe
have to consider a number of possible paths we might take.
When trying to draw an automaton recognizing the language from Exercise 9 (a) you may have been tempted to draw such an automaton.
The language in question is that of all words over {0, 1} that begin with
0 and and with 1. We know that any word we accept has to start with 0,
and if the first letter we see is 1 then we never accept the word, so we begin
by drawing something like this.

0, 1
2
1
0

Now we dont care what happens until we reach the last symbol, and
when that is 1 we want to accept the word. (If it wasnt the last letter then
we shouldnt accept the word.) The following would do that job:

25

0, 1
2
1
0

0, 1
0

0, 1
But now when we are in state 1 and see 1 there are two edges we might
follow: The loop that leads again to state 1 or the edge that leads to state 3.
So now when we follow the word 011 through the automaton there are two
possible paths:
From state 0, read 0, go to state 1.

From state 0, read 0, go to state 1.

Read 1 and go to state 3.

Read 1 and go to state 1.

Read 1 and go to state 2.

Read 1 and go to state 3.

Finish not accepting the word.

Finish accepting the word.

We say that the automaton accepts the word if there is at least one such
path that ends in an accepting state.
So how does the definition of a non-deterministic automaton differ from
that of a deterministic one? We still have a set of states Q, a particular
start state q in Q, and a set of accepting states F Q.
However, it no longer is the case that for every state and every letter from
x there is precisely one edge labelled with x, there may be several. What
we no longer have is a transition function. Instead we have a transition
relation.2 Given a state q, a letter x and another state q the relation tells
us whether or not there is an edge labelled x from q to q .
Exercise 15. Go back and check your solutions to Exercises 13 and 14.
Were they all deterministic as required? If not, redo them.
We can turn this idea into a formal definition.
Definition 8. A non-deterministic finite automaton, short NFA, is
given by
a finite non-empty set Q of states,
a start state q in Q,
a subset F of Q of accepting states as well as
a transition relation which relates a pair consisting of a state and a
letter to a state. We often write
q

- q

if (q, x) is -related to q .
We can now also say when an NFA accepts a word.
2

You will meet relations also in COMP10020.

26

Definition 9. A word s = x1 xn over is accepted by the nondeterministic finite automaton (Q, q , F, ) if there are states
q0 = q , q1 , . . . , qn
such that for all 0 i < n, relates (qi , xi ) to qi+1 and such that qn F ,
that is, qn is an accepting state. The language recognized by an NFA
is the set of all words it accepts.
An NFA therefore accepts a word x1 x2 xn if there are states
q = q0 , q1 , . . . , qn
such that
(q = q0 )

x1

x2

- q1

- q2

xn

- qn ,

and qn is an accepting state. However, the sequence of states is no longer

uniquely determined, and there could potentially be many.
Exercise 16. Consider the following NFA.

b
a

a, b

Which of the following words are accepted by the automaton? , aba,

bbb, bba, baa, baba, babb, baaa, baaaa, baabb, baaba. Can you describe the
language consisting of all the words accepted by this automaton?
Note that in the definition of NFA there is no rule that says that for
a given state there must be an edge for every label. That means that
when following a word through a non-deterministic automaton we might
find ourselves stuck in a state because there is no edge out of it for the letter
we currently see. If that happens then we know that along our current
route, the word will not be accepted by the automaton. While this may
be confusing at first sight, it is actually quite convenient. It means that
pictures of non-deterministic automata can be quite small.
Take the above example of finding an NFA that accepts all the words
that start with 0 and end with 1. We can give a much smaller automaton
that does the same job.

0, 1
0

You should spend a moment convincing yourself that this automaton

does indeed accept precisely the words claimed (that is, the same ones as
the previous automaton).3
This is such a useful convention that it is usually also adopted when
drawing deterministic automata. Consider the problem of designing a DFA
that recognizes those words over the alphabet {0, 1} of length at least 2
for which the first letter is 0 and the second letter is 1. By concentrating
on what it takes to get a word to an accepting state one might well draw
something like this:
3

Note that unless we have to describe the automaton in another way, or otherwise have
reasons to be able to refer to a particular state, there is no reason for giving the states
names in the picture.

27

0, 1
0

This is a perfectly good picture of a deterministic finite automaton. However, not all the states, and not all the transitions, are drawn for this automaton: Above we said that for every state, and every letter from the
alphabet, there must be a transition from that state labelled with that letter. Here, however, there is no transition labelled 1 from the state 0, and
no transition labelled 0 from the state 1.
What does the automaton do if it sees 1 in state 0, or 0 in state 1? Well,
it discards the word as non-acceptable, in a manner of speaking.
We can complete the above picture to show all required states by assuming theres a hidden state that we may think of as a dump. As soon as we
have determined that a particular word cant be accepted we send it off in
that dump state (which is certainly not an accepting state), and theres no
way out of that state. So all the transitions not shown in the picture above
go to that hidden state. With the hidden state drawn our automaton looks
like this:

0, 1
0
1

1
1
0 0, 1
3

This picture is quite a bit more complicated than the previous one, but
both describe the same DFA, and so contain precisely the same information.
I am perfectly happy for you to draw automata either way when it comes
to exam questions or assessed coursework.
Exercise 17. Consider the following DFA. Which of its states are dump
states, and which are unreachable? Draw the simplest automaton recognizing the same language.

b
a
c

a
c

a, b

b, c

a, b
Describe the language recognized by the automaton.
Exercise 18. Go through the automata you have drawn for Exercise 13
and 14. Identify any dump states in them.

3.5

Deterministic versus non-deterministic

So far we have found the following differences between deterministic and nondeterministic automata: For the same problem it is usually easier to design a
non-deterministic automaton, and the resulting automata are often smaller.
28

On the other hand, following a word through a deterministic automaton is

straightforward, and so deciding whether the word is accepted is easy. For
non-deterministic automata we have to find all the possible paths a word
might move along, and decide whether any of them leads to an accepting
state. Hence finding the language recognized by an NFA is usually harder
than to do the same thing for a DFA of a similar size.
So both have advantages and disadvantages. But how different are they
really? Clearly every deterministic automaton can be viewed as a nondeterministic one since it satisfies all the required criteria.
It therefore makes sense to wonder whether there are things we can do
with non-deterministic automata that cant be done with deterministic ones.
It turns out that this is false.
Theorem 3.1. For every non-deterministic automaton there is a deterministic one that recognizes precisely the same words.

Algorithm 1, example
Before looking at the general case we consider an example. Consider the
following NFA.

a
0

a, b

b
a

We want to construct a deterministic automaton from this step by step.

We start with state 0, which we just copy, so it is both initial and an accepting state in our new automaton.

0
With a, we can go from state 0 to states 1 and 2, so we invent a new state
we call 12 (think of it as being a set containing both, state 1 and state 2).
Because 2 is an accepting state we make 12 an accepting state too.

12

With a letter b we can go from state 0 to state 1, so we need a state 1

(think of it as state {1}) in the new automaton too.

1
b
0

12

Now we have to consider the states we have just created. In the original
automaton from state 1, we cant go anywhere with a, but with b we can go
to state 2, so we introduce an accepting state 2 (thought of as {2}) into our
new automaton.
29

1
b
0

12

To see where we can go from state 12 we do the following. In the original

automaton, with a we can go from 1 nowhere and from 2 to 0 and 2, so we
lump these together and say that from 12 we can go to a new state 02 (really
{0, 2}) with a. Because 2 is an accepting state we make 02 one too.

1
b
0

12

02

In the original automaton from state 1 with b we can go to state 2, and

from state 2 we can go to state 1 with the same letter, so from state 12 we
can go back to 12 with b.

1
b
0

12

02

b
In the original automaton with a we can go from state 2 to states 0 and
2, so we need a transition labelled a from state 2 to state 02 in our new
DFA.

1
b
0

12

2
a

02

b
With b from state 2 we can only go back to 1, so we add this transition
to the new automaton.

b
1
b
0

12

2
a

02

b
Now for the new state 02. From 0 we can go with a to states 1 and 2,
and from state 2 we can get to states 0 and 2, so taking it all together from
state 02 we can go to a new accepting state we call 012.

30

b
1
b
0

12

2
a

02

012

b
With b from state 0 we can go to state 1 in the old automaton, and from
state 2 we can also only go to state 1 with a b, so we need a transition from
the new state 02 to state 1 labelled b.

b
1
b
0

b
12

2
a
02

012

b
Following the same idea, from 012 with a we can go back to 012, and
with b we can go to 12.

b
1
b
0

b
12

2
a

a 02

a
012

So what we have done here? We have created a new automaton from

the old one as follows.
States of the new automaton are sets of states from the old one.
The start state for the new automaton is the set containing just the
initial state from
A state of the new automaton is an accepting state if one of its states
in the old automaton is an accepting state.
There is a transition labelled x from some state in the new automaton
to another if there are states of the old automaton in the two sets that
have a corresponding transition.
Exercise 19. For the automaton given below carry out the same process.
Hint: If you do find this very hard then read on a bit to see if that helps.

a
0

1
b
b

31

Algorithm 1, general case

We now give the general case.
Assume that we have an NFA (Q, q , F, ) over some alphabet . Mathematically, we have constructed a new automaton, a DFA this time, as
follows:
The states are given by PQ, where P is the powerset operator, so
PQ = {S | S Q}.
The start state is {q }.
A state S is an accepting state if and only if there is q S such that
q F , that is, q is an accepting state in the original automaton. Hence
the new set of accepting states is
F = {S Q | q S with q F }.
For a new state S PQ and a letter x the transition labelled
with x leads to the state given by the set of all states q for which
there is a q S such that there is a transition labelled x from q to q ,
that is
x
{q Q | q S with q - q }.
In other words, to see where the edge labelled x from S should lead
to, go through all the states q in S and collect all those states q Q
which have an edge labelled x from q.
Note that if there is no state in S which is labelled with x then (S, x) is
still definedit is the empty set in that case. Also note that the state given
by the empty set is always a dump state. Because there are no states in ,
(, x) = is true for all x , so it is a state one can never escape from.
Also, it cannot be an accepting state because it contains no accepting state
as an element.
We call the resulting automaton the DFA generated by the NFA
(Q, q , F, ).
For the above example, the new automaton has states , {0}, {1}, {2},
{0, 1}, {0, 2}, {1, 2} and {0, 1, 2}. In the image above we have kept our
labels shorter by only listing the elements of the corresponding set without
any separators or braces. For larger automata this would make it impossible
to distinguish between the state {12} and the state {1, 2}, but in such a small
example this is harmless.
However, not all the states we have just listed appear in the picture we
drew above. This is so because we used a method that only draws those
states which are reachable from the start state. We next draw the full
automaton as formally defined above.

a, b

b
1

b
0

01 ab

b
12

2
a

a 02
b

b
32

a
012

We already know that we may leave out dump states like when drawing
an automaton.
We have another extra state in this picture, namely {0, 1}. This is a
state we can never get to when starting at the start state {0}, so no word
will ever reach it either. It is therefore irrelevant when it comes to deciding
whether or not a word is accepted by this automaton. We call such states
unreachable and usually dont bother to draw them.
Exercise 20. For the NFA from the previous exercise draw the picture of
the full automaton with all states (given by the set of all subsets of {0, 1, 2},
including the unreachable ones) and all transitions.
Exercise 21. For each of the following NFAs, give a DFA recognizing the
same language.
(a)

a, b
0

1
b

(b)

a
a
a

b
1
(c)

a, b

2
b

a
0

a
2
a
b
b
a
b
1
b

a, b

It is possible to avoid drawing unreachable states by doing the following:

Start with the start state of the new automaton, {q }.
Pick a state S you have already drawn, and a letter x for which you
have not yet drawn a transition out of the state S. Now there must
be precisely one state S to which the transition labelled x leads.
If the state S has already been drawn, draw in the new transition
labelled x
If the state S has not been drawn yet, add it to the automaton
and draw in the new transition labelled x.
The DFA (PQ, {q }, F , ) defined above accepts precisely the same
words as the original NFA (PQ, {q }, F, ). This establishes Theorem 3.1,
which tells us that we can do precisely the same things with NFAs that we
can do with DFAs.
Note that we can use these automata also to encode information that at
first sight looks quite different from that of recognizing a language!
Exercise 22. You are asked to design a vending machine that accepts the
following coins: 10p, 20p, 50p, 1. Products handled by the machine cost
1. In order to keep track of when the machine has received one pound (or
more), draw a DFA.
33

We have three different ways now in which we can describe the same
language.
Exercise 23. Consider following language over the alphabet {a, b, c} which
consists of all words of odd length. Describe this language in the following
ways:
(a) Using a regular expression.
(b) Using a DFA.
(c) As a set. Unless you work through the Appendix you will probably find
this difficult.

3.6

From automata to patterns

We have three ways now of talking about collections of strings, or languages.

We can describe it set-theoretically, using a regular expression, or using a
(deterministic or non-deterministic) finite automaton. The second is particularly good when we have to talk to a computer, and sometimes its quite
easy to come up with the right regular expression. For more complex problems, however, its often easier (or at least less error-prone) to come up with
an automaton. But how do these last two descriptions connect?
If we have found an automaton for the language we want, how do we
communicate it to a computer? In this section we describe an algorithm
that takes a DFA and turns it into a pattern describing the same language.
Why do we need an algorithm? Sometimes we can just read off the
pattern from the automaton, even if they contain cycles.

a
b

a
b

Clearly any word that will be accepted has to start with a, can then
contain arbitrarily many further as, has b, and arbitrarily many further bs.
That gets the word into an accepting state. After that, the word may have
another a and then it all repeats.
A pattern describing the same language is
(aa bb )(aaa bb ) .
However, if the automaton is more complicated then reading off a pattern
can become very difficult, in particular if there are several accepting states.
If the automaton is moreover non-deterministic the complexity of the task
worsens furtherusually it is therefore a good idea to first convert an NFA
to a DFA using Algorithm 1 from page 32.
In order to show that it is possible to construct a regular expression
defining the same language for every automaton we have to give an algorithm
that works for all automata. This algorithm may look overly complex at first
sight, but it really does work for every automaton. If you had to apply it a
lot you could do the following:
Define a data structure of finite automata in whatever language youre
using. In Java you would be creating a suitable class.
34

 Implement the algorithm so that it takes as input an object of that

class, and produces a pattern accordingly.

Algorithm 2, first example

Because the algorithm required is complicated we explain it first using a
simple example. With a bit of experience you will probably be able to read
off a pattern from the automaton, but you wont be able to do this with
more complicated automata (nor will you be able to program a computer to
do the same).
Consider the following DFA.

0
c

1
a

2
There are two accepting states. Hence a word that is accepted will either
start in state 0 and end in state 0 or start in state 0 and end in state 2.
That means the language L accepted by this automaton is the union of two
languages which we write as follows:
L = L00 L02
The indices tell us in which state we start and in which state we finish. This is already a useful observation since we can now concentrate on
calculating one language at a time.
To calculate L00 we note that we can move from state 0 to state 0
in a number of ways. It is so complicated because there are loops in the
automaton. What we do is to break up these ways by controlling the use of
the state with the highest number, state 2, as follows:
To get from state 0 to state 0 we can
either not use state 2 at all (that is, go only via states 0 and 1) or
go to state 2, return to it as many times as we like, and then go from
state 2 to state 0 at the end.
At first sight this does not seem to have simplified matters at all. But
we have now gained control over how we use the state 2 because we can use
this observation to obtain the following equality.
1
1
1
1
L00 = L2
00 = L00 L02 (L22 ) L20

We have introduced new notation here, using L1

ij to mean that we go
from state i to state j while using only states 0 and 1 in between, that is,
states with a number 1. While our expression has grown the individual
languages are now easier to calculate.
L1
00 . We cannot go from state 0 to state 0 while only using states 0
and 1 in between. Hence this language is equal to {}. (You may have
expected the empty set here, but we are trying to concatenate words
encountered on this way with other words, and that means the empty
word is the right thing to put here.)

35

 L1
02 . To go from state 0 to state 2 without using state 2 in between
we must see a followed by b. Hence this language is equal to {ab}.
L1
22 . To go from state 2 to state 2 without using state 2 in between
there are two possibilities:
we can go from state 2 to state 1 and back, seeing ab along the
way or
we can go from state 2 to state 0 to state 1 to state 2 and see cab
along the way. (It would be a good idea now to convince yourself
that there is no other possibility.)
Hence this language is equal to {ab} {cab} = {ab, cab}.
L1
20 . The only way of getting from state 2 to state 0 using only
states 0 and 1 in between is the direct route, which means we must
see c. Hence this language is equal to {c}.
Putting all these together we get the following.
1
1
1
1
L00 = L2
00 = L00 L02 (L22 ) L20

= {} {ab} {ab, cab} {c}

We can now read a pattern4 off from this expression, namely

|ab(ab|cab) c,
and we have
L00 = L(|ab(ab|cab) c).
Note that this is not the simplest pattern we could possibly have used
(a(ba) bc) would also have worked. The advantage of the algorithm is that
it always gives you a pattern in the end, no matter how complicated the
automaton.
In this case the languages we were required to calculate were not so
difficult to work out. If we had started with a more complicated automaton
we would have had to employ the same trick again, namely taking another
state and controlling how that is used. The second example shows how this
works.
We do yet have to calculate the second language, L02 . Again we do so
by controlling when we use state 2, but this time it is slightly easier:
To get from state 0 to state 2 we can
go to state 2 and return to it as many times as we like.5
This leads us to the following equality.
1
1
L02 = L2
02 = L02 (L22 ) .

Moreover, we already have worked out the languages required here, so

we can continue directly to write the following.
4
If you find this difficult go back and do Exercises 1, 2, and 8 and have another look
at page 17.
5
It would be a good idea to work out why this is different from the case of language L00 .

36

1
L02 = L1
02 (L22 )

= {ab} {ab, cab}

From this we can read off a regular expression, namely ab(ab|cab) , and
we get
L02 = L(ab(ab|cab) ).
Altogether that means
L = L00 L02
= L(|ab(ab|cab) c) L(ab(ab|cab) )
= L(|ab(ab|cab) c|ab(ab|cab) )
If we like we can simplify that as follows.
L(|ab(ab|cab) c|ab(ab|cab) ) = L(|ab(ab|cab) (c|))
Alternatively, with a bit of experience one might read off this regular
expression directly from the automaton:
(a(ba) bc) (|a(ba) b)
Exercise 24. Carry out the algorithm just described for the following automaton. Use the notation with the various languages Lk
ji for the first two
or three lines at least, even if you can read off a pattern directly.

a
1

a
0

3
c

b
2
b

Algorithm 2, second example

Lets look at a more complicated example.

b
a
a, c

What is confusing about this automaton is not its size in terms of the
number of states, but the way the transitions criss-cross it.
In order to find all the words accepted by the automaton we have to
identify all the words that
when starting in state 0 end up in state 0. We call the resulting
language L00 . We also need all the words that
37

 when starting in state 0 end up in state 2. We call the resulting

language L02 .
In other words the language L recognized by the automaton can be
calculated as
L = L00 L02 .
This now allows us to calculate these two languages separately, which
already makes the task a little bit easier. Note that what we have done is to
split up the recognized language into all the languages of the form Lq q ,
where q is the start state and q ranges over all the accepting states (that
is all the elements of F ).
In general we would therefore write for the language L recognized by
some automaton (Q, q , F, )
[
L=
Lq q .
qF

But finding all the different paths that lead from 0 to 0, or from 0 to 2
is still pretty tough. The way we simplify that is by taking the state with
the highest index, namely 2, out of consideration as follows.
Every path from the state 0 to the state 0 can do one of the following:
It either doesnt use the state 2 at all or
it goes from the state 0 to the state 2, then goes back to the state 2
as often as it likes, and ultimately goes to the state 0.
At first sight this doesnt look like a very useful observation. But what
we have done now is to break up any path that starts at the state 0 and
finishes at the state 0 into a succession of paths that only use the state 2 at
controlled points.
We use the same notation as before: All words that follow a path that
goes from state 0 to state 0 while only using states 0 and 1 (but not state 2)
in between make up the language L1
00 . This works similarly for other
start or end states. Reformulating our last observation means then that
every word that follows a path from state 0 to state 0 satisfies one of the
following:
It either is an element of L1
00 or
it is an element of
1
1
L1
02 (L22 ) L20 .

That means that we have the equation

1
1
1
1
L00 = L2
00 = L00 L02 (L22 ) L20 .

While the equation may appear to make things more confusing at first
sight, we now have languages which we can more easily determine on the
right hand side of the equation.
We now have the choice between trying to determine the languages on
the right directly, or applying the same idea again.
L1
00 . How do we get from state 0 to state 0 only using states 0
and 1? The simple answer is that we cant move there, but we are
already there so L1
00 = {}.
38

 L1
02 . Going from state 0 to state 2 using only states 0 and 1 can be
done in two ways, either directly using the letter b or via state 1 using
ab or cb. Hence L1
02 = {b, ab, cb}.
L1
22 . This is more complicated. Instead of trying to work this out
directly we apply our rule again: When going from state 2 to state 2
using only states 0 and 1 we can either go directly from state 2 to
state 2 or we can go from state 2 to state 1, return to state 1 as often
as we like using only state 0, and then go from state 1 to state 2. In
other words we have.
0
0
0
0
L1
22 = L22 L21 (L11 ) L12 .
0
0
We now read off L0
22 = {b, ab}, L21 = {aa, ac, c}, L11 = {} and
L0
12 = {b}. That gives us

L1
22 = {b, ab} {aa, ac, c} {} {b}

= {b, ab} {aa, ac, c} {} {b}

= {b, ab, aab, acb, cb}.
One step in this calculation merits further explanation, namely ,
which is calculated to be {} on page 12.
L1
20 . This one we can read off directly, it is {a}.
Altogether we get
L00 = {} {b, ab, cb} {b, ab, aab, acb, cb} {a}
= L() L(b|ab|cb) (L(b, ab, aab, acb, cb)) L(a)
= L(|(b|ab|cb)(b|ab|aab|acb|cb) a).
See the top of page 17 for an explanation of the last step.
This leaves the calculation of L02 . We once again apply the earlier
trick and observe this time that a path from state 0 to state 2 will have to
reach state 2 (for the first time) using only states 0 and 1 along the way,
and then it can return to state 2 as often as it likes, giving
1
1
L02 = L2
02 = L02 (L22 ) .

Now L1
02 we already calculated above, it is equal to {b, ab, cb}. We also
know already that L1
22 = {b, ab, aab, acb, cb}. Hence
L02 = {b, ab, cb} {b, ab, aab, acb, cb} = L((b|ab|cb)(b|ab|aab|acb|cb) ).
Hence the language recognized by the automaton is
L00 L02
= L(|(b|ab|cb)(b|ab|aab|acb|cb) a) L((b|ab|cb)(b|ab|aab|acb|cb) )
= L(|((b|ab|cb)(b|ab|aab|acb|cb) a)|((b|ab|cb)(b|ab|aab|acb|cb) ))
= L((b|ab|cb)(b|ab|aab|acb|cb) (|a)),
and a regular expression giving the same language is
|(b|ab|cb)(b|ab|aab|acb|cb) (|a).
39

Algorithm 2, general case

It is time to generalize what we have done. It is useful to assume that the
states in the automaton are given by {0, 1, . . . , n}, and that 0 is the start
state. If we get an automaton that does not conform with this we can always
relabel the states accordingly. Then the language L of all the words accepted
by the automaton is equal to
[
[ n
L0i =
L0i ,
iF

iF

where L0i is the language of all words that, when starting in state 0 end
in state i. Since a word is accepted if and only if it ends in an accepting
state the above equation is precisely what we need.
We can now think of some L0i as equal to Ln
0i : the language of all
words that, when starting in state 0, end up state i is clearly the language of
all words that do so when using any of the states in {0, 1, . . . , n}. In general,
Lk
ji is the language of all those words that, when starting in state j end
in state i, use only states with a number less than or equal to k in between.
It is the languages of the form Lk
ji for which we can find expressions that
reduce k by 1: Any path that goes from state j to state i using only states
with numbers at most k will
either go from state j to state i only using states with number at most
k 1 in between (that is, not use state k at all)
or go from state j to state k (using only states with number at most
k 1 in between), return to state k an arbitrary number of times,
and then go from state k to state i using only states with number at
most k 1 in between.
Hence we have
k1
k1
k1
k1
Lk
ji = Lji Ljk (Lkk ) Lki .

We note that if j = k or i = k then in the above expression only two

different languages appear, namely
if j = k

then

k1
k1
Lji
= Lki

and

k1
k1
Ljk
= Lkk

if i = k

then

k1
k1
Lji
= Ljk

and

k1
k1
.
= Lkk
Lki

and

Thus we get slightly simpler expressions (compare L02 in the above example):
j1 j1
Lj
ji = (Ljj ) Lji
i1
i1
Li
ji = Lji (Lii )

Because the number of states involved in the paths in questions goes

down the languages on the right hand side of the equation are easier to
calculate. If we have to we can keep applying the above equation until
we have reached languages where the only state allowed on the way is a
state with number below 0that is, no state at all. Of course the overall
40

expression we have will become longer and longer, so it pays to read off
languages from the automaton as soon as that is possible.
Once we have reached paths which are not allowed to use any other
states we have the following.
For j = i:

if there is no transition from i to i

{}
{,
x
,
x
,
.
.
.
x
}
if the xi are all the labels on the
L1
=
1 2
l
ii

transition from i to itself.

For j 6= i:

L1
ji

if there is no transition from j to i

{x1 , x2 , . . . xl } if these are all the labels on the
=

transition from j to i.

There are a number of equations that allow us to simplify the expression

we obtain in this way.
L = L = L ;
= {} = {} ;
L = = L ;
{} L = L = L {}
({} L) = L = (L {}) .
Once we have an expression consisting entirely of the simple languages
of the form L1
ji we can insert regular expressions generating these, and use
the rules from page 17 to get one regular expression for the overall language.
The advantage of having a general algorithm that solves the problem is
that one can now write a program implementing it that will work for all
automata, even large ones.
Exercise 25. Using the construction just described find a pattern for the
following automata. Can you read off a pattern from each automaton? Does
it match the one constructed?
(a)

2
b
1

(b)

b
1

a
Exercise 26. Give regular expressions defining the languages recognized by
the following automata using Algorithm 2. Hint: Recall that the way you
number the states has an impact on how many steps of the algorithm you
will have to apply!

41

(a)

a
b
b
b

(b)

a
b

(c)

a, b
a

By giving the algorithm we have established the following result.

Theorem 3.2. For every DFA there is a regular expression such that the
language recognized by the DFA is the language defined by the regular expression.
We get the following consequence thanks to Theorem 3.1.
Theorem 3.3. For every DFA or NFA it is the case that the language it
recognizes is regular.

3.7

From patterns to automata

We have a way of going from an automaton to a pattern that we can communicate to a computer, so a natural question is whether one can also go
in the opposite direction. This may sound like a theoretical concern at first
sight, but it is actually quite useful to be able to derive an automaton from
a pattern. That way, if one does come across a pattern that one doesnt
entirely understand one can turn it into an automaton. Also, changing existing patterns so that they apply to slightly different tasks can often be
easier done by first translating them to an automaton.
For some patterns we can do this quite easily
Exercise 27. Design DFAs over the alphabet {a, b, c} that recognize the
languages defined by the following patterns.
(a) (a|b)cc.
(b) cc(a|b).
(c) aa|bb|cc.
(d) c(a|b) c.
Now assume that instead we want to recognize all words that contain a substring matching those patterns. How do you have to change your automata
to achieve that?
For slightly more complicated patterns we can still do this without too
many problems, provided we are allowed to use NFAs.
42

Exercise 28. Design NFAs over the language {0, 1} that recognize the languages defined by the following patterns.
(a) (00) |(01)
(b) (010) |0(11)
Now turn your NFAs into DFAs.
However, in general it can be quite difficult to read off an automaton from
a pattern.6 We therefore introduce an algorithm that works for all regular
expression. This algorithm is recursive, and it builds on Definition 1, making
use of the recursive structure of patterns. It is very easy to build automata
for the base cases. However, to build automata for the constructors of
patterns alternative, concatenation, and star, we need to be a bit cleverer.

Algorithm 3, example
Assume that we want to build an automaton for the regular expression a b
based on already having automata for the patterns a and b .
The latter are quite easily constructed:

Now we would somehow like to stick these automata together, so that

the first part accepts the a part of our word, and the second part the b
part. For that wed like to have a way of going from the left state above to
the right state above in a way that doesnt consume any of the letters of our
word, something like this:

That then is precisely what we do: We generalize our notion of automaton to include transitions labelled not by a letter from our alphabet, but by
the empty word .
Definition 10. Let be an alphabet not containing . An NFA with transitions over is an NFA over the alphabet that may have transitions
labelled with . Hence the transition relation relates pairs of the form (q, x),
where q is a state and x is either an element of or equal to , to states.
We now have to worry about what it means for an NFA with -transitions
to accept a word. Whenever there is a transition labelled with in the
automaton we are allowed to follow it without matching the next letter in
our word.
Definition 11. A word s = x1 xn over is accepted by the NFA
with -transitions (Q, q , F, ) over if there are states
q0 = q , q1 , . . . , ql
6

You can always write down some reasonably complex patterns and give it a go.

43

and, for all 1 i n, transitions

qmi1

- qm +1
i1

- ...

xi

- qm 1
i

- qm
i

as well as transitions

- q1
mn +1

qmn

- ...

- ql ,

such that m0 = 0 and such that ql is an accepting state. Here in each case
the number of -transitions may be 0. The language recognized by an
NFA with -transitions is the set of all words it accepts.
While we do need NFAs with -transitions along the way to constructing an automaton from a pattern we do not want to keep the -transitions
around since they make the automata in question much more confusing. We
introduce an algorithm here that removes the -transitions from such an
automaton. For that let be an arbitrary alphabet not containing .
Before we turn to describing the general case of Algorithm 3 we
investigate the algorithm that removes -transitions.

Algorithm 4, general case

This is the only time when it makes sense to give the general case and then
looking at an example.
Let (Q, q , F, ) be an NFA with -transitions over some alphabet . We
define an NFA (Q, q , F , ) without -transitions over the same alphabet as
follows:
Let
F = F {q Q | q = q0 , . . . , qn Q with
qi

- qi+1 for all 0 i n 1 and qn F }.

In other words we add to F those states for which we can get to an

accepting state by following only moves.
For q, q Q and x , the transition relation relates (q, x) to q if
and only if there are states q = q1 , q2 . . . qn = q such that
q = q1

- q2 qn2

- qn1

- qn = q .

In other words for x and q, q Q there is a transition labelled x

from q to q in if it is possible to get from q to q in by following an
arbitrary number of -transitions followed by one transition labelled x.
The resulting automaton may then contain unreachable states that can
be safely removed. It recognizes precisely the same language as the automaton we started with.

Algorithm 4, example
Here is an example. Consider the following automaton with -transitions.

44

We copy the states as they are, and create some new accepting states as
follows:
Pick a non-accepting state. If from there we can reach an accepting state
(in the original automaton) using only -transitions, we make this state an
accepting state.

For the example, this means the initial state and the third state from
the left are now accepting states.

We then copy all the transitions labelled with a letter other than from
the original automaton.

b
a

We now look at each of these

labelled a. We add a transition
is possible to get from state i to
followed by the transition labelled

transitions in turn. Take the transition

labelled a from state i to state j if it
state j using a number of -transitions
a.

This gives us two new transitions.

b
a
a

We now do the same for the transition labelled b.

This gives us one new transition, but we already have a transition from
the initial state to the third state from the left, so we can just add the new
label b to that.7
7

We could also make do with just one loop transition being drawnI only left in two
to make the process clearer.

45

a, b

b
a
a

Now we may safely remove the unreachable states and any transitions
involving them.

a, b

a
We now have an automaton without -transitions that accepts precisely
the same words as the origin.
Exercise 29. Turn the following automaton into one that does not have
-transitions.

There is further opportunity for practising this algorithm below.

Theorem 3.4. For every NFA with -transitions there is an NFA over the
same alphabet that recognizes the same language.

Algorithm 3, general case

We are now ready to look at the process by which we recursively turn a
pattern over some alphabet into an NFA with -transitions over the same
alphabet.8

The pattern
An automaton recognizing the language defined by this pattern is given
below.

An automaton that accepts no word at all.9

8
9

Go back to page 16 if you cant see how the following works.

We could leave out the transition entirely.

46

The pattern
An automaton recognizing the language defined by this pattern is given
below.

An automaton that accepts precisely the word .10

The pattern x for x

An automaton recognizing the language defined by this pattern is given
below.

An automaton that accepts precisely the word x.

For the next two cases assume that we have patterns p1 and p2 as well
as automata A1 and A2 such that Ai recognizes the language defined by pi .

Concatenation
An automaton recognizing the language defined by the pattern p1 p2 is given
below. Here we turn every accepting state of A1 into a non-accepting state
and draw an -transition from it to the start state of A2 . The only accepting
states in the new automaton are those from A2 .
A1

A2

A1

combine
to

A1

A1

A2

Alternative
An automaton accepting the language defined by the pattern p1 |p2 is given
below. We add a new start state and connect it with -transitions to the
start states of A1 and A2 (so these are no longer start states).

A1
combine to

A2

A1
A2

Kleene Star
We assume that we have a pattern p and an automaton A that recognizes
the language defined by p.
An automaton accepting the language defined by the pattern p is given
below. Given an automaton A we introduce a new start state. This state
10

Again we could leave out the transition and the right hand state, which is a dump
state.

47

is accepting, and it has an -transition to the old start state. Further we

introduce an -transition from each accepting state to the old start state.

becomes

Exercise 30. Refer to your solutions to Exercise 27 to do the following.

Design NFAs with -transitions for the following:
(a) All words over {a, b, c} that match the pattern (aa|bb|cc)(a|b)cc.
(b) All words over {a, b, c} that match the pattern ((a|b)cc)|(cc(a|b)).
(c) All words over {a, b, c} that match the pattern (aa|bb|cc) .
We illustrate the process with an example. We return to the pattern
a b for that.
For the pattern a we have the automaton

a
For this automaton we apply the Kleene star.

We get the corresponding automaton for the pattern b , so putting them

together gives the following.

We can now use Algorithm 4 for removing -transitions from page 44 to

obtain the following.

b
b

There are unreachable states which we may remove.

48

By coincidence this automaton is deterministicin most cases we would

have to apply Algorithm 1 from page 32 to obtain a DFA. This isnt the
smallest automaton for the purpose, and in general the algorithm will not
find that. Its usefulness comes from being able to cope with all patterns,
even the most complicated ones. The smallest automaton recognizing the
language L(a b ) is the following.

b
b

In practice strictly following Algorithm 3 from page 3.7 introduces rather

more -transitions than strictly necessary. In particular when concatenating
very simple automata, the -transition connecting them can often be safely
left out. I am happy for you to do so in assessed work.
Exercise 31. Construct DFAs that recognize the languages defined by the
following patterns.
(a) (ab|a)
(b) (01) |001
(c) ((00) 11|01)
(d) ((ab) b|ab )
Exercise 32. Carry out the following tasks.
(a) Use Algorithm 3 described on page 46 to construct an NFA with transitions that recognizes the language defined by the pattern (a|b) a(a|b).
(b) Take your result from (a) and remove the -transitions by applying
Algorithm 4 on page 44.
(c) Take your result from (b) and turn it into a DFA by using Algorithm 1
on page 32. How many states would your DFA have if you didnt remove
unreachable states first? How many states would your DFA have if you
didnt only draw the reachable ones?
We have given an idea of how to prove the following.
Theorem 3.5. For every regular language we can find the following recognizing it:
deterministic finite automaton;
non-deterministic finite automaton;
non-deterministic finite automaton with -transitions.
Altogether, we have shown in various parts of this section the following
result.
Theorem 3.6. The following have precisely the same power when it comes
to describing languages:
regular expressions,
deterministic finite automata,
non-deterministic finite automata.
Hence for every regular language we can find a description using any one of
the above, and given one description we can turn it into any of the others.
49

3.8

Properties of regular languages

In Section 2 there are examples of how we can build new regular languages
from existing ones. At first sight these may seem like theoretical results
without much practical use. However, what they allow us to do is to build
up quite complicated languages from simpler ones. This also means that we
can build the corresponding regular expressions or automata from simple
ones, following established algorithms. That makes finding suitable patterns
of automata less error-prone, which can be very important.
If our language is finite to start with then finding a pattern for it is very
easy.
Exercise 33. Show that every finite language is regular.
Assume we have two regular languages, L1 and L2 . We look at languages
we can build from these, and show that these are regular.

Concatenation
In Section 2 it is shown that the concatenation of two regular languages
is regular. If L1 and L2 are two regular languages, and p1 is a regular
expression defining the first, and p2 works in the same way for the second,
then
L1 L2 = L(p1 ) L(p2 ) = L(p1 p2 ),
so this is also a regular language. There is a description how to construct
an automaton for L1 L2 from of those for L1 and L2 on page 46.

Kleene star
Again this is something we have already considered. If L is a regular language then so is L . If p is a regular expression for L then p is one for L ,
and again our algorithm for turning patterns into automata shows us how
to turn an automaton for L into one for L .

Reversal
If s is a word over some alphabet then we can construct another word over
the same alphabet by reading s backwards, or, in other words, reversing it.
For a language L over we define
LR = {xn xn1 x2 x1 | x1 x2 xn1 xn L}.
Exercise 34. Here are some exercises concerned with reversing strings
that offer a good opportunity for practising what you have learned so far.
Parts (a) and (e) require material that is discussed in Appendix A, so it only
makes sense to do these if you are working through this part of the notes.
(a) Define the reversal of a string as a recursive function.
(b) Look at an automaton for the language of all non-empty words over
the alphabet {a, b} which start with a. How can it be turned into one for
the language of all words which end with a? Hint: How could we have a
given word take the reverse path through the automaton than it would do
ordinarily?

50

(c) Look at the language given in Exercise 9 (i). What do the words in its
reversal look like? Now look at an automaton for the given language and
turn it into one for the reversed language. Hint: See above.
(d) In general, describe informally how, given an automaton for a language
L, one can draw one for the language LR .
(e) Take the formal description of a DFA recognizing a language L as in Definition 5 and turn that into a formal definition for an NFA with -transitions
which recognizes the language LR .

Unions
If we have regular expressions for L1 and L2 , say p1 and p2 respectively, it is
easy to build a regular expression for L1 L2 , namely p1 |p2 . But how do we
build an automaton for L1 L2 from those for L1 and L2 ? We have already
seen how to do that as wellform an NFA with a new start state which has
-transitions to the (old) start states of the two automata, as illustrated on
page 46. If we like we can then turn this NFA with -transitions into a DFA.

Intersections
This isnt so easy. Its not at all clear how one would go about about it
using patterns, whereas with automata one can see how it might work. The
problem is that we cant say first get through the automaton for L1 , then
through that for L2 : When we have followed the word through the first
automaton it has been consumed, because we forget about the letters once
we have followed a transition for them. So somehow we have to find a way
to let the word follow a path through both automata at the same time.
Lets try this with an example: Assume we want to describe the language
L of all words that have an even number of as and an odd number of bs.
Clearly
L = {s {a, b} | s has an even number of as}
{s {a, b} | s has an odd number of bs}.
DFAs for those two languages are easy to construct.

a
b
a
0

b
a

1
b

So how do we get an automaton that checks for both properties at the

same time? Well, we think of the first automaton as telling us whether the
number of as is even so far, or odd, and similarly for the second automaton,
and the number of bs. What we need to do in the automaton we want to
construct then is to keep track of the as and bs at the same time. So we
need four states:
(as even, bs even), (as even, bs odd), (as odd, bs even), (as odd, bs odd).
The automaton we want looks as follows:
51

b
00

01
b

a
b

10

11
b

What have we done? We have formed pairs of states, a state from the
first automaton and a state from the second one. We have then added
transitions that literally do follow both automata at the same time.
In general, given DFAs (Q1 , q1 , F1 , 1 ) and (Q2 , q2 , F2 , 2 ) we can form
an automaton that recognizes the intersection of the languages recognized
by the two DFAs as follows.
States: Q1 Q2 .
Start state: (q1 , q2 ).
Accepting states: F1 F2 .
Transition function: maps (q1 , q2 ) and x to ((q1 , x), (q2 , x)). In
other words, there is a transition
x

- (q , q )
1 2

(q1 , q2 )
if and only if there are transitions
q1

- q
1

and

q2

- q .
2

We call the result the product of the two automata. If we do need

a pattern for the intersection we can now apply Algorithm 2 from page 40.
Exercise 35. Use this construction to draw DFAs recognizing the following
languages over the alphabet {a, b}. First identify the two languages whose
intersection you want to form, then draw the automata for those languages,
then draw one for their intersection.
(a) All non-empty words that begin with a and end with b.
(b) All words that contain at least two as and at most one b.
(c) All words that have even length and contain an even number of as.
(d) All words that have length at least 3 and whose third symbol (if it
exists) is a.

Complements
If L is a language over the alphabet then we may want to consider its
complement, that is
L,
the set of all words over that do not belong to L.
Exercise 36. This is a question concerned with the complement of a language.

52

(a) Consider the language discussed on page 21 of all words which have a
0 in every position that is a multiple of 3. Begin by defining the complement of this language in {0, 1} , either using set-theoretic notation
or in English.
Now take the DFA for the original language given on page 22 and turn
it into one for the complement.
(b) Consider the following DFA that recognizes the language of all words
over {0, 1} of length at least 2 whose first letter is a 0 and whose second
letter is a 1.

0, 1
0

Again, describe the complement of this language either using sets or in

English. Then turn this DFA into one that recognizes this complement.
(c) Assume you are given a picture that describes a DFA for some arbitrary
regular language. Describe in general how you can turn this into a
picture for a DFA that recognizes the complement of that language.
(d) Now assume you are given a DFA in the formal description as used in
Definition 5, (Q, q , F, ). Describe how to turn this into a formal description of a DFA recognizing the complement of the original language.
(Again you may want to consult Appendix A for this part.)

3.9

Equivalence of Automata

Here is an issue we have not thought about so far. Is there a way of telling
whether two automata recognize the same language? You might particularly
care about this if you and a friend have created different automata for the
same problem and youre not sure whether youre both correct. We say that
two automata are equivalent if they recognize the same language. Note
that if we have an NFA, and we carry out Algorithm 1 for it, then the
resulting DFA is equivalent to the NFA we started with.
If youre lucky, this is easy to decide: If you have both drawn a picture of
the same automaton, but have merely put different names in your states,
then the automata are what we call isomorphic. It is easy to show that
isomorphic automata recognize the same language.
There are several ways of addressing the more general issue.

Via minimization
Given a DFA it is possible to minimize the number of states it has. There is
an algorithm which does this, and this calculation has very nice properties.
In particular two DFAs recognize the same language if and only if their
minimized automata are isomorphic.
However, the algorithm is non-trivial, and showing that it does what it
needs to do is fairly involved. For this reason we do not give any detail
here.11 Note that if you have NFAs you first have to find equivalent DFAs.
11

Ask me for recommendations if you would like to read more about this! This algorithm
is taught in the third year course on compilers.

53

Via complementation and intersection

We actually have a method which allows us to calculate whether two automata define the same language. Assume we have automata A And A ,
which recognize the languages L and L respectively. We are interested in
the question whether L = L . Note that
L = L

if and only if

L L and L L

if and only if L ( L ) = and L ( L) = .

But we know how to construct automata for the complement of a language, and we also know how to construct automata for the intersection
of two languages, so we can reduce our question to that whether the two
automata corresponding to the two languages appearing in the above accept
no words at all.

Via bisimulation
Note that both the methods for deciding equivalence of automata we have
mentioned so far work for DFAs, but not for their non-deterministic relations. Clearly the question of whether two NFA recognize the same language
is even harder than that for two DFAs. Nonetheless there is an idea that
helps with this.
The idea is this: Assume we have two NFAs, say A = (Q, q , F, ) and
B = (P, p , E, ). If we can show that for each state q of A there is an
analogous state p of B, and vice versa then they should recognize the same
language if only we can get our definition of analogous right.
Definition 12. We say that a relation from Q to P is a bisimulation
between automata A and B if and only if
relates q and p ;
if relates q Q with p P then
x
- q then there exists p P such that p
if q
relates q and p

x
- p then there exists q Q such that q
if p
relates q and p ;

- p and
- q and

if relates q Q with p P then (q F if and only if p E).

Why are bisimulations useful? It is because of the following result.
Theorem 3.7. Two NFAs are equivalent if and only if a bisimulation exists
between them.
Here is an example. Consider the following two automata.
b
a

a
a

a
a

We claim that they define the same language, and we demonstrate this by
showing there is a bisimulation between them. We draw the two automata
above each other and show which states the bisimulation relates using grey
(instead of black) lines.
54

b
a

a
a

a
a

b
a
a

Exercise 37. For the following automata, either find a bisimulation, or

argue that none exists.
a

(a)

b
0

(b)

0
1

0 1

3.10

Limitations of regular languages

So far we have assumed implicitly that all languages of interest are regular, that is, that they can be described using regular expressions or finite
automata. This is not really true, but the reason regular languages are so
important is that they have nice descriptions, and that they suffice for many
purposes.
Something a finite automaton cannot do is to countor at least, it cannot count beyond a bound defined a priori. Such an automaton has a finite
number of states, and its only memory is given by those states. Hence it
cannot remember information that cannot be encoded into that many states.
If we want an automaton that decides whether or not a word consists
of at least three letters then this automaton needs at least four states: One
to start in, and three to remember how many letters it has already seen.
Similarly, an automaton that is to decide whether a word contains at least
55 as must have at least 56 states.
However, if we try to construct an automaton that decides whether a
word contains precisely as many 0s as 1s, then we cannot do this: Clearly
the automaton must have a different state for every number of 0s it has
already seen, but that would require it to have infinitely many states, which
is not allowed.
Similarly, how would one construct a pattern for the language
L = {0n 1n | n N}?
We can certainly cope with the language {(01)n | n N} by using the
pattern (01) , but that is because once we have seen 0, and the subsequent 1,
we may forget about it again (or return to the start state of our automaton).
In order to describe L, on the other hand, we really have to remember how
many 0s there are before we find the first 1. But there could be any number
55

of 0s, so our automaton would have to have arbitrarily many statesor,

if we built a pattern |01|0011|000111| it would be infinite. This is not
allowed.
There is a result that allows one to conclude that a particular language is
not regular, the Pumping Lemma. Looking at that is beyond the scope of
this course, but if you really want to understand what this issue you should
look this result up. There are automata that do have a memory device which
allows them to cope with such languages, but we do not treat them here.
Exercise 38. Which of the following languages are regular? If you answer
yes, prove it. If it is no, try to say why.
(a) The language of all words that contain at least as many as as bs.
(b) The language of all strings of brackets that are balanced (so scanning
the word from left to right we cannot have more closing brackets at any time
than we have seen opening brackets, and for the whole word the number of
closing brackets is equal to the number of opening brackets).
(c) The language of all strings over the alphabet {0, 1} which, when viewed
as a binary number, are divisible by 3.
(d) The language of all strings over the alphabet {0, 1} which, when viewed
as a binary number, are prime numbers.
(e) The language of all non-empty strings over the alphabet {0, 1, . . . , 9}
such that the last symbol is one that occurred earlier in the string.
(f) The language of all strings over {0, 1} where the number of 0 is different
from the number of 1s.
(g) The language of all strings over {a, b} such that each consecutive block
of five letters contains at least two as. (Note that these blocks do overlap.)
(h) The language of all palindromes over {0, 1}, that is, all strings that read
forwards as backwards or, to put it differently, the strings which are equal
to their reversal.
(i) The language of all strings over {a, b} such that there are two 0s in the
string which are separated by a number of characters that is a multiple of 4.
(j) The language of all odd-length palindromes over {a, b, c}.
(k) The language of all words over the alphabet {a, b} that have the same
number of occurrences of the substring ab as that of the substring ba. The
word bab does belong to that language, since it has one occurrence of the
substring ab ([ab]a) and one of the substring ba (a[ba]). *Does your answer
change when the alphabet is changed to {a, b, c}?
(l) The set of all words of the form s1n , where s is a word over {a.b.c}, and
n is the number of letters in s.

3.11

Summary

We can use finite automata to describe the same languages as those

given by regular expressions, that is, the regular languages.

56

 Automata can be deterministic (DFA) or non-deterministic (NFA).

Every DFA can be viewed as an example of an NFA, and for every NFA
we can construct a DFA that recognizes precisely the same words.
Given a DFA we can construct a regular expression such that the
language defined by this expression is precisely the language recognized
by the automaton we started with.
Given a regular expression we can construct an NFA with -moves
such that the words accepted by the automaton are precisely those
that match the regular expression. We can then eliminate the -moves
from the NFA, and use the algorithm mentioned above to turn that
into a DFA that recognizes the same language.
As a result, what we can express using regular expressions, or DFAs,
or NFAs, are precisely the same languages.
We can form the union, intersection, complement, reversal, concatenation and Kleene star operation on regular languages and we will then
get another regular languages.
Not all languages are regular. In particular those languages for which
an automaton would have to count up to a boundless number are not
regular.

57

Chapter 4

Describing more complicated

languages
At the end of Section 3 it is demonstrated that regular expressions are not
sufficient to describe all the languages we are interested in. In particular we
cannot even generate the language of all balanced bracketsclearly that is
something we have to do when trying to cope with programming languages.
How is a compiler to break up a Java program if it cannot cope with checking
whether the braces {, } line up properly?
In this section we introduce a way of talking about languages that is
more general. However, it is also more complicated, so whenever possible it
is a good idea to stick with regular languages.

4.1

Generating words

Instead of thinking of patterns that may by matched by strings it is our aim

here to generate all the words in our language following some well-defined
rules. The way of reading the rules is to see them as ways of rewriting the
string generated so far, by using one of the rules given to to change one
symbol into a string of others. These rules are given by what we call a
grammar.
Consider the following example:
B
B
B
B
B
B
B
B
B
B

0
1
2
3
4
5
6
7
8
9

S
S
S
S
S
S

B
(S)
S+S
SS
SS
S/S

We have to know where to start, and we always use S as the only symbol
we may create from nothing. Once we have S, we can use any of the rules on
the right hand side to replace it by a more complicated string, for example
S S S S (S) S (S S) B (S S) 5 (S S)
5 (B S) 5 (B B) 5 (3 B) 5 (3 4).
58

We call this a derivation from the grammar.

Exercise 39. Write out derivations for 3/4 + 5, and 9. Can you write a
derivation of 12? Write out a different derivation for 3/4 + 5. How many of
these can you see?
In other words, a grammar is a set of rules that allows us to build certain
expressions in a purely mechanical manner. We could program these rules
into a computer and ask it to begin to generate strings from it, but unless
we were careful regarding what exactly we told it it might build one very
long string until it ran out of memory.
Note that the only words we are really interested in are those which
consist entirely of the digits, and the arithmetic operatorsthey no longer
contain either S or B. We merely need these two symbols to help us build the
string in question: They can be seen as placeholders for longer expressions
that havent yet been built. We call the letters we want our final strings to
be built of terminal symbols (because they are the only ones there at the
end). Once we have introduced a terminal symbol it cannot be replaced
by another. The other symbols are called non-terminal symbols (because
they are not terminal). They are also sometimes called variables. Think
of them as auxiliary symbols, or placeholders: Once they have done their
job they disappear. The rules for the grammar only apply to non-terminal
symbolsthere can never be a rule that allows us to rewrite a terminal
symbol.
Exercise 40. Try to write down a grammar with terminal symbols {0, 1}
for all non-empty words that start with 0 and end with 1. Show how your
grammar generates the strings 01 and 001 and convince yourself that it does
not allow you to generate invalid strings such as 11 or 00. Hint: If you find
that difficult at this stage then wait until you have seen the formal definition
for a grammar and another example or two.
We find the above description of a grammar a bit lengthy, and often use
an abbreviated form of giving the rules that tell us how we may rewrite a
string. For the example this looks as follows.
B

0|1|2|3|4|5|6|7|8|9

B | (S) | S + S | S S | S S | S/S

What we have done is to group together all the ways in which we may
replace either of the two symbols B and S.

4.2

Context-free grammars

It is time for a formal definition.

Definition 13. A context-free grammar or CFG is given by the following:
an alphabet of terminal symbols, also called the object alphabet;
an alphabet of non-terminal symbols, the elements of which are also
referred to as auxiliary characters, placeholders or, in some books,
variables, with = ;
59

 a special non-terminal symbol S called the start symbol;

a finite set of production rules of the form R X where R
is a non-terminal symbol and X ( ) is an arbitrary string of
terminal and non-terminal symbols, which can be read as R can be
replaced by X.1
In the example above, the object alphabet is
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, +, , /, , (, )}.
The non-terminal symbols are B and S.
Why do we call grammars whose production rules satisfy these conditions
context-free? It is because the rules tell us that we may replace any occurrence of a non-terminal as directed. A context-sensitive grammar would
be able to have rules such as You may replace the non-terminal A by the
string (A) provided that there is an S immediately to the left of A, so a
production rule might look like this: SA S(A).
In general, what context-sensitivity means is that according to their
production rules strings consisting of terminal and non-terminal symbols
can be replaced by other such strings. The advantage of context-freeness is
that we know that once a terminal character is in place, it will remain in
the word even if further reduction rules are applied. Further, we dont have
to worry about having to choose between different production rules taking
us in different directions to the same degree.
Assume we had rules
AB A

and

AB B.

Given AB occurring in some string we would then have to make a choice

whether to keep the A or the B. For context-free grammars, if we have two
distinct non-terminal symbols (such as A and B here) we can apply any one
rule for A, and any one rule for B, and it does not matter which order we
do that in.
If we allowed context-sensitive rules then our grammars would be much
more complicated, and while it would be possible to express more complicated features it would be much harder to see, for example, which language
a grammar generates. Natural languages certainly require context-sensitive
grammars, but when it comes to programming we can go a very long way
without having to use this idea.
The following definition says how we may generate strings from a grammar, and how a grammar defines a language.
Definition 14. A string X ( ) is generated by a grammar G if
there is a sequence of strings
S = X0 X1 Xn = X
such that each step Xi Xi+1 is obtained by the application of one of Gs
production rules to a non-terminal occurring in Xi as follows.
Let R occur in Xi and assume that there is a production rule R Y .
Then Xi+1 is obtained from Xi by replacing one occurrence of R in Xi by
the string Y .
The language generated by a grammar G is the set of all strings
over which are generated by G.2
1

This assumes that we have a special symbol, namely which should not be contained
in either or to avoid confusion.
2
Note that such a string may not contain any non-terminal symbols.

60

Let us look at another example. Let us assume that we want to generate

a grammar for the language of all strings that are non-empty and start with
0 over the alphabet {0, 1}. How do we go about it? Well, we know what
our underlying alphabet of terminal symbols is going to be. How about
non-terminal ones? This is harder to decide a priori. We know that we need
at least one of these, namely the start symbols S. Whether or not further
non-terminal symbols are required depends on what it is that we have to
remember when creating a word (just as the number of states required for
a finite automaton depends on what we need to remember about a word3 ).
How do we make sure that our word starts with 0? Well, we have a
production rule for S that says S 0 then we certainly create 0 as the
left-most symbol, and this will stay the left-most symbol. But what should
the dots be?
If we try S 0S then we do get a problem: Eventually we have to allow
1s into our string, but if we have a rule that says S 1S then we could
generate words whose first letter is 1. An obvious solution to this problem
is to use a new non-terminal symbol, say A. Then we can have a rule
S 0A,
and all we now have to do is to allow A to create whatever letters it likes,
one at a time, or to vanish into the empty string. Hence we want rules
A 0A | 1A | .
Exercise 41. Can you create a CFG for the same language using just one
non-terminal symbol? Hint: If you find this tough now then come back to it
after you have had more practice.
Exercise 42. This exercise asks you to create CFGs for various languages
over the alphabet {0, 1}.
(a) All words that contain two consecutive 1s. Now give a derivation for
the word 0111. Can you create a grammar for this language with just one
non-terminal symbol?
(b) All words that contain at least two 1s. Now give a derivation for 01010.
(c) All words whose length is precisely 3.
(d) All words whose length is at most 3.
(e) All words whose length is at least 3.
(f) All words whose length is at least 2 and whose last but one symbol is 0.
Now give a derivation for 0101.
(g) All words for which every letter at an even position is 0. Give a derivation of 101, and of .
(h) All words consisting of an arbitrary number of 0s followed by the same
number of 1s. Try to explain how to generate a word of the form 0n 1n where
n N.
(i) All words that contain an even number of 0s. Make sure that your
language can generate strings such as 001010 and 0001001000.
3

However, in a finite state machine we only get to scan the word from left to right;
when creating a derivation we may be working on several places at the same time.

61

(j) All words that do not contain the string 01.

Which ones of those are easy to do, which ones harder? Look at other
languages from Exercises 9 and 10. Which ones of those, do you think,
would be particularly hard?
Part (h) of this exercise already indicates that we can express languages
using CFGs that we cannot express using finite automata or regular expressions.
We give another example. Assume we want to generate the set of all
words of the language
{s1n | s {a, b, c} , n N and n is the number of letters in s}.
This cannot be a regular language since it would require an automaton
to remember how many letters it has seen, and that number might rise above
any bound (compare Exercise 38.
Our grammar has to be such that every letter is matched by 1, so we
have to create them simultaneously. If we use = {a, b, c, 1} and = {S}
the following production rules will do just that.
S aS1 | bS1 | cS1 | .
Here is another exercise that asks you to find grammars for other nonregular languages (compare this with Exercise 38).
Exercise 43. Give CFGs for the following languages.
(a) The language of all strings of brackets that are balanced (so we cannot
have more closing brackets at any time than we have seen opening brackets,
and for the whole word the number of closing brackets is equal to the number
of opening brackets).
(b) The language of all palindromes over {a, b, c}.
(c) The language of all odd-length palindromes over {a, b, c} whose middle
letter is a.
(d) The language
{ai bj ck | i, j, k N, k = i + j}.
We have a name for the languages that can be described by a context-free
grammar.
Definition 15. A language L over is context-free if there is a contextfree grammar whose alphabet of terminal symbols is such that L is generated by the grammar
Exercise 44.* Consider the following grammar. = {a, b}, = {S},
S SaSaSbS | SaSbSaS | SbSaSaS | . Argue that this CFG generates
the language of all words that contain precisely twice as many as as bs.
Exercise 45. Design a CFG for the language consisting of all words that
contain precisely as many as as bs. Try to argue that you really can generate
all such words using your production rules. Hint: You could analyse the
language from the previous exercise. Alternatively, try to come up with a
way that keeps track of the number of as or bs generated so far.
62

4.3

Regular languages are context-free

We have already seen that we can capture non-regular languages using

context-free grammars. It may seem fairly clear that CFGs are more powerful than regular expressions, but how do we know that we can express every
regular language using a CFG?
This turns out not to be so very difficult. Maybe surprisingly it is easier
here to start with a DFA than with a regular expression.
Assume that we have a regular language L over some alphabet . By
Theorem 3.5 we can find a DFA, say (Q, q , F, ), that recognizes this language.
We now define a CFG from the automaton as follows:
We use as the alphabet of terminal symbols.
We use = Q as the alphabet of non-terminal symbols.
We use S = q as the start symbol.
For every transition
x

- q

in the automaton (that is, for every pair (q, x) Q and q = (q, x))
we introduce a production rule
q xq .
For every accepting state q F we add a production rule q .
We recall the following automaton from Section 3 where we have changed
the names of the states.

0
S

A
0

We can now use non-terminal symbols S and A as well as terminal symbols 0 and 1 for a CFG with the following production rules:
S

| 1S | 0A

1A | 0S

This grammar will generate the language of those words consisting of 0s

and 1s which contain an even number of 0s.
Exercise 46. Use this construction of a context-free grammar to produce
grammars for the regular languages defined as follows:
(a) By the pattern (ab|a) . Hint you should already have a DFA for this
pattern from Exercise 31.
(b) (01) |001 Hint: as above.

63

(c) The words that do not contain the string 010. Hint: This language
appears in Exercises 10 and 14 so you already have a suitable DFA. Could
you have found such a grammar without using this method?
(d) By the pattern ab c |a b c. Could you have found such a grammar
without using this method?
We note that the production rules we have used here are very limited:
for an arbitrary non-terminal symbol R they are of the form R xR ,
where x and R , or R x, where4 x , or R . We call
grammars were all production rules are of this shape right-linear. Such
grammars are particularly simple, and in fact every language generated by
a right-linear grammar is regular.
We can do even more for regular languages using CFGs. This is our first
truly applied example of a context-free grammar.
There is a context-free grammar for the language of regular expressions
over some alphabet .
The underlying alphabet of terminal symbols5 we require here is
{, , |, }.
The alphabet of non-terminal symbols can be {S}. We use the following
production rules:
S .
S .
S x, for all x .
S SS for concatenation.
S S|S for alternative.
S S for the Kleene star.

4.4

Parsing and ambiguity

When a compiler deals with a piece of code it has to parse it. In other
words, it has to break it down into its constituent parts in a way that makes
sense. Similarly, if you were to write a program that could take (possibly
quite long) arithmetic expressions as input and evaluate it, it would have to
break down the expression to decide in which order to carry out the various
operations.
When we give a derivation we automatically provide one way of breaking
down the given string. Parsing is an attempt of carrying out the opposite
process, namely to take a string and find out how to assemble it from simpler
parts. Instead of finding a derivation it is often more meaningful to create a
parse tree which gives a better overview of how the various bits fit together.
4
5

We did not need that rule above, but we need it to be part of this definition.
It would not be a good idea to call this alphabet this time.

64

S
S

S
S

We can read off the string generated from the leaves of the tree,6 here it
is 5 (3 4), and we can also see which production rules to use to get this
result. If we compare this with a derivation like that given on page 58 we
can see that in general, there will be several derivations for every parse tree.
This is because the parse tree does not specify in which order the rewriting
of non-terminal symbols should be carried out.
If we have S S, should we first replace the left S by B, or the right
one by (S)? The answer is that it doesnt really matter, since we can do
this in either order, and the strings we can get to from there are just the
same. What the parse tree does then is to remember the important parts
of a derivation, namely which rules to apply to which non-terminals, while
not caring about the order some of these rules are applied.
Any non-trivial word in a context-free language typically has many different derivations. However, ideally we would like every word to only have
one parse tree for reasons explained below.
Consider the string 5 3 4. This has the following two parse trees.
S

S
S

S
S

Now if a computer (or a calculator) is to decide how to calculate this

number it has to break down the string. The parse trees above say that this
can be done in two ways, which we might write as (5 3) 4 and 5 (3 4).
The first is calculated as 15 4 = 11, while the second gives 5 (1) = 5,
so they really do give rise to quite different things.
Of course we have generally adopted the convention that it is the first
expression that we mean when we write 534, because the convention says
that we should multiply before we add. But if we look at our grammar then
there is nothing that specifies that a computer should treat this expression
in this way.
If there are two different parse trees for the same word we say that it
is ambiguously generated and we say that the grammar is ambiguous.
When designing grammars for languages that require that the generated
6

Note that may appear as a leaf, in which case we ignore it from reading off the word
in question.

65

words are evaluated in some way then one should always aim to give a unambiguous grammar. However, it is not always possible to do this, and if
that is the case then we call the language in question inherently ambiguous.
We can change the grammar given above in such a way that every word
it generates can be parsed unambiguously as follows.
B

0|1|2|3|4|5|6|7|8|9

B | (S + S) | (S S) | (S S) | (S/S)

Now when we derive a word we automatically say in which order the

various operations should be carried out because there are brackets doing
this for us.7 Note that a word still has a number of different derivations, it
is the number of parse trees that should be equal to one.
The first thing a compiler has to do when given a program is to recognize
various key words, which are often called tokens. Examples for tokens for the
Java programming languages are words such as if, else, public, class,
and many others, but also the names of variables and methods. This is
referred to as a lexical analysis, and this is typically carried out using regular
expressions. The Unix command lex generates such analysers.
Once this is done the compiler has to parse the programin other words,
it produces a parse tree for the program. If that cannot be done then
the compiler will raise a syntax error. There are some standard tools to
help with this kind of thing: The Unix YACC command takes a context-free
grammar and produces a parser for that grammar. Clearly for a program
it is particularly important that it can be uniquely parsed so that there is
precisely one way for the computer to carry out the constructions contained
in it. Because programmers do not like to put in a lot of brackets typically
a compiler will have a list of which operations should be carried out first.
We look at another example of ambiguity. Consider the following grammar.
S

aT | T a

aT | bT |

The word aa has two parse trees for this grammar:

S
a

S
T

T
T

a
T

Hence we know that this grammar is also ambiguous. How can we turn
it into an unambiguous one? First of all we have to find out which language
is described by the grammar. In this case this is not so difficult: It is the
set of all words over the alphabet {a, b} which start with a or end with a
(or both). The ambiguity arises from the first rule: The a that is being
7

The usual convention that tells us that multiplication should be carried out before
addition is there so that we do not have to write so many brackets for an expression while
it can still be evaluated in a way that leads to a unique result.

66

demanded may either be the first, or the last, letter of the word, and if the
word starts and ends with a then there will be two ways of generating it.
What we have to do to give an unambiguous grammar for the same
language is to pick the first or last symbol to have priority, and the other
one only being relevant if the chosen one has not been matched. A suitable
grammar is as follows:
S

aT | bU

aT | bT |

aU | bU | a

Now we generate any word from left to write, ensuring the grammar is
unambiguous. We also remember whether the first symbol generated is a,
in which case the remainder of the word can be formed without restrictions,
or b, in which case the last symbol has to be a. It turns out that we do not
need three non-terminal symbols to do this: Can you find a grammar that
uses just two and is still unambiguous?
Exercise 47. For both grammars over the alphabet = {a, b} given below
do the following: Show that the grammar is ambiguous by finding a word
that has two parse trees, and give both the parse trees. Now try to determine the language generated by the grammar. Finally give an unambiguous
grammar for the same language.
(a) Let = {S, T } with production rules S T aT and T aT | bT | .
(b) The grammar with = {S} and S aS | aSbS | .

4.5

A programming language

Defining a programming language is a huge task: First we have to describe

how programs can be formed (we have to define the syntax of the language),
and then we have to describe how a computer should act when given a
particular program (we have to give a semantics).
The syntax of a language tells us how to write programs that are considered well-formed (so if it is a compiled language, for example, the compiler
will not give any syntax errors). However, this says nothing about what any
program actually means. When you see a fragment of a program you probably have some kind of idea what at least some of the instructions within it
are supposed to do, but that is because program designers stick to certain
conventions (so that a + b will typically mean adding the content of the
variable a to that of the variable b), but a computer doesnt know anything
about these preconceptions. Therefore in order to fully define a programming language one also has to say what a program actually means. There
is a third year course unit that concerns itself with different ways of doing
so.
Most real programming languages such as Java, C, or ML, or PHP, typically have very large grammars and we do not have the time here to study
such a big example.
We look at a very small programming language called While. At first
sight it looks as if there cant possibly anything interesting to be computable
with it: It only knows about boolean data types (such as true, T , here tt,
and false, F , here ff), and natural numbers, say from 0 to some number
67

we call maxint. We also need variables which can take on the value of any
of our natural numbers. Lets assume for the moment we only have three
variables, x, y, and z. The language has three kinds of entities:
arithmetic expressions A
boolean expressions B and
statements S.
For each of these we have a production rule in our grammar.
A

0 | 1 | | maxint | x | y | z | A + A | A A | A A

tt | ff | A = A | A A | B B | B

x := A | y := A | z := A | skip | S; S | if B then S else S | while B do S

This may surprise you, but if we add more variables to our language8 then
it can calculate precisely what, say, Java can calculate for natural numbers
and booleans.
We can now look at programs in this language. Assume that the value
held in the variable y is 10.
x := 1; while y = 0 do x := 2 x; y := y 1
This program will calculate 210 and store it in x. In fact, it will calculate
2y (assuming y holds a natural number).
Exercise 48. We look at this example in more detail.
(a) Give a derivation for the program above.
(b) Is this grammar unambiguous? If you think it isnt, then give an example and show how this grammar might be made unambiguous, otherwise
explain why you think it is.
(c) Give a parse tree for the above program. Can you see how that tells you
something about what computation is supposed to be carried out?

4.6

The Backus-Naur form

In the literature (in particular that devoted to programming languages)

context-free grammars are often described in a slightly different form to the
one used here so far. This is known as the Backus-Naur form (sometimes
also described as Backus Normal Form but that isnt a very accurate name).
There is a simple translation process.
Non-terminal symbols dont have to be capitals and are distinguished
from other symbols by being in italics, or they are replaced by a descriptive term included in angled brackets h and i.
The rewriting arrow is replaced by the symbol ::=.
8

When people define grammars for programming languages they typically use abbreviations that allow them to stipulate any variable without listing them all explicitly, but
we dont want to introduce more notation at this stage.

68

We repeat the grammar for the While language from above to illustrate
what this looks like.
Here we assume that a ranges over arithmetic expressions, AExp, b
ranges over boolean expressions, BExp, and S ranges over statements, Stat.
a ::= 0 | 1 | | maxint | x | y | z | a + a | a a | a a
b ::= tt | ff | a = a | a a | b b | b
S ::= x := a | y := a | z := a | skip | S; S | if b then S else S | while b do S
In the next example we assume that instead we are using haexpi to range
over AExp, hbexpi over BExp, and hstati over Stat respectively.
haexpi ::= 0 | 1 | | maxint | x | y | z |
haexpi + haexpi | haexpi haexpi | haexpi haexpi
hbexpi ::= tt | ff | haexpi = haexpi | haexpi haexpi |
hbexpi hbexpi | hbexpi
hstati ::= x := haexpi | y := haexpi | z := haexpi | skip | hstati; hstati |
if hbexpi then hstati else hstati | while hbexpi do hstati

4.7

Properties of context-free languages

There is a description for building new regular languages from old ones in
Section 3. We can use almost any way we like to do so; set-theoretic ones
such as unions, intersections and complements, as well as ones using concatenation or the Kleene star; even reversing all the words work. Context-free
languages are rather more fiddly to deal with: Not all of these operations
work for them.
Concatenation. This does work. We do the following. Assume that
we have two grammars with terminal symbols taken from the alphabet ,
and non-terminal symbols 1 respective 2 . We now take every symbol in
1 and put a subscript 1 onto it, and similarly for every symbol in 2 , and
the subscript 2. We have now forced those two alphabets to be disjoint, so
when we form their union the number of symbols in the union is the sum
of the symbols in 1 and 2 . We add a new start symbol S to the set of
non-terminal symbols. We now take all the production rules from the first
grammar, and put subscripts of 1 onto each non-terminal symbol occurring
in it, and do the same for the second grammar with the subscript 2. We
add one new production rule S S1 S2 .
Exercise 49. Use your solution to Exercise 42 (c) to produce a grammar
for the language of all words whose length is at least 6, using this procedure.
Kleene star. This looks as if it should be more complicated than concatenation, but it is not. We merely add two production rules to the grammar (if they arent already there) for our given language, namely S SS,
and S . If we then wish to generate a word that is the n-fold concatenation of words in our language, where n N+ we merely start by applying
the rule S SS (n 1) times, giving us n copies of S. For each one of
these we can then generate the required word. If we wish to generate the
empty word we can do this by applying the second new rule, S .

69

Exercise 50. Use your solution to Exercise 42 (c) to produce a grammar for
the language of all words whose length is divisible by 3, using this procedure.
Reversal. This is quite easy. Leave the two alphabets of terminal
and non-terminal symbols as they are. Now take each production rule, and
replace the string on the right by its reverse. So if there is a production rule
of the form R 00R1, replace it by the rule R 1R00.
Exercise 51. Use this procedure to turn your solution for Exercise 40 into
a grammar that generates all words over {0, 1} that start with 1 and end
with 0.
Unions. This does work.
Exercise 52. Show that the union of two context-free languages over some
alphabet is context-free.
Intersections. Somewhat surprisingly, this is not an operation that
works. The intersection of a context-free language with a regular one is
context-free, but examining this issue in more detail goes beyond the scope
for this course.
Complements. This also does not work. Just because we have a way
of generating all the words in a particular set does not mean we can do the
same for all the words not belonging to this set.

4.8

Limitations of context-free languages

Given the idea of context-sensitive grammars mentioned at the start of this

section it should not come as a surprise that there are languages which
are not context-free. We do not develop the technical material here that
allows us to prove this statement. In Section 3 there is an explanation why
finite state automata can only do a very limited amount of counting, and
context-free grammars can certainly do more than that. However, there are
limitations to this as well. The language
{an bn cn | n N}
over {a, b, c} is not context-free.
There is a formal result, called the Pumping Lemma for context-free
languages that describes some of the possible properties that indicate that
a language fails to be context-free.
There are automata which can be used to describe the context-free languages which are known as pushdown-automata. These do have a memory
device that takes the form of a stack. That makes them fairly complicated
to cope with, and that is why they are beyond the scope of this course.
Moreover, for these automata, deterministic and non-deterministic versions
are not equivalent, making them even more complex to handle.

4.9

Summary

A context-free grammar or CFG is given by two alphabets, one of

terminal, and one of non-terminal symbols and by production rules
that tell us how to replace non-terminals by strings consisting of both,
terminal and non-terminal symbols.

70

 We generate words for a CFG by starting with the non-terminal start

symbol S. In the word generated so far we then pick one non-terminal
symbol and replace it by a string according to one of the production
rules for that symbol.
A CFG generates a language made up of all the words consisting entirely of terminal symbols that can be generated using it. The languages that can be generated in this way are known as the context-free
languages.
Every regular language is context-free, but there are some context-free
languages that are not regular.
Rather than a derivation for a word we can give a parse tree which
is sufficient to tell us how to carry out calculations, for example.
Programming languages are defined using grammars, and a compiler
parses a program to check whether it is correctly formed.
We can apply the union, concatenation, Kleene star and reversal operations to context-free languages and that gives us another context-free
language.
There are languages that are not context-free.

71

Glossary
accept
A word can be accepted, or not accepted, by an automaton. 25, 27,
44
Algorithm 1
Turns an NFA into a DFA. 31
Algorithm 2
Turns an automaton (works for NFA and DFA, but more complicated
for NFA) into a pattern describing the language recognized by the
automaton. 39
Algorithm 3
Takes a pattern and turns it into an NFA with -transition such that
the language recognized by the automaton is that defined by the pattern. 46
Algorithm 4
Turns an NFA with -transitions into an NFA without these. 44
alphabet
A set of letters. 11
ambiguous
A word can be ambiguously generated if we have a grammar that has
two parse trees for it, and if such a word exists we call the grammar
ambiguous. A language is inherently ambiguous if all the grammars
generating it are ambiguous. 65
Backus-Naur form
A particular way of writing down a grammar popular within computer science, in particular for describing the syntax of programming
languages. 68
bisimulation
A relation between the states of two automata; its existence implies
that the two automata are equivalent. 54
CFG
Rules for generating words by using rewrite rules known as a grammar,
in our case a context-free one. 60

72

concatenation
An operation on words (or letters) that takes two words and returns
one word by attaching the second word to the end of the first word.
11
context-free
A language is context-free if there is a context-free grammar generating
it. (Note that grammars can also be context-freewe do not consider
other grammars on this course. 62
context-free grammar
Same as CFG. 60
derivation
A way of demonstrating that a particular word is generated by a grammar. 58
deterministic finite automaton
Same as DFA. 24
DFA
An alternative way of defining a language, more suitable for human
consumption than a regular expression. 24

The empty word. 11

equivalence
Defined for automata. Two automata are equivalent if and only if they
recognize the same language. 53
Kleene star
An operation allowing an arbitrary number of concatenations of words
from a given language. 13
language
A set of words. 12
language defined by pattern
The set of all the words matching the pattern. See also the definition
on page 17
language generated by grammar
The set of all words consisting entirely of terminal symbols which can
be built using the rules of the grammar. 60
letter
Building block for word. Same as symbol. 11
matching
A word can match a pattern. 16
73

NFA
An alternative way of defining a language, often the easiest for people
to devise. 26
NFA with -transitions
An NFA which also allows transitions labelled with which do not
have to be matched against a letter. 43
non-deterministic finite automaton
Same as NFA. 26
non-terminal symbol
An auxiliary symbols used to describe a grammarwe want to create
words that do not contain auxiliary symbols. 60
parse tree
A parse tree shows how a string can be generated from a given grammar in a more useful way than a derivation. 64
pattern
A particular kind of string used to define languages. 15
recognize
Automata recognize languages. 25, 27, 44
regular
A language is regular if it can be defined using a pattern. 19
regular expression
Same as pattern. 15
right-linear
A grammar is right-linear if its production rules are particularly simple. The definition is given on page 63
string
Same as word. 11
string generated by grammar
A word consisting of terminal and non-terminal symbols that we can
build by using the rules of the grammar. 60
symbol
Same as letter. 11
terminal symbol
An element of the alphabet over which we are trying to create words
using a grammar. 60
unambiguous
A grammar is unambiguous if it isnt ambiguous. 65
word
Obtained from concatenating 0 or more letters. Same as string. 11
74

Appendix A

In the language of
mathematics
The account given so far is informal as far as possible. To be completely
rigorous the language of mathematics has to be used to define the various
notions we have used. For the interested reader we here give a glimpse of
how this is done. It is possible to pass this course, and pass it with a good
mark, without knowing the content of this chapter, but those who want to
get the whole picture, and get a really good mark, should work through this
part as well.

A.1

The basic concepts

We begin by giving mathematical definitions for the various notions from

Section 2.1.
Definition 16. An alphabet is a finite set of characters (primitive indecomposable symbols) called the letters of .
Definition 17. Let be an alphabet. A word over is a finite string
x1 x2 xn
where n N and the xi are letters from . If n = 0 we write for the
resulting string.
Exercise A.1. When we are given an alphabet we can look at various kinds
of words we can build from it.
(a) Form all the three letter words over the alphabet {a, b, 1}.
(b) Form all the words over the alphabet {0}.
(c) Form all the words over the alphabet {}.
We can alternatively use a recursive definition for a word. This is particularly useful if we want to define operations on words recursively. It also
fits well with how some data structures are defined.
Alternative definition. A word over the alphabet is obtained from the
following rules:
The empty word is a word.
If s is a word and x then sx is a word.
75

Now that we have the basic concept we can define operations for words,
such as the length of a word. Note how this definition follows the recursive
definition of a word.
Definition 18. The length |s| of a word s over some alphabet is defined
as follows:

0
s=
|s| =
|s | + 1 s = s x
Exercise A.2. Try to come up with a non-recursive definition of the length
function. Hint: Look at the original definition of word, or at the definition
of concatenation to get an idea. Use the recursive definition of a word to
argue that your definition agrees with the original.
We have a binary operation on words, namely concatenation.
Definition 19. Given an alphabet , concatenation is an operation from
pairs of words to words, all over , which, for word s and t over , we write
as s t. It is defined as follows:
x1 . . . xm y1 . . . yn = x1 . . . xm y1 . . . yn .
Exercise A.3. Recall the definition of an associative or commutative operation from COMP11120.
(a) Argue that the concatenation operation is associative.
(b) Show that the concatenation operation is not commutative.
Exercise A.4. Use recursion to give an alternative definition of the concatenation operation. (You may find this difficulttry to give it a go anyway.)
Exercise A.5. Show that |s t| = |s| + |t|. Hint: You may well find it easier
to use the non-recursive definition for everything.
Exercise A.6. Practice your understanding of recursion by doing the following.
(a) Using the recursive definition of a word give a recursive definition of
the following operation. It takes a word, and returns the word where every
letter is repeated twice. So ab turns into aabb, and aba into aabbaa, and aa
to aaaa.
(b) Now do the same thing for the operation that takes a word and returns
the reverse of the word, so abc becomes cba.
In order to describe words of arbitrary length concisely we have adopted
notation such as a3 for aaa. This works in precisely the same way as it does
for powers of numbers: By a3 we mean the word that results from applying
the concatenation operation to three copies of a to obtain aaa, just as in
arithmetic we use 23 to indicate that multiplication should be applied to
three copies of 2 to obtain 2 2 2. So all we do by writing an is to find
a shortcut that tells us how many copies of a we require (namely n many)
without having to write them all out.
Because we know both these operations to be associative we do not have
to use brackets here: (2 2) 2 is the same as 2 (2 2) and therefore the
notation 2 2 2 is unambiguous, just as is the case for 2 2 2.
76

What should a1 mean? Well, this is simple, it is merely the word consisting of one copy of a, that is a. The question of what a0 might mean is
somewhat trickier: What is a word consisting of 0 copies of the letter a? A
useful convention in mathematics is to use this to mean the unit for the underlying operation. Hence in arithmetic 20 = 1. The unit for concatenation
is the empty word and so we think of a0 as a way of referring to the empty
word .
This way we obtain useful rules such as am an = an+m . Similarly
we have (am )n = anm , just as we have for exponentiation in arithmetic.
However, note that (ab)n consists of n copies of ab concatenated with each
other, rather than an bn , as we would in arithmetic.1
Exercise A.7. Write out in full the words 05 , 03 13 , (010)2 , (01)3 0, 10 .
Languages are merely collections of words.
Definition 20. Let be an alphabet. A language over is a set of words
over .
As mentioned in Chapter 2 using this definition we automatically obtain
set-theoretic operations: We can form the unions, intersections, complements and differences of languages in precisely the same way as we do this
for other sets. Hence expressions such as L1 L2 , L1 L2 and L1 \ L2 are
immediately meaningful.
Exercise A.8. Let be the alphabet {0, 1, 2} and let
L1 = {s | s is a word consisting of 0s and 1s only},
L2 = {s | s is a word beginning with 0 and ending with 2}.
Calculate the following languages: L1 L2 , L1 L2 and the complement of
L1 in the language of all words over .
Note that we have notation to find a more compact description of the
language
{, 1, 11, 111, 1111, . . .}
of all words over the alphabet {1} as
{1n | n N}.
Exercise A.9. Write down the following languages using set-theoretic notation:
(a) All the words consisting of the letters a and b which contain precisely
two as and three bs.
(b) All the words consisting of the letter 1 that have even length.
(c) All the words consisting of an arbitrary number of as followed by at
least one, but possibly more, bs.
(d) All the non-empty words consisting of a and b occurring alternatingly,
beginning with an a and ending with a b.
1
The reason this rule doesnt hold is that the concatenation operation isnt commutative (see Exercise A.3), and so we cant swap over the as and bs to change the order in
which they appear.

77

(e) All the non-empty words consisting of a and b occurring alternatingly,

beginning with an a and ending with an a.
(f) All the words consisting of an arbitrary number of 0s followed by the
same number of 1s.
(g) All the words over some alphabet .
Note that if we have two alphabets then every word over can
be viewed as a word over the alphabet . Alternatively, we may restrict a
language L over to all those words that only use letters from by forming
{s L | s only uses letters from }.
The definitions in Chapter 2 for forming new languages from existing
ones are rigorous as given. You may want to improve your understanding
of them by carrying out the following exercises.
Exercise A.10. (a) Calculate {a, b}2 and {ab, ba}2 .
(b) Find the shortest set-theoretic expression you can find for the set of all
words over the alphabet {0, 1, 2} which consist of precisely three letters.
(c) Calculate {00, 11} .
(d) If 1 2 are two alphabets, argue that 1 2 .

A.2

Regular expressions

The formal definition of a pattern, and that of a word matching a pattern is

given in Chapter 2. There are exercises in that chapter that encourage you
to explore the recursive nature of the two definitions. The following should
not be considered a formal exercise. Just think about it.
Exercise A.11. What would it take to demonstrate that a given word does
not match a given pattern?
We have already seen that different patterns may describe the same
language. In fact, we can come up with a number of rules of rewriting
regular expressions in such a way that the language they define stays the
same. Here are some examples, where p, p1 and p2 are arbitrary patterns
over some alphabet .
L(p) = L(p).
L(p1 |p2 ) = L(p2 |p1 ).
L(p ) = L(|pp ).
L((p1 |p2 )p) = L(p1 p|p2 p).
Exercise A.12. Come up with three further such rules.
Exercise A.13. Argue that the equalities used in the recursive definition
of a language matching a pattern on page 17 have to hold according to
Definition 3.
Exercise A.14. For Exercise 8 find set-theoretic notation for the sets you
described in English.
Exercise A.15. Prove Proposition 2.1. Hint: Take a closer look at the
recursive definition of the language defined by a pattern above.
78

A.3

Finite state automata

Matching up the formal definition of an automaton with its image is fairly

straightforward.
As an example consider the picture on page 21

0
E

O
0

of an automaton that accepts precisely those words over {0, 1} that contain an even number of 0s. We have two states, which in the picture are
labelled E and O, so the set of all states is {E, O}. The initial state is E,
and there is only one accepting state which is also E.
The transition function for this example is given by the following table:
input
(E, 1)
(E, 0)
(O, 1)
(O, 0)

output
E
O
O
E

If we follow a word such as 001 through the automaton we can now do

this easily: We start in state E and see a 0, so we next need to go to (E, 0)
which is O. The next letter is 0 again, and from the new state O we need to
go to (O, 0) = E. The last letter is 1, so from the current state E we need
to go to (E, 1) = E. Since E is an accepting state we accept the word 001.
Exercise A.16. Describe the final picture from Section 3.1 in this way.
When moving to non-deterministic automata we no longer have a transition function, but a transition relation. Therefore we cannot use a table
as above to write it out.
As an example, take the non-deterministic automaton from page 25.
0, 1
2
1
0

0, 1
0

0, 1

It has states {0, 1, 2, 3}, start state 0 and accepting states {3}. The
transition relation is defined as follows.
relates
state letter
0
0
0
1
1
0
1
1
2
0
2
1
3
0
3
1

to
0

1
X

X
X
X

X
X
X
X
X

79

Think of the ticks as confirming that from the given state in the left
hand column there is an edge labelled with the letter in the second column
to the state in the top row. We can see that here is a relation rather than
a function because for the input state 1, letter 1, we find two ticks in the
corresponding row.
Exercise A.17. Draw a non-deterministic automaton for the language described in Exercise 9 (d). Then describe it in the same way as the above
example.
Exercise A.18. Draw the automata with states {0, 1, 2, 3}, start state 0,
accepting states {0, 2} and the following transitions.
(a)

(b)

relates
state letter
0
a
0
b
1
a
1
b
2
a
2
b
3
a
3
b
relates
state letter
0
a
0
b
1
a
1
b
2
a
2
b
3
a
3
b

to
0

1
X

X
X
X
X
X
X
X
to
0
X
X
X
X

1
X

X
X
X

X
X

Can you say anything about the resulting automata?

Exercise A.19. Using Algorithm 1 on page 32 we know how to turn a nondeterministic automaton into a deterministic one. Write out a definition of
the transition function for the new automaton.
Exercise A.20. Prove Theorem 3.1. All that remains to be established is
that a DFA generated by an NFA using Algorithm 1 from page 32 accepts
precisely the same words as the original automaton.
Pictures are good and well when it comes to describing automata, but
when such a description has to be given to a computer it is best to start
from the mathematical description and turn it into a data structure.
Exercise A.21. What would it take to define an automaton class in Java?
How would a class for deterministic automata differ from one for nondeterministic ones? Hint: You may not know enough Java yet to decide
this. If this is the case then return to the question later.
Exercise A.22. Algorithm 2 is quite complicated. To try to understand
it better I recommend the following for the general description which starts
on page 40.
80

i
(a) Justify to yourself the expressions for Lj
ji and Lji from the general

ones for Lk
ji .
(b) Justify the equalities given at the end of the description of the general
case.
Exercise A.23.* Explain why it is safe not to draw the dump states for the
two automata when constructing an automaton for the intersection of the
languages recognized by the automata as described on page 52 by forming
the product of the two automata.
Exercise A.24. Exercise 34 has two parts that are mathematical, namely a
and (e). If you have delayed answering these do so now.
Let us now turn to the question of when to automata recognize the same
language. We begin by giving a proper definition of what it means for two
automata to be isomorphic.
Definition 21. An automaton (Q, q , F, ) over the alphabet is isomorphic
to an automaton (P, p , E, ) over the same alphabet if and only if there
exists a function f : Q - P such that
f is bijective;
f (q ) = p ;
q is in F if and only if p is in E for all q Q and all p P ;
in case the automata are
deterministic: f ((q, x)) = (f (q, x)) for all q Q and all x ;
x

- q according to is true if and only if

non-deterministic: q
x
f (q) - f (q ) according to also holds.

Exercise A.25. Convince yourself that you understand this definition! Also
convince yourself that two automata are isomorphic precisely when they have
the same picture (if the states remained unlabelled).
On page 54 we had a calculation used to turn the problem of equivalence
of two automata into one of deciding whether two (different) automata recognize any words at all. You should be able to prove this, using definitions
from the set-theory part of COMP111.
Exercise A.26. For subsets S and S of some set T show that S = S if
and only if S (T S ) = and S (T S) = .
The notion of bisimulation deserves additional study and, indeed, this is
a concept that is very important in concurrency. Here we restrict ourselves
to one more (if abstract) example.
Exercise A.27. Show that there is a bisimulation between an NFA and the
result of turning it into a DFA using Algorithm 1.
Exercise 38 is somewhat mathematical in the way it demands you to
think about languages. Try to do as many of its parts as you can.

81

A.4

Grammars

Exercises in this section that asks you to reason mathematically are Exercises 44 and 45they ask you to demonstrate something.
In Section 4.3 there is a method that takes an automaton and turns it
into a context-free grammar. How can we see that the words generated by
this grammar are precisely those accepted by the automaton?
If we have a word x1 x2 xn accepted by the automaton then we get
the following for the grammar.
In the automaton:

For the grammar:

We start in state q and go to

We have S = q x1 q1 .

q1 = (q , x1 ).
Once we have reached the word

Once we have reached state

x1 x2 xi qi

qi = (qi1 , xi ),

we apply a production rule to obtain

we go to
qi+1 = (qi , xi+1 )

x1 x2 xi xi+1 qi+1 ,

and keep going until we reach

where

qn = (qn1 , xn ).

qi+1 = (qi , xi+1 ),

and keep going until we reach
x1 x2 xn qn .
Since we know that qn is an accepting state we may now use the
production rule qn to obtain
x1 x2 xn .

Now qn is an accepting state.

Exercise A.28. Use a similar argument to reason that if s is a word over

generated by the grammar then it is also accepted by the automaton.
Exercise A.29.* We have already demonstrated that every regular languages can be generated by a right-linear grammar. Try to show the opposite, that is, that every language generated by a right-linear grammar is
regular.

82

COMP11212 Fundamentals of
Computation
Exercise Sheet 1
For examples classes in Week 2

Exercises for this week

Exercises marked this week:
Exercise 4 (a), (c), (j), (k)
Exercise 9 (a)(c)
Exercise 9 d, (e)
Exercise 10 (a), (b)
Exercise 13 (a)(c)
Exercises preparing the marked exercises
If you find the exercises we will mark difficult you should first try the easier
exercises preparing them.
Exercise 3 might help with Exercise 4.
Exercise 11 and Exercise 12 should help with Exercise 13.
Finding regular expressions and automata for a given language requires
practice. Theres no algorithm for this!

Foundational exercises
The foundational exercises prepare important concepts appearing in Chapters 3 and 4.
Exercise 1
Exercise 2
Exercise 8 (a)(c)

Optional exercises from Appendix A

The exercises in Sections A.1 and A.2.

Optional further suitable exercises

Exercise 5
Exercise 6
Exercise 7 (This exercise makes a connection between our regular expressions
and those standard in Unix.)
Further parts of Exercises 4, 8, 9, 10, 13

83

COMP11212 Fundamentals of
Computation
Exercise Sheet 2
For examples classes in Week 3

Exercises for this week

Exercises marked this week
Exercise 13 (d), (e)
Exercise 14 (a), (b)
Exercise 16
Exercise 19
Exercise 21 (a)
Exercises preparing the marked exercises
If you find the exercises we will mark difficult you should first try the easier
exercises preparing them.
Exercise 11 and Exercise 12 should help with Exercises 13 and 14.
Exercise 15 will help with understanding the difference between NFAs
and DFAs, and so help with Exercise 16.
The last two exercises are about applying Algorithm 1, which has an
example spelled out in the notes.
Foundational exercises
Exercise 17
Exercise 18
Exercise 20

Optional exercises from Appendix A

Exercises A.16A.20

Optional further suitable exercises

Further parts of Exercises 14 and 21

84

COMP11212 Fundamentals of
Computation
Exercise Sheet 3
For examples classes in Week 4

Exercises for this week

Exercises marked this week
Exercise 22
Exercise 23
Exercise 25 (a)
Exercise 26 (b)
Exercise 32 (a) and (b)
Exercises preparing the marked exercises
If you find the exercises we will mark difficult you should first try the easier
exercises preparing them.
Theres nothing required for Exercise 22 but a bit of common sense.
Be warned that the resulting automaton is quite large!
Exercise 23 mostly consists of tasks you should begin to find easy.
Exercise 24 and Exercise 26 (a) ask you to apply Algorithm 2 to simpler
automata and should help with Exercises 25 and 26.
Exercise 32 asks you to apply Algorithms 3 and 4, and Exercise 29
and Exercise 31 (a) allow you to practice these.

Optional exercises from Appendix A

Exercises A.21A.23

Optional further suitable exercises

Exercise 27
Exercise 28
Exercise 30
Further parts of Exercises 25, 26, 31 and 32

85

COMP11212 Fundamentals of
Computation
Exercise Sheet 4
For examples classes in Week 5

Exercises for this week

Exercises marked this week:
Exercise 35 (a)
Exercise 36 (a) and (b)
Exercise 36 (c) and (d) Note that part (d) requires having read the Appendix.
Exercise 37
Exercise 38 (a)(c)
Exercises preparing the marked exercises
You might find the following helpful.
There is a worked example of the method used in Exercise 35 in the
notes.
Exercise 34 (b)(d) will help with Exercise 36.
Exercise 37 is preceded by an example.
In order to answer Exercise 38 youll have to practice thinking in the
right way. Theres no fixed method for this.

Foundational exercises
Exercise 33

Optional exercises from Appendix A

Exercises A.24A.27

Optional further suitable exercises

Further parts of Exercises 35 and 38

86

COMP11212 Fundamentals of
Computation
Exercise Sheet 5
For examples classes in Week 6

Exercises for this week

Exercises marked this week:
Exercise 42 (c)
Exercise 43 (a)(b)
Exercise 46 (a)
Exercise 47 (a)
Exercise 50
Exercises preparing the marked exercises
You might want to note the following.
Exercise 39 helps you understand how to derive a word from a grammar, and so to design grammars as in Exercises 42 and 43.
Exercise 42 (a)(b) might be easier to start with than part (c).
There is a worked example demonstrating what has to be done in
Exercise 46 in the notes.
Exercise 47 is preceded by an example in the notes. There is no
method that will solve this problem generally; you will have to think
about each one separately.
Exercise 50 is preceded by a general description of the method to be
applied here.

Optional exercises from Appendix A

Exercises in Section A.4

Optional further suitable exercises

Exercise 40
Exercise 41
Exercise 44
Exercise 45
Exercise 48
Exercise 51
Exercise 49
Exercise 52
Further parts of Exercises 42, 43, and 46

87