LESSON 15

Overview
of
Previous Lesson(s)
 Over View
 Strategies that have been used to implement and optimize pattern
matchers constructed from regular expressions.

 The first algorithm is useful in a Lex compiler, because it constructs a

DFA directly from a regular expression, without constructing an
intermediate NFA.

3
 Over View..
 The second algorithm minimizes the number of states of any DFA,
by combining states that have the same future behavior.

 The algorithm itself is quite efficient, running in time O(n log n),
where n is the number of states of the DFA.

 The third algorithm produces more compact representations of

transition tables than the standard, two-dimensional table.

4
 Over View...
 A state of NFA can be declared as important if it has a non-ɛ out-
transition.

 NFA has only one accepting state, but this state, having no out-
transitions, is not an important state.

 By concatenating a unique right endmarker # to a regular expression

r, we give the accepting state for r a transition on #, making it an
important state of the NFA for (r) #.

5
 Over View...

 firstpos(n) is the set of positions in the sub-tree rooted at n that

correspond to the first symbol of at least one string in the language
of the sub-expression rooted at n.

 lastpos(n) is the set of positions in the sub-tree rooted at n that

correspond to the last symbol of at least one string in the language
of the sub expression rooted at n.

6
 Over View...

 followpos(p) , for a position p, is the set of positions q in the entire

syntax tree such that there is some string x = a1 a2 . . . an in L((r)#)
such that for some i, there is a way to explain the membership of x
in L((r)#) by matching ai to position p of the syntax tree and ai+1 to
position q

7
 Over View...
 nullable, firstpos, and lastpos can be computed by a straight
forward recursion on the height of the tree.

8
 Over View...

 Two ways that a position of a regular expression can be made to

follow another.

 If n is a cat-node with left child C1 and right child C2 then for every
position i in lastpos(C1) , all positions in firstpos(C2) are in
followpos(i).

 If n is a star-node, and i is a position in lastpos(n) , then all positions in

firstpos(n) are in followpos(i).

9
 Over View...
 Ex. DFA for the regular expression r = (a|b)*abb
 Putting together all previous steps:

Augmented Syntax Tree r = (a|b)*abb#

Nullable is true for only star node
firstpos & lastpos are showed in tree
followpos are:

10
 Over View...

 Start state of D = A = firstpos(rootnode) = {1,2,3}

 Then we compute Dtran[A, a] & Dtran[A, b]
 Among the positions of A, 1 and 3 corresponds to a, while 2
corresponds to b.
 Dtran[A, a] = followpos(1) U followpos(3) = { l , 2, 3, 4}
 Dtran[A, b] = followpos(2) = {1, 2, 3}

 State A is similar, and does not have to be added to Dstates.

 B = {I, 2, 3, 4 } , is new, so we add it to Dstates.
 Proceed to compute its transitions..

11
 TODAY’S LESSON

12
 Contents
 Optimization of DFA-Based Pattern Matchers
 Important States of an NFA
 Functions Computed From the Syntax Tree
 Computing nullable, firstpos, and lastpos
 Computing followups
 Converting a RE Directly to DFA
 Minimizing the Number of States of DFA
 Trading Time for Space in DFA Simulation
 Two dimensional Table
 Terminologies

13
 Minimizing DFA States
 Following FA accepts the language of regular expression
 (aa + b)*ab(bb)*

 Final states are colored yellow while rejecting states are blue.

14
 Minimizing DFA States..
 Closer examination reveals that states s2 and s7 are really the same
since they are both final states and both go to s6 under the input b
and both go to s3under an a.

 So, why not merge them and form a smaller machine?

 In the same manner, we could argue for merging states s0 and s5.

 Merging states like this should produce a smaller automaton that

accomplishes exactly the same task as our original one.

15
 Minimizing DFA States...

 From these observations, it seems that the key to making finite

automata smaller is to recognize and merge equivalent states.

 To do this, we must agree upon the definition of equivalent states.

Two states in a finite automaton M are equivalent if and only if

for every string x, if M is started in either state with x as input, it
either accepts in both cases or rejects in both cases.

 Another way to say this is that the machine does the same thing
when started in either state

16
 Minimizing DFA States...
 Two questions remain.

 First, how does one find equivalent states ?

 Exactly how valuable is this information?

 For a deterministic finite automaton M, the minimum number of

states in any equivalent deterministic finite automaton is the same
as the number of equivalence groups of M's states.

 Equivalent states go to equivalent states under all inputs.

17
 Minimizing DFA States...
 Now we know that if we can find the equivalence states (or groups
of equivalent states) for an automaton, then we can use these as
the states of the smallest equivalent machine.

 Ex Automaton

18
 Minimizing DFA States...

 Let us first divide the machine's states into two groups: Final and
Non-Final states.

 These groups are:

Final states = A = {s2, s7}
Non Final States = B = {s0, s1, s3, s4, s5, s6}

 Note that these are equivalent under the empty string as input.

19
 Minimizing DFA States...

 Now we will find out if the states in these groups go to the same
group under inputs a and b

 The states of group A both go to states in group B under both

inputs.

 Things are different for the states of group B.

20
 Minimizing DFA States...
 The following table shows the result of applying the inputs to these
states.
 For example, the input a leads from s1 to s5 in group B and input b leads to
to s2 in group A.

 Looking at the table we find that the input b helps us distinguish between
two of the states (s1 and s6) and the rest of the states in the group since
it leads to group A for these two instead of group B.
21
 Minimizing DFA States...
 The states in the set {s0, s3, s4, s5} cannot be equivalent to those in
the set {s1, s6} and we must partition B into two groups.

 Now we have the groups:

A = {s2, s7}, B = { s0, s3, s4, s5}, C = { s1, s6}

 The next examination of where the inputs lead shows us that s3 is

not equivalent to the rest of group B.

 We must partition again.

22
 Minimizing DFA States...

 Continuing this process until we cannot distinguish between the

states in any group by employing our input tests, we end up with
the groups:

A = {s2, s7}, B = {s0, s4, s5}, C = {s1}, D = {s3}, E = { s6}

In view of the above theoretical definitions and results, it is easy to

argue that all of the states in each group are equivalent because they all
go to the same groups under the inputs a and b.

23
 Minimizing DFA States...
 Building the minimum state finite automaton is now rather
straightforward.

 We merely use our groups as states and provide the proper

transitions.

24
 Minimizing DFA States...
 State Minimization Algorithm:

25
 Trading Time for Space in DFA

 The simplest and fastest way to represent the transition function of

a DFA is a two-dimensional table indexed by states and characters.

 Given a state and next input character, we access the array to find
the next state and any special action we must take, e.g. returning a
token to the parser.

 Since a typical lexical analyzer has several hundred states in its DFA
and involves the ASCII alphabet of 128 input characters, the array
consumes less than a megabyte.

26
 Trading Time for Space in DFA..

 Compilers are also appearing in very small devices, where even a

megabyte of storage may be too much.

 For such situations, there are many methods that can be used to
compact the transition table.

 For instance, we can represent each state by a list of transitions - that

is, character-state pairs - ended by a default state that is to be chosen
for any input character not on the list.

27
 Two dimensional Table
 A more subtle data structure that allows us to combine the speed
of array access with the compression of lists with defaults.

 A structure of four arrays:

28
 Two dimensional Table..

 The base array is used to determine the base location of the

entries for state s , which are located in the next and check arrays.

 The default array is used to determine an alternative base location

if the check array tells us the one given by base[s] is invalid.

 To compute nextState(s,a) the transition for state s on input a, we

examine the next and check entries in location l = base[s] +a
 Here character a is treated as an integer.
 Range 0 to 127.

29
 Two dimensional Table...

 If check[l] = s then this entry is valid, and the next state for state s
on input a is next[l]

 If check[l] ≠ s then we determine another state t = default[s] and

repeat the process as if t were the current state.

 Function nextState

30
 Terminologies
 Tokens

 The lexical analyzer scans the source program and produces as output a
sequence of tokens, which are normally passed, one at a time to the parser.
 Some tokens may consist only of a token name while others may also have
an associated lexical value that gives information about the particular
instance of the token that has been found on the input.

 Lexemes

 Each time the lexical analyzer returns a token to the parser, it has an
associated lexeme - the sequence of input characters that the token
represents.

31
 Terminologies..
 Patterns

 Each token has a pattern that describes which sequences of

characters can form the lexemes corresponding to that token.
 The set of words, or strings of characters, that match a given pattern
is called a language.

 Buffering

 Because it is often necessary to scan ahead on the input in order to

see where the next lexeme ends, it is usually necessary for the lexical
analyzer to buffer its input.

32
 Terminologies...
 Regular Expressions
 These expressions are commonly used to describe patterns.
 Regular expressions are built from single characters, using union,
concatenation, and the Kleene closure, or any-number-of, operator.

 Regular Definitions
 Complex collections of languages, such as the patterns that describe
the tokens of a programming language, are often defined by a regular
definition, which is a sequence of statements that each define one
variable to stand for some regular expression.
 The regular expression for one variable can use previously defined
variables in its regular expression.

33
Thank You