Lexical Analyzer

• Lexical Analyzer reads the source program character by character to

produce tokens.
• Normally a lexical analyzer doesn’t return a list of tokens at one shot,
it returns a token when the parser asks a token from it.

source Lexical token

Parser
program Analyzer get next token

1
 Token
• Token represents a set of strings described by a pattern.
– Identifier represents a set of strings which start with a letter continues with letters and digits
– The actual string (newval) is called as lexeme.
– Tokens: identifier, number, addop, delimeter, …
• Since a token can represent more than one lexeme, additional information should be
held for that specific lexeme. This additional information is called as the attribute of
the token.
• For simplicity, a token may have a single attribute which holds the required
information for that token.
– For identifiers, this attribute a pointer to the symbol table, and the symbol table holds the actual
attributes for that token.
• Some attributes:
– <id,attr> where attr is pointer to the symbol table
– <assgop,_> no attribute is needed (if there is only one assignment operator)
– <num,val> where val is the actual value of the number.
• Token type and its attribute uniquely identifies a lexeme.
• Regular expressions are widely used to specify patterns.

2
 Terminology of Languages
• Alphabet : a finite set of symbols (ASCII characters)
• String :
– Finite sequence of symbols on an alphabet
– Sentence and word are also used in terms of string
–  is the empty string
– |s| is the length of string s.
• Language: sets of strings over some fixed alphabet
–  the empty set is a language.
– {} the set containing empty string is a language
– The set of well-wormed C programs is a language
– The set of all possible identifiers is a language.
• Operators on Strings:
– Concatenation: xy represents the concatenation of strings x and y. s  = s s=s
– sn = s s s .. s ( n times) s0 = 

3
 Operations on Languages
• Concatenation:
– L1L2 = { s1s2 | s1  L1 and s2  L2 }

• Union
– L1 L2 = { s | s  L1 or s  L2 }

• Exponentiation:
– L0 = {} L1 = L L2 = LL

• Kleene Closure


L

– L* = L i i
i 0

• Positive Closure
i0

– L+ =
L
i 1
i

4
 Example
• L1 = {a,b,c,d} L2 = {1,2}

• L1L2 = {a1,a2,b1,b2,c1,c2,d1,d2}

• L1  L2 = {a,b,c,d,1,2}

• L13 = all strings with length three (using a,b,c,d}

• L1* = all strings using letters a,b,c,d and empty string

• L1+ = doesn’t include the empty string

5
 Regular Expressions
• We use regular expressions to describe tokens of a programming
language.
• A regular expression is built up of simpler regular expressions (using
defining rules)
• Each regular expression denotes a language.
• A language denoted by a regular expression is called as a regular set.

6
 Regular Expressions (Rules)
Regular expressions over alphabet 

Reg. Expr Language it denotes

 {}
a  {a}
(r1) | (r2) L(r1)  L(r2)
(r1) (r2) L(r1) L(r2)
(r)* (L(r))*
(r) L(r)

• (r)+ = (r)(r)*
• (r)? = (r) | 

7
 Regular Expressions (cont.)
• We may remove parentheses by using precedence rules.
– * highest
– concatenation next
– | lowest
• ab*|c means (a(b)*)|(c)

• Ex:
–  = {0,1}
– 0|1 => {0,1}
– (0|1)(0|1) => {00,01,10,11}
– 0* => { ,0,00,000,0000,....}
– (0|1)* => all strings with 0 and 1, including the empty string

8
 Regular Definitions
• To write regular expression for some languages can be difficult, because
their regular expressions can be quite complex. In those cases, we may
use regular definitions.
• We can give names to regular expressions, and we can use these names
as symbols to define other regular expressions.

• A regular definition is a sequence of the definitions of the form:

d1  r1 where di is a distinct name and
d2  r2 ri is a regular expression over symbols in
. {d1,d2,...,di-1}
dn  rn
basic symbols previously defined names
9
 Regular Definitions (cont.)
• Ex: Identifiers in Pascal
letter  A | B | ... | Z | a | b | ... | z
digit  0 | 1 | ... | 9
id  letter (letter | digit ) *
– If we try to write the regular expression representing identifiers without using regular definitions, that
regular expression will be complex.
(A|...|Z|a|...|z) ( (A|...|Z|a|...|z) | (0|...|9) ) *

• Ex: Unsigned numbers in Pascal

digit  0 | 1 | ... | 9
digits  digit +
opt-fraction  ( . digits ) ?
opt-exponent  ( E (+|-)? digits ) ?
unsigned-num  digits opt-fraction opt-exponent

10
 Finite Automata
• A recognizer for a language is a program that takes a string x, and answers “yes” if x
is a sentence of that language, and “no” otherwise.
• We call the recognizer of the tokens as a finite automaton.
• A finite automaton can be: deterministic(DFA) or non-deterministic (NFA)
• This means that we may use a deterministic or non-deterministic automaton as a
lexical analyzer.
• Both deterministic and non-deterministic finite automaton recognize regular sets.
• Which one?
– deterministic – faster recognizer, but it may take more space
– non-deterministic – slower, but it may take less space
– Deterministic automatons are widely used lexical analyzers.
• First, we define regular expressions for tokens; Then we convert them into a DFA to
get a lexical analyzer for our tokens.
– Algorithm1: Regular Expression  NFA  DFA (two steps: first to NFA, then to DFA)
– Algorithm2: Regular Expression  DFA (directly convert a regular expression into a DFA)

11
 Non-Deterministic Finite Automaton (NFA)
• A non-deterministic finite automaton (NFA) is a mathematical model
that consists of:
– S - a set of states
–  - a set of input symbols (alphabet)
– move – a transition function move to map state-symbol pairs to sets of states.
– s0 - a start (initial) state
– F – a set of accepting states (final states)

• - transitions are allowed in NFAs. In other words, we can move from

one state to another one without consuming any symbol.
• A NFA accepts a string x, if and only if there is a path from the starting
state to one of accepting states such that edge labels along this path
spell out x.

12
 NFA (Example)

a 0 is the start state s0

{2} is the set of final states F
a b S = {a,b}
0 1 2
start S = {0,1,2}
b Transition Function: a b
0 {0,1} {0}
Transition graph of the NFA 1 _ {2}
2 _ _
The language recognized by this NFA is (a|b) * a b

13
 Deterministic Finite Automaton (DFA)

• A Deterministic Finite Automaton (DFA) is a special form of a NFA.

• no state has - transition
• for each symbol a and state s, there is at most one labeled edge a leaving s.
i.e. transition function is from pair of state-symbol to state (not set of states)

a
b a
The language recognized by
a b
0 1 2
this DFA is also (a|b) * a b
b

14
 Converting A Regular Expression into A NFA
(Thomson’s Construction)
• This is one way to convert a regular expression into a NFA.
• There can be other ways (much efficient) for the conversion.
• Thomson’s Construction is simple and systematic method.
It guarantees that the resulting NFA will have exactly one final state,
and one start state.
• Construction starts from simplest parts (alphabet symbols).
To create a NFA for a complex regular expression, NFAs of its sub-
expressions are combined to create its NFA,

15
 Thomson’s Construction (cont.)

i f
• To recognize an empty string 

• To recognize a symbol a in the alphabet  i

a
f

• If N(r1) and N(r2) are NFAs for regular expressions r1 and r2

• For regular expression r1 | r2

 N(r1) 
i  f NFA for r1 | r2

N(r2)

16
 Thomson’s Construction (cont.)

• For regular expression r1 r2

i N(r1) N(r2) f Final state of N(r2) become final state of N(r1r2)

NFA for r1 r2

• For regular expression r*



 
i N(r) f

NFA for r*

17
 Thomson’s Construction (Example - (a|b) * a )
a a 
a: 

(a | b)  
b b
b:


a 

 
(a|b) *
 
b



a 


(a|b) * a 
 a

b

18
 Converting a NFA into a DFA (Example)
2 a 3 

0  1  a
 6 7 8

4 b 5


S0 = -closure({0}) = {0,1,2,4,7} S0 into DS as an unmarked state

 mark S0
-closure(move(S0,a)) = -closure({3,8}) = {1,2,3,4,6,7,8} = S1 S1 into DS
-closure(move(S0,b)) = -closure({5}) = {1,2,4,5,6,7} = S2 S2 into DS
transfunc[S0,a]  S1 transfunc[S0,b]  S2
 mark S1
-closure(move(S1,a)) = -closure({3,8}) = {1,2,3,4,6,7,8} = S1
-closure(move(S1,b)) = -closure({5}) = {1,2,4,5,6,7} = S2
transfunc[S1,a]  S1 transfunc[S1,b]  S2
 mark S2
-closure(move(S2,a)) = -closure({3,8}) = {1,2,3,4,6,7,8} = S1
-closure(move(S2,b)) = -closure({5}) = {1,2,4,5,6,7} = S2
transfunc[S2,a]  S1 transfunc[S2,b]  S2 19
 Converting a NFA into a DFA (Example – cont.)

S0 is the start state of DFA since 0 is a member of S 0={0,1,2,4,7}

S1 is an accepting state of DFA since 8 is a member of S 1 = {1,2,3,4,6,7,8}

S1

S0 b a

S2

20
 Converting Regular Expressions Directly to DFAs
• We may convert a regular expression into a DFA (without creating a
NFA first).
• First we augment the given regular expression by concatenating it with
a special symbol #.
r  (r)# augmented regular expression
• Then, we create a syntax tree for this augmented regular expression.
• In this syntax tree, all alphabet symbols (plus # and the empty string) in
the augmented regular expression will be on the leaves, and all inner
nodes will be the operators in that augmented regular expression.
• Then each alphabet symbol (plus #) will be numbered (position
numbers).

21
DFA based pattern matcher: Regular Expression  DFA (cont.)

(a|b) * a  (a|b) * a # augmented regular expression


Syntax tree of (a|b) * a #
 #
4
* a
3 • each symbol is numbered (positions)
| • each symbol is at a leave
a b
1 2 • inner nodes are operators

22
 followpos

Then we define the function followpos for the positions (positions

assigned to leaves).

followpos(i) -- is the set of positions which can follow

the position i in the strings generated by
the augmented regular expression.

For example, ( a | b) * a #
1 2 3 4

followpos(1) = {1,2,3} followpos is just defined for leaves,

it is not defined for inner nodes.
followpos(2) = {1,2,3}
followpos(3) = {4}
followpos(4) = {}
23
 How to evaluate firstpos, lastpos, nullable
n nullable(n) firstpos(n) lastpos(n)
leaf labeled  true  

leaf labeled false {i} {i}

with position i

| nullable(c1) or firstpos(c1)  firstpos(c2) lastpos(c1)  lastpos(c2)

c1 c2 nullable(c2)
 nullable(c1) and if (nullable(c1)) if (nullable(c2))
c1 c2 nullable(c2) firstpos(c1)  firstpos(c2) lastpos(c1)  lastpos(c2)
else firstpos(c1) else lastpos(c2)
* true firstpos(c1) lastpos(c1)
c1

24
 How to evaluate followpos
• Two-rules define the function followpos:

1. If n is concatenation-node with left child c1 and right child c2,

and i is a position in lastpos(c1), then all positions in firstpos(c2) are in
followpos(i).
2. If n is a star-node, and i is a position in lastpos(n), then all positions in
firstpos(n) are in followpos(i).

• If firstpos and lastpos have been computed for each node, followpos
of each position can be computed by making one depth-first traversal
of the syntax tree.

25
 Example -- ( a | b) * a #

{1,2,3}  {4} green – firstpos

{1,2,3}  {3} {4} # {4} blue – lastpos
4
{1,2} *{1,2} {3} a{3}
3
{1,2} | {1,2}
Then we can calculate followpos
{1} a {1} {2} b {2}
2 followpos(1) = {1,2,3}
1
followpos(2) = {1,2,3}
followpos(3) = {4}
followpos(4) = {}

• After we calculate follow positions, we are ready to create DFA

for the regular expression.
26
 Algorithm (RE  DFA)
• Create the syntax tree of (r) #
• Calculate the functions: followpos, firstpos, lastpos, nullable
• Put firstpos(root) into the states of DFA as an unmarked state.
• while (there is an unmarked state S in the states of DFA) do
– mark S
– for each input symbol a do
• let s1,...,sn are positions in S and symbols in those positions are a
• S’  followpos(s1)  ...  followpos(sn)
• move(S,a)  S’
• if (S’ is not empty and not in the states of DFA)
– put S’ into the states of DFA as an unmarked state.

• the start state of DFA is firstpos(root)

• the accepting states of DFA are all states containing the position of #

27
 Example -- ( a | b) * a #
1 2 3 4

followpos(1)={1,2,3} followpos(2)={1,2,3} followpos(3)={4} followpos(4)={}

S1=firstpos(root)={1,2,3}
 mark S1
a: followpos(1)  followpos(3)={1,2,3,4}=S2 move(S1,a)=S2
b: followpos(2)={1,2,3}=S1 move(S1,b)=S1
 mark S2
a: followpos(1)  followpos(3)={1,2,3,4}=S2 move(S2,a)=S2
b: followpos(2)={1,2,3}=S1 move(S2,b)=S1
b a
a
S1 S2

start state: S1 b
accepting states: {S2} 28
 Minimizing Number of States of a DFA
• partition the set of states into two groups:
– G1 : set of accepting states
– G2 : set of non-accepting states

• For each new group G

– partition G into subgroups such that states s1 and s2 are in the same group iff
for all input symbols a, states s1 and s2 have transitions to states in the same group.

• Start state of the minimized DFA is the group containing

the start state of the original DFA.
• Accepting states of the minimized DFA are the groups containing
the accepting states of the original DFA.

29
 DFA Minimization

Fall 2003 CS416 Compiler Design 30

 Minimizing DFA - Example
a
G1 = {2}
2
a
G2 = {1,3}
1 b a
b
3 G2 cannot be partitioned because
b move(1,a)=2 move(1,b)=3
move(3,a)=2 move(2,b)=3

So, the minimized DFA (with minimum states)

b a

{1,3} a {2}

31
 Minimizing DFA – Another Example
a
2
a a

1 4
Groups: {1,2,3} {4}
b
b a
{1,2} {3} a b
3 b no more partitioning 1->2 1->3
2->2 2->3
b 3->4 3->3

So, the minimized DFA b

{3}
a b
{1,2} a b

a {4}

32
Minimization using Myhill Nerode Theorem:

33
Minimization using Myhill Nerode Theorem:

34
Minimization using Myhill Nerode Theorem:

35
 Some Other Issues in Lexical Analyzer (cont.)
• Skipping comments
– Normally we don’t return a comment as a token.
– We skip a comment, and return the next token (which is not a comment) to the parser.
– So, the comments are only processed by the lexical analyzer, and the don’t complicate the
syntax of the language.

36