Chapter 2
Chapter 2
Lexical analysis
1
Outline
Introduction
Interaction of Lexical Analyzer with Parser
Token, pattern, lexeme
Specification of patterns using regular expressions
Regular expressions
Regular expressions for tokens
2
Introduction
❑ The role of lexical analyzer is:
• to read a sequence of characters from the source
program
• group them into lexemes and
• produce as output a sequence of tokens for each
lexeme in the source program.
The scanner can also perform the following
secondary tasks:
stripping out blanks, tabs, new lines
stripping out comments
keep track of line numbers (for error reporting)
3
Interaction of the Lexical Analyzer
with the Parser
Source
Program
symbol
table
(Contains a record
for each identifier)
5
Token, pattern, lexeme…
Example: The following table shows some tokens and
their lexemes in Pascal (a high level, case insensitive
programming language)
Token Some lexemes pattern
begin Begin, Begin, BEGIN, Begin in small or capital
beGin… letters
if If, IF, iF, If If in small or capital letters
ident Distance, F1, x, Dist1,… Letters followed by zero or
more letters and/or digits
8
Attributes of tokens…
9
9
Errors
Very few errors are detected by the lexical
analyzer.
For example, if the programmer mistakes
ebgin for begin, the lexical analyzer cannot
detect the error since it will consider ebgin as
an identifier.
Nonetheless, if a certain sequence of
characters follows none of the specified
patterns, the lexical analyzer can detect the
error.
10
Errors…
When an error occurs, the lexical analyzer
recovers by:
skipping (deleting) successive characters from the
remaining input until the lexical analyzer can find a
well-formed token (panic mode recover)
deleting one character from the remaining input
inserting missing characters into the remaining input
replacing an incorrect character by a correct
character
transposing two adjacent characters
11
Specification of patterns using
regular expressions
Regular expressions
Regular expressions for tokens
12
Regular expression: Definitions
13
Regular expressions…
A regular expression is one of the following:
Symbol: a basic regular expression consisting of a single
character a, where a is from:
▪ an alphabet Σ of legal characters;
▪ the metacharacter ε: or
▪ the metacharacter ø.
▪ In the first case, L(a)={a};
▪ in the second case, L(ε)= {ε};
▪ in the third case, L(ø)= { }.
▪ {} – contains no string at all.
▪ {ε} – contains the single string consists of no character
14
Regular expressions…
Alternation: an expression of the form r|s, where r
and s are regular expressions.
In this case , L(r|s) = L(r) U L(s) ={r,s}
15
Regular expression: Language Operations
Union of L and M
L ∪ M = {s |s ∈ L or s ∈ M}
Concatenation of L and M
LM = {xy | x ∈ L and y ∈ M}
Exponentiation of L
L0 = {ε}; Li = Li-1L The following shorthands
are often used:
Kleene closure of L
L* = ∪i=0,…,∞ Li r+ =rr*
r* = r+| ε
Positive closure of L
r? =r|ε
L+ = ∪i=1,…,∞ Li
16
Regular expressions…
Examples:
1- a | b = {a,b}
2- (a|b)a = {aa,ba}
3- (ab) | ε ={ab, ε}
4- ((a|b)a)* = {ε, aa,ba,aaaa,baba,....}
Reverse
1 – Even binary numbers (0|1)*0
2 – An alphabet consisting of just three alphabetic
characters: Σ = {a, b, c}. Consider the set of all strings
over this alphabet that contains exactly one b.
(a | c)*b(a|c)* {b, abc, abaca, baaaac, ccbaca, cccccb}
17
Exercises
Describe the languages denoted by the following
regular expressions:
1- a(a|b)*a
2- ((ε|a)b*)*
3- (a|b)*a(a|b)(a|b)
4- a*ba*ba*ba*
5- (aa|bb)*((ab|ba)(aa|bb)*(ab|ba)(aa|bb)*)*
18
Regular expressions for tokens
19
Regular expressions for tokens…
Special symbols: including arithmetic operators,
assignment and equality such as =, :=, +, -, *
Identifiers: which are defined to be a sequence of
letters and digits beginning with letter,
we can express this in terms of regular definitions as
follows:
letter = A|B|…|Z|a|b|…|z
digit = 0|1|…|9
or
letter= [a-zA-Z]
digit = [0-9]
identifiers = letter(letter|digit)*
20
Regular expressions for tokens…
Numbers: Numbers can be:
sequence of digits (natural numbers), or
decimal numbers, or
numbers with exponent (indicated by an e or E).
Example: 2.71E-2 represents the number 0.0271.
We can write regular definitions for these numbers as
follows:
nat = [0-9]+
signedNat = (+|-)? Nat
number = signedNat(“.” nat)?(E signedNat)?
Literals or constants: which can include:
numeric constants such as 42, and
string literals such as “ hello, world”.
21
Regular expressions for tokens…
22
Example: Divide the following Java program into
appropriate tokens.
public class Dog {
private String name;
private String color;
24
The Whole Scanner Generator Process
Overview
❑ Direct construction of Nondeterministic finite
Automation (NFA) to recognize a given regular
expression.
❑ Easy to build in an algorithmic way
❑ Requires ε-transitions to combine regular sub expressions
❑ Construct a deterministic finite automation
(DFA) to simulate the NFA Optional
❑ Use a set-of-state construction
❑ Minimize the number of states in the DFA
❑ Generate the scanner code.
25
Design of a Lexical Analyzer …
Token ➔ Pattern
Pattern ➔ Regular Expression
Regular Expression ➔ NFA
NFA ➔ DFA
DFA’s or NFA’s for all tokens ➔ Lexical Analyzer
26
Non-Deterministic Finite Automata
(NFA)
Definition
An NFA M consists of five tuples: ( Σ,S, T, s0, F)
A set of input symbols Σ, the input alphabet
a finite set of states S,
a transition function T: S × (Σ U { ε}) -> S (next state),
a start state s0 from S, and
a set of accepting/final states F from S.
The language accepted by M, written L(M), is defined as:
The set of strings of characters c1c2...cn with each ci from
Σ U { ε} such that there exist states s1 in T(s0,c1), s2 in
T(s1,c2), ... , sn in T(sn-1,cn) with sn an element of F.
27
NFA…
It is a finite automata which has choice of
edges
• The same symbol can label edges from one state to
several different states.
An edge may be labeled by ε, the empty
string
• We can have transitions without any input
character consumption.
28
Transition Graph
The transition graph for an NFA recognizing the
language of regular expression (a|b)*abb
all strings of a's and b's ending in the
particular string abb
a
start a b b
0 1 2 3
b S={0,1,2,3}
Σ={a,b}
S0=0
F={3}
29
Transition Table
The mapping T of an NFA can be represented
in a transition table
State Input Input Input
a b ε
0 {0,1} {0} ø
a a b b
0 0 1 2 3 YES
a a b b
0 0 0 0 0 NO
31
Another NFA
a
a
start
b
b
aa*|bb*
32
Deterministic Finite Automata (DFA)
33
DFA example
A DFA that accepts (a|b)*abb
34
Simulating a DFA: Algorithm
How to apply a DFA to a string.
INPUT:
An input string x terminated by an end-of-file character
eof.
A DFA D with start state So, accepting states F, and
transition function move.
OUTPUT: Answer ''yes" if D accepts x; "no" otherwise
METHOD
Apply the algorithm in (next slide) to the input string x.
The function move(s, c) gives the state to which there is
an edge from state s on input c.
The function nextChar() returns the next character of
the input string x.
35
Simulating a DFA
s = so;
c = nextchar();
while ( c != eof ) {
s = move(s, c);
c = nextchar();
}
if ( s is in F ) return
"yes";
DFA accepting (a|b)*abb
else return "no";
37
Design of a Lexical Analyzer Generator
Two algorithms:
1- Translate a regular expression into an NFA
(Thompson’s construction)
38
From regular expression to an NFA
It is known as Thompson’s construction.
Rules:
1- For an ε, a regular expressions, construct:
start a
39
From regular expression to an NFA…
2- For a composition of regular expression:
Case 1: Alternation: regular expression(s|r), assume
that NFAs equivalent to r and s have been
constructed.
40
40
From regular expression to an NFA…
Case 2: Concatenation: regular expression sr
ε
…r …s
Case 3: Repetition r*
41
From RE to NFA:Exercises
42
From an NFA to a DFA
(subset construction algorithm)
Rules:
Start state of D is assumed to be unmarked.
Start state of D is = ε-closer (S0),
where S0 - start state of N.
43
NFA to a DFA…
ε- closure
ε-closure (S’) – is a set of states with the following
characteristics:
1- S’ € ε-closure(S’) itself
2- if t € ε-closure (S’) and if there is an edge labeled
ε from t to v, then v € ε-closure (S’)
3- Repeat step 2 until no more states can be added
to ε-closure (S’).
E.g: for NFA of (a|b)*abb
ε-closure (0)= {0, 1, 2, 4, 7}
ε-closure (1)= {1, 2, 4}
44
NFA to a DFA…
Algorithm
While there is unmarked state
X = { s0, s1, s2,..., sn} of D do
Begin
Mark X
For each input symbol ‘a’ do
Begin
Let T be the set of states to which there is a transition ‘a’ from state si
in X.
Y= ε-Closer (T)
If Y has not been added to the set of states of D then {
Mark Y an “Unmarked” state of D add a transition from X to Y labeled a
if not already presented
}
End
End 45
NFA for identifier: letter(letter|digit)*
ε
letter
3 4
ε ε
start
letter ε ε
0 1 2 7 8
digit ε
ε 5 6
46
NFA to a DFA…
Example: Convert the following NFA into the corresponding
DFA. letter (letter|digit)*
A={0}
B={1,2,3,5,8}
start letter C={4,7,2,3,5,8}
A B
D={6,7,8,2,3,5}
letter digit
letter
digit D digit
C
letter
47
Exercise: convert NFA of (a|b)*abb in to DFA.
48
Other Algorithms
49
The Lexical- Analyzer Generator: Lex
The first phase in a compiler is, it reads the
input source and converts strings in the source
to tokens.
Lex: generates a scanner (lexical analyzer or
lexer) given a specification of the tokens using
REs.
The input notation for the Lex tool is referred to as
the Lex language and
The tool itself is the Lex compiler.
The Lex compiler transforms the input patterns into a
transition diagram and generates code, in a file
called lex.yy.c, that simulates this transition diagram.
50
Lex…
51
General Compiler Infra-structure
Parse tree
Program source Tokens Parser
Scanner Semantic
(tokenizer) Routines
(stream of
characters) Annotated/decorated
tree
Analysis/
Transformations/
Symbol and optimizations
literal Tables
IR: Intermediate
Representation
Code
Generator
Assembly code
52
Scanner, Parser, Lex and Yacc
5353
Generating a Lexical Analyzer using Lex
Lex is a scanner generator ----- it takes lexical specification as
input, and produces a lexical analyzer written in C.
Lex source
program Lex compiler lex.yy.c
lex.l
lex.yy.c
C compiler a.out
Lexical Analyzer
54
Lex specification
➢ Program structure C declarations in %{
...declaration section... %}
%%
P1 { action1 }
...rule section... P2 { action2 }
%%
...user defined functions...
Rules section – regular expression <--> action.
• The actions are C program.
Declaration section – variables, constants
55
Skeleton of a lex specification (.l file)
x.l *.c is generated after
running
%{
< C global variables, prototypes, This part is copied as–is to
comments > the top of the generated
C file
%}
Substitutions simplifies
[DEFINITION SECTION] pattern matching
57
Design of a Lexical Analyzer Generator:
RE to NFA to DFA
Thompson’s
construction
59
Combining and simulation of NFAs of a Set of
Regular Expressions: Example 1
start a
a {action1} 1 2
start b
abb {action2} a b
3 4 5 6
a*b+ {action3}
start a
Must find the longest b
prefix match: 7 b 8
Continue until no further
moves are possible a Action 1
ε 1 2
start b
a a b a*b+ b
0 ε 3 a 4 5 6
0 2 7 8
1 4 a ε Action 2
b
3 7 7 8 b
7 None a
Action 3 Action 3
60
Simulating NFA…
ε-closure({0}) = {0,1,3,7}
move({0,1,3,7},a) = {2,4,7}
ε-closure({2,4,7}) = {2,4,7}
move({2,4,7},a) = {7}
ε-closure({7}) = {7}
move({7},b) = {8}
ε-closure({8}) = {8}
move({8},a) = ∅
61
Combining and simulation of NFAs of a Set of
Regular Expressions: Example 2
start a
a {action1} 1 2
start b
abb {action2} a b
3 4 5 6
a*b+ {action3}
start a
When two or more b
accepting states are 7 b 8
reached, the first action
is executed a Action 1
ε 1 2
start b
a b b b
0 ε 3 a 4 5 6
0 2 5 6
1 4 8 8 ε Action 2
a b
3 7 7 8 b
7 None a Action 3
Action 2
Action 3 62
DFA's for Lexical Analyzers
NFA DFA. Transition table for DFA
State a b Token
found
0137 247 8 None
247 7 58 a
8 - 8 a*b+
7 7 8 None
58 - 68 a*b+
68 - 8 abb
64
Pattern matching examples
65
Meta-characters
66
Lex Regular Expression: Examples
• an integer: 12345
[1-9][0-9]*
• a word: cat
[a-zA-Z]+
• a (possibly) signed integer: 12345 or -12345
[-+]?[1-9][0-9]*
• a floating point number: 1.2345
[0-9]*”.”[0-9]+
67
Regular Expression: Examples…
• a delimiter for an English sentence
“.” | “?” | ! OR
[“.””?”!]
• C++ comment: // call foo() here!!
“//”.*
•white space
[ \t]+
• English sentence: Look at this!
([ \t]+|[a-zA-Z]+)+(“.”|”?”|!)
68
Two Rules
69
Lex variables
yyin - of the type FILE*. This points to the current file
being scanned by the lexer.
yyout - Of the type FILE*. This points to the location
where the output of the lexer will be written.
• By default, both yyin and yyout point to standard input
and output.
yytext – variable, a pointer to the matched strings (char
*)
yyleng - Gives the length of the matched pattern.
yylineno - Provides current line number information.
70
Lex functions
71
Lex predefined variables
72
Let us run a lex program
73
Lex : programs
The first example is the shortest possible lex file:
%%
Input is copied to output, one character at a time.
The first %% is always required, as there must
always be a rules section.
However, if we don’t specify any rules, then the
default action is to match everything and copy it to
output.
Defaults for input and output are stdin and stdout,
respectively.
Here is the same example, with defaults explicitly
coded:
74
Rule %%
section /* match everything except newline */
. ECHO;
/* match newline */
\n ECHO;
%%
int yywrap(void) { Invokes the
return 1; Lexical
analyzer
}
int main(void) {
User yylex();
definition return 0;
section
}
75
Developing Lexical analyzer using
Lex : Linux (Fedora)
vi – used to edit lex and yacc source files.
w – save
q – quit
w filename – save as
wq – save and quit
q! – exit overriding change
78
How to compile and run LEX programs...
4. Press esc
5. Press :wq
6. lex lab1.l
7. gcc lex.yy.c -ll
8. ./a.out <hello.c
79
Examples (more) Regular
definitions
%% %{
/*Match every thing #include <stdio.h>
except new line*/ %}
digit [0-9]
. ECHO;
letter [A-Za-z]
/*Match new line*/ id {letter}({letter}|{digit})*
\n ECHO; %%
%% {digit}+ { printf(“number: %s\n”, yytext); }
int yywrap(void) { {id} { printf(“ident: %s\n”, yytext); }
. { printf(“other: %s\n”, yytext); }
return 1; %%
} main()
int main(void) { {
yylex(); yylex();
} Translation
retrun 0; rules
}
80
Example :Finding the number of identifier in a given
program
digit [0-9]
letter [A-Za-z]
%{
int count=0;
%}
%%
{letter}({letter}|{digit})* count++;
%%
int main(void) {
yylex();
printf(“The number of identifiers are=%4d\n”, count);
return 0;
}
81
Example: Here is a scanner that counts the number of
characters, words, and lines in a file.
%{
int nchar, nword, nline;
%}
%%
\n { nline++;}
[^ \t\n]+ { nword++, nchar += yyleng; }
. { nchar++; }
%%
int main(void) {
yylex();
printf("%d\t%d\t%d\n", nchar, nword, nline);
return 0;
}
82
%{ /* definitions of manifest constants */
LT, LE, EQ, NE, GT, GE, IF, THEN, ELSE, ID, NUMBER,
Regular definitions
RELOP */
%}
delim [ \t\n]
ws {delim}+ Return token to parser
letter [A-Za-z]
digit [0-9]
id {letter}({letter}|{digit})*
number {digit}+(\.{digit}+)?(E[+\-]?{digit}+)?
%%
{ws} {/*no action and no return*/ }
if {return IF;} Token attribute
then {return THEN;}
else {return ELSE;}
{id} {yylval = install_id(); return ID;}
{number} {yylval = install_num(); return NUMBER;}
“<“ {yylval = LT; return RELOP;}
“<=“ {yylval = LE; return RELOP;}
“=“ {yylval = EQ; return RELOP;}
Install yytext as identifier
“<>“ {yylval = NE; return RELOP;} in symbol table
“>“ {yylval = GT; return RELOP;}
“>=“ {yylval = GE; return RELOP;}
%%
int install_id() {}
int install_num() {} 83
Assignment on Lexical Analyzer
84
1. Write a program in LEX to count the no of
consonants and vowels for a given C and C++ source
programs.
2. Write a program in LEX to count the no of:
(i) positive and negative integers
(ii) positive and negative fractions.
For C and C++ source programs
3. Write a LEX program to recognize a valid C, C++, and
Java programs.
85
The MINI Language Introduction
Assumptions:
Source code – MINI language
Target code – Assembly language
Specifications:
➢ There are no procedures and declarations.
➢ All variables are integer variables, and variables are
declared simply by assigning values to them.
➢ There are only two control statements:
✓ An if – statement and
✓ A repeat statement
➢ Both the control statements may themselves
contain statement sequences.
86
The MINI Language Introduction...
➢ An if – statement has an optional else part and must
be terminated by the key word end.
➢ There are also read and write statements that
perform input/output.
➢ Comments are allowed with curly brackets,
comments cannot be nested.
➢ Expression in MINI are also limited to Boolean and
integer arithmetic expressions.
➢ A Boolean expressions consists of a comparison of
two arithmetic expressions using either of the two
comparison operators < and =.
87
The MINI Language...
➢ An arithmetic expression may involve integer constants,
variables, parenthesis, and any of the four integer
operators +, -, *, and / (integer division).
➢ Boolean expressions may appear only as tests in
control statements – i.e. There are no Boolean
variables, assignment, or I/O.
➢ Here is a sample program in this language for factorial
function.
88
{ sample program
in MINI language – computes factorials
}
read x; { input an integer }
if x > 0 then { don’t compute if x<= 0}
fact:= 1;
repeat
fact := fact * x ;
X:= x-1
until x = 0;
write fact { output factorial of x}
end
89
The MINI Language...
In addition to the tokens, MINI has the following
lexical conventions:
➢ Comments : are enclosed in curly brackets {...} and
cannot be nested.
➢ White space : consists of blanks, tabs, and
newlines.
➢ The principles of longest substring is followed in
recognizing tokens.
90
Design a scanner for MINI language
91