Compiler design

Department of Computer Science

FACULTY OF TECHNOLOGY
Debre Markos University Burie campus
Year IV Semester I

by…
Mr. Birku L.
B.Sc. CS, M.Sc. SE
The Academic year of 2015 E.C
 Revision chapter 1
 Answer the following question among
them
1. What are translators?
2. What is ? Explain about
3. Explain about
4. Define and r?
5. What are the and between
and ?
6. Explain about and multi-pass
7. Explain about the
8. Explain about and ?
9. Write about with ?
 Chapter 2 Outline
 Lexical Analysis?
 Role of lexical analysis
 Issue in lexical analysis
 Token ,Patter, Lexeme
 Attributes of a token
 Input Buffering
 Buffer pairs
 Sentinels (Guards)
 Specification of token using Regular expression
 Regular Expression
 Recognizing of token using Transition diagram
 Transition diagram
 Lexical Analysis

 The first phase of a compiler, This phase is the part

of and the program text: The text is
read and divided into , is a of a
program.
 so the translated in to a
.by removing

Source code Lexical Analysis List of Token

Error Messages
 Role of Lexical Analysis
1. To the characters
2. To them into
3. To a sequence of used by the
for syntax analysis As an of parser
4. To Interact with the
◦ To Insert
token
Source Lexical To semantic
Parser
program Analyzer analysis
getNextToken

Symbol table
 Role of Lexical Analysis

5. To
• Comments : such as // or /* -------*/
• Whitespaces : such as blank, newline, tab, …

• To correlate generated by the

with the
• To keep track of the number of seen
• To a line with each
Sample of is
 A mistake in a lexeme,
 for examples, typing far instead of for,
 Unclosed comment
 For example /* read a value….
 Role of Lexical Analysis
 Sometimes are in a cascade of
two phases
: - to produce a sequence of tokens as an
output.(main task)
:-the scanner is responsible for simple task

 There are several reasons for separating the analysis phase

into lexical and syntax analysis.

: is perhaps the most important

consideration
: liberty to apply specialized
techniques that serves only lexical tasks,
:- Input device peculiarities are
restricted to the lexical analyzer
 Token, Lexeme, Pattern:
: Token is a sequence of characters that can be treated
as a or a of a program.
 There are two categorized of token those are:-
1. Valid Tokens
2. Invalid tokens

Keyword White Space (blank)

Identifiers New line (/n)

Operators Tab space (/t)

Special symbols Comment // or /* -------*/

Number Pre-processor directives # include<header.h>

Literal/Constant Pre-processor directives # define avg(a+b/2)

 Token, Lexeme, Pattern:
: A set of in the input for which the same token is
produced as output. This set of strings is by a called a
pattern associated with the token. Patter uses a
to identify tokens.
: A lexeme is a in the source
program that is by the for a token.
: Description of token

Keyword(if) If characters i, f
Id X Letter followed by seq. of alphanumeric l
(l+d)*
Relation <,<=,=,<>,>=,> < or <= or = or < > or >= or lette
Operator followed by letters & digi (l+d)*
Special symbols :,:,(,),{,},’,”,., : or : or ( or ) or { or } or’ or ” or . or ,
Number 3.14 Any numeric constant
Literal/Constant "core" Anything but “, surrounded by “
 Attributes of a token
 When lexemes match the then
the must provide with the
token.
is placed in a symbol-table entry and the
lexical analyzer represents these lexemes in the form of tokens
as:< , >
: an abstract symbol is used during syntax analysis,
: points to an entry in the symbol table for this
token.
. E = M * C ** 2 has 7 tokens and associated attribute-values:
E :- <(id,1) reference to symbol-table entry for E>
= :- <assign_op,>
M :- < (id,2) reference to symbol-table entry for M>
* :- <mult_op,>
C :- < (id,3) reference to symbol-table entry for C>
** :- <exp_op,>
2 :- <num, integer value 2>
 Attributes of a token

y := 31 + 28*x Lexical analyzer

<id, “y”> <assign, > <num, 31> <+, > <num, 28> <*, > <id, “x”>

token-name
Token(var)
(token attribute) Parser
11
 Some example of token

1. #include <iostream.h>
Handle by
2. #include <conio.h>
1 2 3 4
3. void main()
void main ( )
4. // his is a hello world program X Comment, Invalid token
5. { 5
{

6. cout << “Hello World!\n"; 6 7 8 9 10

cout < < “Hello world!\n” ;
7. } 11
}

 So The of is 11
 Token of a Hello World program
void main ( )
{ cout < <
“Hello world!\n” ; }
Keyword Identifier Numbers Literal/ Special Operators
Constant symbols

void main - “Hello ( <<

world!\n
”
cout - - - ) -
- - - - { -
- - - - } -
- - - - ; -
2 1 0 1 5 1
 Some example of token

1. #include <iostream.h>
Handle by
2. #include <conio.h>

3. void main() void main ( )

4. { {
5. int a, b, c; int a , b , c ;
6. cout << "Enter two numbers to add\n"; cout < < "Enter two .." ;
7. cin >> a >> b; Cin > > a > > b ;
8. c = a * b+10^2; c = a * b + 10 ^ 2 ;

9. cout <<“The value of c = " << c << endl; cout < < “The…"
< < c < < ;

10. } }

 So The of is
46
 Tokens of Adding two Numbers
Keyword Identifier Numbers Literal/ Special Operators
Constant symbols
void a 10 “Enter...” ( <<
int b “Sub....” ) >>
cout c { +
cin main } *
; =
,
4 4 1 2 6 5

A.O R.O L.O Unary. O Assi. O B.O

+ = <<
* >>
^
2 0 0 0 1 3
 Some example of token
 So The of is 60
1. #include <iostream.h>
Handle by
2. #include <conio.h>
3. void main() void main ( )
4. { {
5. int n,m, r; int m , n , r ;
6. cout << "Enter two No to get G,C,F\n"; cout < < "Enter two ….." ;
7. cin >> n >> m;
Cin > > m > > n ;
8. while (n!=0)
while ( n ! = 0 ) ;
9. { {
10. r = n % m; r = n % m ;
11. m=n; m = n ;
12. n=r; n = r ;
13. } }
14. cout <<“G.C.F of entered numbers = " << m << endl; cout < < “Sum."
15. } } < < M < < ;
 E.g. Token of a G.C.F program
Keyword Identifier Numbers Literal/ Special Operators
Constant Symbols
void n 0 “Enter...” ( <<
int m “C.F.G....” ) >>
cout r { %
cin Main } !=
while ; =
,
4 4 1 2 6 5

A.O R.O L.O Unary. O Assi. O B.O

% != = <<
>>
2 1 0 0 1 2
 Input Buffer
is also commonly known as the
When referring to computer memory,
is a location that holds all incoming information
before it continues to the .
• This is the of the that the entire
one a
• The scanner accounts for ~ of total compile time.
• There are general to the implementation of a
:
such as Lex or Flex compiler
to produce the lexical analyzer from a regular expression

using I/O facilities of that language to

read the input.
and explicitly
manage the reading of input.
 Input Buffer
 To speed up reading the source program, input buffer using
d/t technique
the last lexeme under process will
be over-written when we reload the buffer.
handling large look
ahead safely
–improvement which saves time checking buffer
end
 There are two pointers to the input are maintained
– marks the beginning of the current
lexeme
– scans ahead until a pattern match is found

i ,i ,a b, a b, b, a b, a b, a b, a b
 Buffer Pair
The input buffer has two halves with N
characters in each half.
might be the of a block like 1024 or 4096.
 Mark the stream with a special character
.
 Maintain two pointers into the buffer marking
of the current lexeme.
 two pointers to the input are maintained
– marks the beginning of the current
lexeme
– scans ahead until a pattern match is
found
 Buffer Pair
The input buffer has two halves with N
characters in each half.
might be the of a block like 1024 or 4096.
 Mark the with a special character
. Maintain two pointers into the buffer marking
of the current lexeme.
 two pointers to the input are maintained
– marks the beginning of the current
lexeme
– scans ahead until a pattern match is
found
 Simple Input Buffering algorithm
both to the of the
next to be found.
 The is until a for a
pattern is found. the lexeme is set both
pointers to the following the lexeme.
 Code to advance forward pointer:
forward at end of half
second half;
:= forward + 1
forward at end of half
reload first half;

forward := forward + 1;
 Sentinels (Guards)

• to test if it is at the
• to determine what character is read ( )
added at each
: Optimize the common case by the
of to one per of fwd.
: Extend each to hold a at the end.
• This is a that cannot occur in a
( )
• It the need for some (fill
, or processing).
• when other than at the of a it means that the
is at
 Simple Sentinels algorithm
• For almost character perform :
1. Is the character ?
2. Is the at the end of the ?
3. Is the at the end of the ?
• This can be to by using .
• Add an eof character past the end of .
• Use the code from next slide to .
:= forward + 1;
forward = then
forward at end of then
second half;
:= forward + 1 end
forward at the then
first half;
move forward to of
end
else end;
 Specification of Tokens
 There are Two Dual Notions to specification and
recognition of tokens.

• (grammar or regular expression)

• (automaton)

• Many theorems to transforms one approach automatically to

another
 Specification of Tokens
Definition of : - defined in regular
expression
is an important notation for specifying
patterns.
 Each matches a set of , so regular
expressions serve as names for a set of .
 Programming language can be described by
.
E.g. letter A|B|…|Z|a|b|…|z
digit 0|1|2|…|9
Id  letter(letter|digit)*
Id {x ,y, y, count, d12, h45r5}
 The specification of is an example of a
.
are easy to and have
efficient implementation.
 Specification of Tokens
 Let us understand how the undertakes the
following terms, Terminology:

 An abstract entity that we shall not define formally

e.g. (such as “ ” in ) , and is
examples of symbols.

A of out of which we build larger structures.

 An alphabet is typically denoted using the Greek word ,
e.g., is a set of binary alphabets,
is a set of Hexadecimal
alphabets,
is a set of English language alphabets.

 Any is called a .
of the is the of occurrence of
,
 Specification of Tokens
denoted by a Greek word
,
of denoted in
 e.g. 0010, |0010| = 4, oo7,|oo7| = 3, ,||=0

 is as a finite set of over some

.
are considered as , and
can be performed on them.
 can be described by means of
.
set is a
The set containing is a language
• The set of all is a language
e.g. r = {ab} => L(r) => L(ab)
 Specification of Tokens
 There are important that can be to
.
 Let be the set of and be the set of
: - L U M { x | x is in L or in M} E.g. L UM = {a,b,c,1,2}
:- {st| s is in L and t is in M}
E.g. L M ={a1,a2,b1,b2,c1,c2}
:- L* = i=0,…, Li , means
concatenation of L.
E.g. L* is the set of all string of letter and empty set,
{} or {a} or {a,b} or {a,b,c},
:- L+ = i=1,…, Li means
concatenation of L.
E.g. L+ is the set of all string of one or more latter without empty
set {a} or {a,b} or {a,b,c}
:- Ln , is any number of
E.g. L0 = {}; ,L2 ={2-latter string} = {aa,ab,ac,ba,bb,bc,ca,cb,cc…}
 Regular Expression
 are a / Techniques for Constructing of
( ) From an .
 Regular expressions are an to describe .
 The defined by is known as
.
 The defined by is known as

A notation (rule) that allows us to define a

pattern in a high level language.
Each regular expression denotes a
language (the set of sentences relating to the
)

: Each in a program can be in a

and regular expression denotes a .
 Regular Expression
The rule of regular expression can be categorized in to
and

is a regular expression language

is a regular expression

Suppose s are denoting languages

respectively then
is a denoting
is a denoting
is a denoting
is a denoting
Note:- A defined by a is called a
 Regular Expression
 The precedence of regular expression are
(.), and (|) are left associative
has the highest precedence
has the second highest precedence.
has the lowest precedence of all.
: - All are Left-Associative. Parentheses are dropped as
allowed by precedence rules

Example Let r is a set of {1,2}

Let s is a set of {a, b}
Let t is a set of {7,f}
( s | t r *) r* tr* s|tr*
 Algebraic Properties of RE

r|s=s|r | is commutative
r | (s | t) = (r | s) | t | is associative
(r s) t = r (s t) concatenation is associative
r(s|t)=rs|rt concatenation distributes over |
(s|t)r=sr|tr
r = r  Is the identity element for
r = r concatenation
r* = ( r |  )* relation between * and 
r** = r* * is idempotent
:- A language may be represented by or equivalent
. idempotent operation is one that has no
if it is called than with the same
 Notational shorthand’s
 Certain constructs occur so frequently in
that it is convenient to introduce
for them.
:- The
Means “ or of ” if is a
that denotes the then is a
regular expression that denotes the language
means.
(r)+ denoting (L(r)) +
( r )+ digit+
the * operator
R* is shorthand for |r+

r? is a shorthand for r| (E(+|-)?digits)?

[a-z] denotes a|b|c|…|z [A-Za-z] [A-Za-z0-9]

 Notational shorthand’s

If is a , then:
• means occurrence of x.
, it can generate { }

• means occurrence of x.
i.e., it can generate } or

• means occurrence of
i.e., it can generate

is all language.
is all language.
is all used in mathematics.
.
 Notational shorthand’s
 letter = [a – z] or [A – Z]
 digit = 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 or [0-9]
 sign = [ + | - ]

 Digit = (digit)+
 Letter = (letter) +
 Identifier = (letter)(letter | digit)*
 Decimal = (sign)?(digit)+
 Real Number ([digit] + “.” [digit] *) | (“.” [digit] +)
 keyword -> keyword
 Relation operator -> < | > | <= | >= | = | <>
The only problem left with the is how to verify
the of a used in the
of of a . A well-accepted solution is