Learn Compiler Design: From B. K.

Sharma

UNIT I

Lexical Analysis: Token, Lexeme

Pattern and Role of Lexical
Analyzer
Learn Compiler Design: From B. K. Sharma

Unit I: Syllabus
• Introduction to Compiler
• Structure of a compiler
• Lexical Analysis
• Role of Lexical Analyzer
• Input Buffering
• Specification of Tokens
• Recognition of Tokens
Learn Compiler Design: From B. K. Sharma

Unit I: Syllabus
• Lex
• Finite Automata
• Regular
• Expressions to Automata
• Minimizing DFA.
Learn Compiler Design: From B. K. Sharma

Active Learning Activity: Diagnostic

Assessment

One- Minute Paper

List 6 phases of compiler.

List 8 components of structure of compiler.

Learn Compiler Design: From B. K. Sharma
Summary of Lesson 2: Structure of a Compiler:
The compiler's structure is modular consisting of following
8 components:
1: Lexical Analyzer takes as input a stream of characters and
produces as output a stream of words or phrases or tokens along
with their associated syntactic categories

2: Syntax Analyzer takes as input a stream of tokens and

recognizes the structure of tokens according to grammar of the
language producing parse tree or syntax tree as output

3: Semantic analyzer checks the static semantics of the

language and annotates the syntax tree with type information

4: Intermediate Code Generator produces code that is machine

independent which is portable code.
Learn Compiler Design: From B. K. Sharma
Summary of Lesson 2: Structure of a Compiler:
The compiler's structure is modular consisting of following
8 components:

5: Code Optimizer produces better /semantically equivalent

code

6: Target Code Generator produces final target low-level code

which in assembly language or machine language.

7: Symbol Table is a data structure used to store information

about identifiers, their types, and other attributes.

8: Error Handler ensures the compiler can detect and

report errors to the programmers, aiding in code
correction and improvement.
Learn Compiler Design: From B. K. Sharma

Mapping of Lesson with Course Outcome

(CO)
Lesson CO
Lesson 3: Lexical Apply the knowledge of
Analysis and Role of theory of computation in
Lexical Analyzer specifying and
recognizing tokens
Learn Compiler Design: From B. K. Sharma
Tokens, Lexeme or Pattern
Token:
a group of characters having a collective meaning.

Smallest individual unit of a language that are

recognized.

Token Types or classes:

Identifiers Keywords Operators

Numbers Delimiters Parentheses

Learn Compiler Design: From B. K. Sharma
Lexical Analysis and Lexical Analyzer

Tokens IR
Source Scanner Parser
code

Errors

Lexical Analysis:
The task concerned with breaking an input into its
smallest meaningful units, called tokens.
Lexical Analyzer (Scanner):
Program that reads input characters and produces
a sequence of tokens as output.
Learn Compiler Design: From B. K. Sharma

Tasks or Role of Lexical Analyzer: Scanner

Main Role:
Read the characters of source language (a stream
of characters) and break it up into tokens, the
smallest meaningful units of the source language.

Other Roles:
Remove the white space/tab Remove the comments

Interpret the compiler directives. Insert Tokens into ST

Generate Errors Send Tokens to Parser

Learn Compiler Design: From B. K. Sharma
Tokens, Lexeme or Pattern
Lexeme:

Lexeme is a particular instant of a token.

e.g. token: identifier, lexeme: a, x,y,2.5 etc.

Token: operators, lexeme: +, -, *, /

Token: Parentheses, lexeme: (, )

Token: keywords, lexeme: main, int

Learn Compiler Design: From B. K. Sharma
Tokens, Lexeme or Pattern
Pattern:

the rule describing how a token can be formed.

e.g.: identifier: ([a-z]|[A-Z]) ([a-z]|[A-Z]|[0-9])*

letter followed by letters and digits

e.g.: Number: [0-9]+

Learn Compiler Design: From B. K. Sharma

Lexical Analysis: Token Example

Let us consider a C-Language statement:

if (x==3)
Tokens are:
Keyword, LPAR, IDENT, EQ, NUMBER, RPAR

Token and Lexeme pairs are

<Key, “if”> <LPAR, “(“> <IDENT, “x”>

<LOP, “==”> <NUMB, “3”> <RPAR,”)”>

Learn Compiler Design: From B. K. Sharma

Lexical Analysis: Token Example

i f ( x = = 3 ) scanner IF, LPAR, IDENT, EQ, NUMBER,

RPAR, ..., EOF

character stream token stream

(must end with EOF)
Learn Compiler Design: From B. K. Sharma

Lexical Analysis: Token Attribute Example

y := 31 + 28*x Lexical analyzer

<id, “y”> <assign, “:=“> <num, 31> <+, > <num, 28> <*, > <id, “x”>

token

tokenval
(token attribute)
Parser
Learn Compiler Design: From B. K. Sharma

Lexical Analysis: Token Example

Let us consider another C statement:
if (x==y)
z =12;
else
z = 3;
i f ( x = = y ) \n \t z = 1 2 ; \n e l s e \n \t z = 3 ; \n

<KEY, “if”> <LPAR> <ID, “x”> <LOP, “==”> <ID, “y”>

<RPAR> <ID, “z”> <OP, “=“> <INT, “12“> <SEMIC>

<KEY, “else“> <ID, “z“> <OP, “=“> <INT, “3“> <SEMIC>

Learn Compiler Design: From B. K. Sharma

Active Learning Activity

One- Minute Paper
Consider the following code in C Language:
printf (“i=%d, j=%f, &i=%x\n”, i, j, &i);

The number of tokens find by the lexical analyzer

is ?

a) 10 b) 35 c) 12 d) 46
Learn Compiler Design: From B. K. Sharma

Active Learning Activity

One- Minute Paper

Everything inside “ ” in printf() is counted as a

single token.
Learn Compiler Design: From B. K. Sharma

Active Learning Activity

One- Minute Paper
Consider the following code in C Language:
#include<stdio.h>
int main()
{
printf(“%d + %d =%d”,3,1,4);
return 0;
}

The number of lexemes after pre-processing is ?

a) 10 b)12 c) 20 d) 5
Learn Compiler Design: From B. K. Sharma

Active Learning Activity

One- Minute Paper
During pre-processing, file inclusion, macro substitution
pre-processing directives, comments are removed.
Learn Compiler Design: From B. K. Sharma

What is the need for separating parser

from scanner?

Source Scanner Tokens Parser

code

Errors

Lexical analyzer does not have to be an individual

phase.
But having a separate phase simplifies the design
and improves the efficiency and portability.
Learn Compiler Design: From B. K. Sharma

What is the need for separating parser

from scanner?
Reasons for separating both analysis:

1) Simpler design.

Separation allows the simplification of one or the other.

Example: A parser with comments or white spaces is more

complex

2) Compiler efficiency is improved.

Optimization of lexical analysis because a large amount of

time is spent reading the source program and partitioning it
into tokens.
Learn Compiler Design: From B. K. Sharma

What is the need for separating parser

from scanner?
3) Compiler portability is enhanced.
Only the scanner requires to communicate with the outside
world

Input alphabet peculiarities and other device-specific

anomalies can be restricted to the lexical analyzer.

4) Specialization

Specialized techniques can be applied to improves the

lexical analysis process
Learn Compiler Design: From B. K. Sharma
Summary of Lesson 3: Lexical Analysis and Role of
Lexical Analyzer
1: A group of characters having a collective meaning or
smallest individual unit of a language that are recognized,
is called token.
2: Lexeme is a particular instant of a token.
3: The rule describing how a token can be formed is called
pattern.
4: The primary task of Lexical Analyzer is Read the
characters of source language (a stream of characters) and
break it up into tokens, the smallest meaningful units of the
source language.
5:The secondary tasks of Lexical Analyzer are: remove white
space and tab, remove comments, interpret compiler
directives, insert tokens into symbol table, lexical errors,
send tokens to parser.
Learn Compiler Design: From B. K. Sharma
Summary of Lesson 3: Lexical Analysis and Role of
Lexical Analyzer
6: Separating scanner from parser simplifies the design and
improves the efficiency and portability.
Learn Compiler Design: From B. K. Sharma

Active Learning Activity

One- Minute Paper

When expression sum=3+2 is tokenized then what

is the token category of 3?

a) Identifier b) Assignment operator

c) Integer Literal d) Addition Operator

Learn Compiler Design: From B. K. Sharma
Lexical Analysis: Questions

Explain the term token, lexeme and Pattern with

examples.

What is the need for separating the parser from

scanner?
Learn Compiler Design: From B. K. Sharma

Active Learning Activity: After Leaning

Process to Check Important Take away of
Things and Feedback for Improvement in
teaching and learning process
One-Minute Paper
1) What was the most interesting part of the session?
2) What was the most confusing part of the session?
3) Give a score out of 10 for this session:
If it is not 10 out of 10 tell me why not?