Lexical and Syntax

Analysis
Main books
• Concepts of Programming Languages
• Understanding Programming Languages
Sameha Moogab 1
 Introduction
• Three different approaches to implementing programming languages.
• The compilation approach uses a program called a compiler, which translates
programs written in a high-level programming language into machine code.
Compilation is typically used to implement programming languages that are used for
large applications.
• Pure interpretation systems perform no translation; rather, programs are interpreted
in their original form by a software interpreter. Pure interpretation is usually used
for smaller systems in which execution efficiency.
• Hybrid implementation systems translate programs written in high-level languages
into intermediate forms, which are interpreted.
Sameha Moogab 2
 Introduction(cont.)
• Compilers separate the task of analyzing syntax into two distinct parts,
lexical analysis and syntax analysis.
• The lexical analyzer deals with small- scale language constructs, such as
names and numeric literals.
• The syntax analyzer deals with the large-scale constructs, such as expressions,
statements, and program units.

Sameha Moogab 3
 The compilation process
Sameha Moogab 4
 Introduction(cont.)
• There are three reasons why lexical analysis is separated from syntax analysis:
1. Simplicity—Techniques for lexical analysis are less complex than those required
for syntax analysis, so the lexical-analysis process can be simpler if it is separate.
2. Efficiency—Although it pays to optimize the lexical analyzer, because lexical
analysis requires a significant portion of total compilation time, it is not fruitful to
optimize the syntax analyzer.
3. Portability—Because the lexical analyzer reads input program files and often
includes buffering of that input, it is somewhat platform dependent.
Sameha Moogab 5
 Lexical analysis
• A lexical analyzer is essentially a pattern matcher.
• A pattern matcher attempts to find a substring of a given string of characters
that matches a given character pattern.
• The lexical analyzer collects characters into logical groupings and assigns
internal codes to the groupings according to their structure.
• These logical groupings are named lexemes, and the internal codes for
categories of these groupings are named tokens.

Sameha Moogab 6
• Consider the following example of an assignment statement:
result = oldsum - value / 100;
Following are the tokens and lexemes of this statement:
Token Lexeme
IDENT result
ASSIGN_OP =
IDENT oldsum
SUB_OP -
IDENT value
DIV_OP /
INT_LIT 100
SEMICOLON
Sameha Moogab
; 7
 Lexical Analyzer
• Lexical analyzers extract lexemes from a given input string and produce the
corresponding tokens.
• The lexical-analysis process includes skipping comments and white space
outside lexemes, as they are not relevant to the meaning of the program.
• the lexical analyzer inserts lexemes for user- defined names into the symbol
table, which is used by later phases of the compiler.
• Finally, lexical analyzers detect syntactic errors in tokens, such as ill-formed
floating-point literals, and report such errors to the user.
Sameha Moogab 8
 Lexical Analyzer construction
• Three approaches to build a lexical analyzer:
– Write a formal description of the tokens and use a software tool
that constructs a table-driven lexical analyzer from such a
description.
– Design a state diagram that describes the tokens and write a
program that implements the state diagram.
– Design a state diagram that describes the tokens and hand
construct a table-driven implementation of the state diagram
Sameha Moogab 9
 State Diagram
• A state transition diagram, or just state diagram, is a directed graph.
• The nodes of a state diagram are labeled with state names.
• The arcs are labeled with the input characters that cause the transitions
among the states.
• An arc may also include actions the lexical analyzer must perform when the
transition is taken.

Sameha Moogab 10
 Lexical-analyzer construction
Example
• Suppose we need a lexical analyzer that recognizes only arithmetic
expressions, including variable names and integer literals as operands.
• Assume that the variable names consist of strings of uppercase letters,
lowercase letters, and digits but must begin with a letter.

Sameha Moogab 11
 Lexical-analyzer construction
Example(cont)
• There are 52 different characters (any uppercase or lowercase letter) that can begin a
name, which would require 52 transitions from the transition diagram’s initial state.
However, a lexical analyzer is interested only in determining that it is a name and is
not concerned with which specific name it happens to be.
• Therefore, we define a character class named LETTER for all 52 letters and use a
single transition on the first letter of any name.
• There are 10 different characters that could begin an integer literal lexeme. This
would require 10 transitions from the start state of the state diagram.
• define a character class named DIGIT for digits and use a single transition on any
character in this character class to a state that collects integer literals.
Sameha Moogab 12
 State Diagram

Sameha Moogab 13
/* Function declarations */
void addChar();
void getChar();
void getNonBlank();
int lex();
/* Character classes */
#define LETTER 0
#define DIGIT 1 /**************************************
#define UNKNOWN 99 ***************/ addChar();
/* Token codes */
#define INT_LIT 10 /* lookup - a function to lookup nextToken = SUB_OP;
#define IDENT 11 operators and parentheses
#define ASSIGN_OP 20 break;
#define ADD_OP 21 and return the token */
#define SUB_OP 22 case '*':
#define MULT_OP 23 int lookup(char ch) {
#define DIV_OP 24 addChar();
#define LEFT_PAREN 25 switch (ch) {
#define RIGHT_PAREN 26 nextToken = MULT_OP;
case '(':
/******************************************************/ break;
addChar();
/* main driver */ case '/':
nextToken = LEFT_PAREN;
main() {
break; addChar();
/* Open the input data file and process its contents */
case ')': nextToken = DIV_OP;
if ((in_fp = fopen("front.in", "r")) == NULL)
addChar(); break;4.2 Lexical Analysis 193
printf("ERROR - cannot open front.in \n");
else { nextToken = RIGHT_PAREN; default:
getChar(); break; addChar();
do { case '+': nextToken = EOF;
lex(); addChar(); break;
} while (nextToken != EOF); nextToken = ADD_OP; }
} break; return nextToken;
} case '-':
Sameha Moogab } 14
/* Parse identifiers */ /* Parentheses and operators */
case LETTER: case UNKNOWN:
addChar();
lookup(nextChar);
getChar();
getChar();
while (charClass == LETTER || charClass == DIGIT) {
break;
addChar();
getChar();
/* EOF */
} case EOF:
nextToken = IDENT; nextToken = EOF;
break; lexeme[0] = 'E';
/* Parse integer literals */ lexeme[1] = 'O';
case DIGIT: lexeme[2] = 'F';
addChar();
lexeme[3] = 0;
getChar();
break;
while (charClass == DIGIT) {
} /* End of switch */
addChar();
getChar(); printf("Next token is: %d, Next lexeme is %s\n",
} nextToken, lexeme);
nextToken = INT_LIT; return nextToken;
break; } /* End of function lex */

Sameha Moogab 15
 Lexical-analyzer
• Following is the output of the lexical analyzer of front C when used on
(sum + 47) /total
Next token is: 25 Next lexeme is (
Next token is: 11 Next lexeme is sum
Next token is: 21 Next lexeme is +
Next token is: 10 Next lexeme is 47
Next token is: 26 Next lexeme is )
Next token is: 24 Next lexeme is /
Next token is: 11 Next lexeme is total
Next token is: -1 Next lexeme is EOF

Sameha Moogab 16
 Syntax Analysis
• The part of the process of analyzing syntax that is referred to as syntax
analysis is often called parsing.
There are two distinct goals of syntax analysis:
• First, the syntax analyzer must check the input program to determine
whether it is syntactically correct.
• The second goal of syntax analysis is to produce a complete parse tree, or at
least trace the structure of the complete parse tree, for syntactically correct
input.
Sameha Moogab 17
 Parsing Techniques
• Parsers are categorized according to the direction in which they build parse
trees:
❖Hybrid parser, in which the tree is built from the root downward to the
leaves.
❖Bottom-up parser, in which the parse tree is built from the leaves upward to
the root.

Sameha Moogab 18
 Top-Down Parsers
• A top-down parser traces or builds a parse tree in <assign> => <id> = <expr>
=>A = <expr>
preorder. A preorder traversal of a parse tree begins => A = <expr> + <term>
with the root. => A = <term> + <term>
=> A = <factor> + <term>
=> A = <id> + <term>
• Each node is visited before its branches are followed. => A = B + <term>
=> A = B + <term> * <factor>
Branches from a particular node are followed in left- => A = B + <factor> * <factor>
to-right order. This corresponds to a leftmost => A = B + <id> * <factor>
=> A = B + C * <factor>
derivation. => A = B + C * <id>
=> A = B + C * A
leftmost derivation

Sameha Moogab 19
 Parsing Decision Problem
<assign>

<id> <expr>
=

A
<id> <expr>
*

( <expr> )
B
A=B*(A+C)
The most common top-down parsers choose the correct RHS for the <id>
+ <expr>

leftmost nonterminal in the current sentential form by comparing the

next token of input with the first symbols that can be generated by A <id>
the RHSs of those rules.
Sameha Moogab 20
C
 Recursive-descent Parser
• In general, choosing the correct RHS is not so straightforward, because some of the
RHSs of the leftmost nonterminal in the current sentential form may begin with a
nonterminal.
• The most common top- down parsing algorithms are closely related. A recursive-
descent parser is a coded version of a syntax analyzer based directly on the BNF
description of the syntax of language.
• A recursive-descent subprogram for a rule with a single RHS is relatively simple. For
each terminal symbol in the RHS, that terminal symbol is compared with next
Token. If they do not match, it is a syntax error. If they match, the lexical analyzer
is called to get the next input token. For each nonterminal, the parsing subprogram
for
Sameha that nonterminal is called.
Moogab 21
• <expr> → <term> {(+ | -) <term>}
• <term> → <factor> {(* | /) <factor>}
• <factor> → id | int_constant | ( <expr> )

The recursive-descent subprogram for the first rule in the previous example grammar, written in C, is
/* expr Parses strings in the language generated by the rule:
<expr> -> <term> {(+ | -) <term>}
*/
void expr() {
printf("Enter <expr>\n");
/* Parse the first term */
term();
/* As long as the next token is + or -, get
the next token and parse the next term */
while (nextToken == ADD_OP || nextToken == SUB_OP) {
lex();
term();
}
printf("Exit <expr>\n");
} /* End of function expr */

Sameha Moogab 22
/* term Parses strings in the language generated by the rule: /* Get the next token */
<term> -> <factor> {(* | /) <factor>) */ lex();
void term() { /* If the RHS is ( <expr>), call lex to pass over the
printf("Enter <term>\n"); left parenthesis, call expr, and check for the right

/* Parse the first factor */ parenthesis */

factor(); else {
if (nextToken == LEFT_PAREN) {
/* As long as the next token is * or /, get the next token and parse the next
factor */ lex();
while (nextToken == MULT_OP || nextToken == DIV_OP) { expr();
lex(); if (nextToken == RIGHT_PAREN)
factor(); lex();
} else

printf("Exit <term>\n"); error();

} /* End of if (nextToken == ... */
} /* End of function term */
/* It was not an id, an integer literal, or a left
/* factor Parses strings in the language generated by the rule:
parenthesis */
<factor> -> id | int_constant | ( <expr ) */
else
void factor() {
error();
printf("Enter <factor>\n");
} /* End of else */
/* Determine which RHS */
printf("Exit <factor>\n");;
if (nextToken == IDENT || nextToken == INT_LIT)
Sameha Moogab 23
} /* End of function factor */
 Bottom-Up Parsers

• A bottom- up parser constructs a parse tree by beginning at the leaves and

progressing toward the root.
• Bottom- up parsers are often called shift-reduce algorithms, because shift
and reduce are the two most common actions they specify

Sameha Moogab 24
Sameha Moogab 25