9/18/2012

Derivations vs Parses
Grammar is used to derive string or construct parser

A derivation is a sequence of applications of rules

• Starting from the start symbol

Context Free Grammars • S ⇒ ... ⇒ ... ⇒ ... ⇒ (sentence)

Leftmost and rightmost derivations

• At each derivation step, a leftmost derivation always replaces the
leftmost non-terminal symbol
• Rightmost derivation always replaces the rightmost one

Example Parse Tree

E → E * E | E + E | ( E ) | id Parse tree:
• Internal nodes are non-terminals
Leftmost derivation: • Leaves are terminals
E ⇒ E + E ⇒ E * E + E ⇒ id * E + E ⇒ id * id + E ⇒ ...
⇒ id * id + id * id It filters out the order of replacement and describes the hierarchy

Rightmost derivation: The same parse tree results from both the rightmost and leftmost derivations in
E ⇒ E + E ⇒ E + E * E ⇒ E + E * id ⇒ E + id * id ⇒ ... the previous example:
⇒ id * id + id * id E

E + E

E * E E * E

id id id id

Different Parse Trees Ambiguity

While the two derivations could have the same parse tree for id * id + id * id there A grammar G is ambiguous if there exists a string str  L(G) such that more than
can actually be 3 different trees: one parse trees derive str
E
E
We prefer unambiguous grammars.
E * E E * E
Ambiguity is the property of a grammar and not the language
id E + E E + E id
It is possible to rewrite the grammar to remove ambiguity
id E * E E * E id

id id id id
E

E + E

E * E E * E

id id id id

1
 9/18/2012

Removing Ambiguity Removing Ambiguity

Method 1: Specify precedence. Method 2: Specify associativity.
• When recursion is allowed, we need to specify associativity
You can build precedence into the grammar by having a different non-terminal for
each precedence level: For the previous example,
• Lowest level — highest in the tree (lowest precedence) E → E*E
• Highest level — lowest in the tree Allows both right and left associativity.
• Same level — same precedence E
We can rewrite it to force it either way:
For the previous example, E + T
Left associative :
E → E * E | E + E | ( E ) | id E → E*T
T T * F
rewrite it to: Right associative:
E → E+T|T T E → T*E
* F F id
T → T*F|F
F → id | ( E ) F id In a programming language, most operators are left associative.
id

id

Syntax Analysis Types of Parsers

We’ve only discussed grammar from the point of view of derivation. Universal parser
• Can parse any CFG grammar. (Early’s algorithm)
What is syntax analysis? • Powerful but extremely inefficient
• To process an input string for a given grammar, and compose the
derivation if the string is in the language Top-down parser
• It is goal-directed, expands the start symbol to the given sentence
• Two subtasks: • Only works for certain class of grammars
• to determine if string in the language or not • To start from the root of the parse tree and reach leaves
• to construct the parse tree • Find leftmost derivation
• Can be implemented efficiently by hand

Is it possible to construct such a parser?

Types of Parsers Parser Output

Bottom-up parser We have a choice of outputs from the parser:
• It tries to reduce input string to the start symbol • A parse tree (concrete syntax tree), or
• Works for wider class of grammars • An abstract syntax tree
• Starts at leaves and build tree in bottom-up fashion
• Find reverse order of the rightmost derivation Example Grammar:
• Automated tool generates it automatically E → int | ( E ) | E + E
and an input:
5 + ( 2 + 3 )
After lexical analysis, we have a sequence of tokens
INT:5 ‘+’ ‘(’ INT:2 ‘+’ INT:3 ‘)’

2
 9/18/2012

Parser Output Summary

The parse tree traces the operation of the parser. E We specify the syntax structure using CFG even if the programming language
itself is not context free.
E + E
Captures the nested structure but contains too much
information: INT:5 ( A parser can:
E )
• Parentheses (precedence encoded in tree • Answer if an input str  L(G)
hierarchy) E + E • and build a parse tree
• Single-successor nodes (could be • or build an AST instead
collapsed/omitted) INT:2 INT:3
• and pass it to the rest of compiler.

We prefer an Abstract Syntax Tree (AST):

• AST also captures the nested structure. PLUS
• AST abstracts from the concrete syntax.
• AST is more compact and easier to use. 5 PLUS

2 3

Parsing
We will study two approaches:

Top-down
• Easier to understand and implement manually

Parsing Bottom-up
• More powerful, can be implemented automatically

Top Down Parsers Parsing Using Backtracking

Recursive descent Approach: For a non-terminal in the derivation, productions are tried in some
• Simple to implement, use backtracking order until
• A production is found that generates a portion of the input,
Predictive parser or
• Predict the rule based on the 1st m symbols without backtracking • No production is found that generates a portion of the input, in which
• Restrictions on the grammar to avoid backtracking case backtrack to previous non-terminal.

LL(k) — predictive parser for LL(k) grammar Parsing fails if no production for the start symbol generates the entire input.
• Non recursive and only k symbol look ahead
• Table driven — efficient Terminals of the derivation are compared against input.
• Match — advance input, continue parsing
• Mismatch — backtrack, or fail

3
 9/18/2012

Parsing Using Backtracking Parsing Using Backtracking

Grammar: Input Derivation Action
E → T + E | T int * int E pick rightmost rule E → T
T → int * T | int | ( E ) int * int E⇒T pick rightmost rule T → ( E )
int * int E⇒T⇒(E) “(” does not match “int”
Input string: int * int E⇒T Failure, backtrack one level.
int * int
int * int E ⇒ T ⇒ int pick next rule T → int
int * int E ⇒ T ⇒ int “int” matches input “int”
Start symbol: E
int * int E⇒T We have more tokens, so this is
failure too. Backtrack.
Assume:
int * int E ⇒ T ⇒ int * T Match int * Expand T.
• When there are alternative rules, try right rule first
int * int E ⇒ T ⇒ int * T ⇒ int * ( E ) pick rightmost rule E → ( E )
int * int E ⇒ T ⇒ int * T ⇒ int * ( E ) “(” does not match input “int”
int * int E ⇒ T ⇒ int * T Failure, backtrack one level.
int * int E ⇒ T ⇒ int * T ⇒ int * int pick next rule T → int
int * int E ⇒ T ⇒ int * T ⇒ int * int Match whole input. Accept.

Implementation Problems
Create a procedure for each non-terminal: Unclear what to label the last case with.
1. Checks if input symbol matches a terminal symbol in the grammar rule
2. Calls other procedure when non-terminals are part of the rule What if we don’t label it at all and make it the default?
3. If end of procedure is reached, success is reported to the caller
Consider parsing 5 + 5:
E → int | ( E ) | E + E
We’d find INT and be done with the parse with more input to consume. We’d
void E() { want to backtrack, but there’s no prior function call to return to.
switch(lexer.yylex()) {
case INT: eat(INT); break; What if we put the call to E() prior to the switch/case?
case LPAREN: eat(LPAREN); E(); eat(RPAREN); break;
case ???: E(); eat(PLUS); E(); break; Then E() would always make a recursive call to E() with no end case for the
} recursion.
}

Left Recursion Removing Left Recursion

A production is left recursive if the same nonterminal that appears on the LHS In general, we can eliminate all immediate left recursion:
appears first on the RHS of the production. A → A x | y

Recursive descent parsers cannot deal with left recursion. By changing the grammar to:
A → y A’
However, we can rewrite the grammar to represent the same language without the A’ → x A’ | 
need for left recursion.
Not all left recursion is immediate may be hidden in multiple production rules
A → BC | D
B → AE | F

There is a general approach for removing indirect left recursion, but we’ll not
worry about if for this course.

4
 9/18/2012

Recursive Descent Summary

Recursive descent is a simple and general parsing strategy
• Left-recursion must be eliminated first
• But this can be done automatically

It is not popular because of its inefficiency:

• Backtracking re-parses the string
• Undoing semantic actions (actions taken upon matching a production
much like the actions from our lexer) may be difficult!

Techniques used in practice do no backtracking at the cost of restricting the class

of grammar