Types of parsing Top-down parsers start at the root of derivation tree and ll in picks a production and tries to match

ch the input may require backtracking some grammars are backtrack-free (predictive)

Example grammar This is a grammar for simple expressions: 1 2 3 4 5 6 7 8 9 <goal> ::= <expr> <expr> ::= <expr> + <term> | <expr> - <term> | <term> <term> ::= <term> * <factor> | <term> / <factor> | <factor> <factor> ::= number | id

Bottom-up parsers start at the leaves and ll in start in a state valid for legal rst tokens as input is consumed, change state to encode possibilities (recognize valid prexes) use a stack to store both state and sentential forms

Consider parsing the input string x - 2 * y

CMSC 430

Lecture 4, Page 1

CMSC 430

Lecture 4, Page 3

Top-down parsing A top-down parser starts with the root of the parse tree. It is labeled with the start symbol or goal symbol of the grammar. To build a parse, it repeats the following steps until the fringe of the parse tree matches the input string. 1. At a node labeled A, select a production with A on its lhs and for each symbol on its rhs, construct the appropriate child. 2. When a terminal is added to the fringe that doesnt match the input string, backtrack. 3. Find the next node to be expanded. (Must have a label in N T )

Backtracking parse example One possible parse for x - 2 * y Prodn 1 3 4 7 9 7 9 5 7 9 9

The key is selecting the right production in step 1. should be guided by input string

Sentential form <goal> <expr> <expr> - <term> <term> - <term> <factor> - <term> <id> - <term> <id> - <term> <id> - <term> <id> - <factor> <id> - <num> <id> - <num> <id> - <term> <id> - <term> * <factor> <id> - <factor> * <factor> <id> - <num> * <factor> <id> - <num> * <factor> <id> - <num> * <factor> <id> - <num> * <id> <id> - <num> * <id>

Input x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y x - 2 * y
Lecture 4, Page 4

CMSC 430

Lecture 4, Page 2

Example Another possible parse for x - 2 * y Prodn 1 2 2 2 2 2 Sentential form <goal> <expr> <expr> + <term> <expr> + <term> + <term> <expr> + <term> + <term> + <term> <expr> + <term> + <term> + <term> + Input x x x x x x x -

Eliminating left recursion To remove left recursion, we can transform the grammar. Consider the grammar fragment: 2 2 2 2 2 2 2 * * * * * * * y y y y y y y <foo> ::= <foo> | where and do not start with <foo>. We can rewrite this as: <foo> ::= <bar> <bar> ::= <bar> | where <bar> is a new non-terminal.

If the parser makes the wrong choices, the expansion doesnt terminate. This isnt a good property for a parser to have.

This fragment contains no left recursion.

CMSC 430

Lecture 4, Page 5

CMSC 430

Lecture 4, Page 7

Left recursion Top-down parsers cannot handle left-recursion in a grammar. Formally, a grammar is left recursive if A N T such that a derivation A + A for some string .

Example Our expression grammar contains two cases of left recursion <expr> ::= | | <term> ::= | | <expr> + <term> <expr> - <term> <term> <term> * <factor> <term> / <factor> <factor>

Applying the transformation gives <expr> ::= <term> <expr > <expr > ::= + <term> <expr > | - <term> <expr > | <term> ::= <factor> <term > <term > ::= * <factor> <term > | / <factor> <term > |
CMSC 430 Lecture 4, Page 6 CMSC 430 Lecture 4, Page 8

Our simple expression grammar is left recursive.

Eliminating left recursion A general technique for removing left recursion arrange the non-terminals in some order A1, A2, . . . , An for i 1 to n for j 1 to i-1 replace each production of the form Ai ::= Aj with the productions Ai ::= 1 | 2 | . . . | k , where Aj ::= 1 | 2 | . . . | k are all the current Aj productions. eliminate any immediate left recursion on Ai using the direct transformation This assumes that the grammar has no cycles (A + A) or productions (A ::= ).

How much lookahead is needed? We saw that top-down parsers may need to backtrack when they select the wrong production Do we need arbitrary lookahead to parse CFGs? in general, yes Fortunately large subclasses of CFGs can be parsed with limited lookahead most programming language constructs can be expressed in a grammar that falls in these subclasses Among the interesting subclasses are LL(1) and LR(1).

CMSC 430

Lecture 4, Page 9

CMSC 430

Lecture 4, Page 11

How does this algorithm work? 1. impose an arbitrary order on the non-terminals 2. outer loop cycles through N T in order 3. inner loop ensures that a production expanding Ai has no non-terminal Aj with j < i 4. It forward substitutes those away 5. last step in the outer loop converts any direct recursion on Ai to right recursion using the simple transformation showed earlier 6. new non-terminals are added at the end of the order and only involve right recursion At the start of the ith outer loop iteration for all k < i, a production expanding Ak that has Al in its rhs, for l < k. At the end of the process (n < i), the grammar has no remaining left recursion.
CMSC 430 Lecture 4, Page 10

Recursive Descent Parsing Properties top-down parsing algorithm parser built on procedure calls procedures may be (mutually) recursive Algorithm write procedure for each non-terminal turn each production into clause insert call to procedure A() for non-terminal A to match(x) for terminal x start by invoking procedure for start symbol S Example A ::= a B c
CMSC 430

A() { match(a); B(); match(c); }

Recursive Descent Parsing Example S 1 2 3 A Helpers tok; // current token match(x) { if (tok != x) error(); tok = getToken(); } grammar ::= a A | b ::= S c Parser S() { if (tok == a) match(a); A(); else if (tok == b) match(b); else error(); } A() { S(); match(c); }

Left Factoring What if a grammar does not have the LL(1) property? Sometimes, we can transform a grammar to have this property. For each non-terminal A nd the longest prex common to two or more of its alternatives. if = , then replace all of the A productions A ::= 1 | 2 | | n | with A ::= L | L ::= 1 | 2 | | n where L is a new non-terminal. Repeat until no two alternatives for a single non-terminal have a common prex.

CMSC 430

Lecture 4, Page 13

CMSC 430

Lecture 4, Page 15

Predictive Parsing Basic idea For any two productions A ::= | , we would like a distinct way of choosing the correct production to expand. FIRST sets For some rhs G, dene FIRST() as the set of tokens that appear as the rst symbol in some string derived from . That is, x FIRST() i x for some . LL(1) property Whenever two productions A ::= and A ::= both appear in the grammar, we would like F IRST () F IRST () = This would allow the parser to make a correct choice with a lookahead of only one symbol! Pursuing this idea leads to predictive LL(1) parsers.
CMSC 430 Lecture 4, Page 14

Example Consider a right-recursive version of the expression grammar: 1 <goal> ::= <expr> 2 <expr> ::= <term> + <expr> 3 | <term> - <expr> 4 | <term> 5 <term> ::= <factor> * <term> 6 | <factor> / <term> 7 | <factor> 8 <factor> ::= number 9 | id To choose between productions 2, 3, & 4, the parser must see past the number or id and look at the +, -, *, or /.
FIRST(2)

FIRST(3) FIRST(4) =

This grammar fails the test. Note: This grammar is right-associative.

CMSC 430

Lecture 4, Page 16

Example There are two nonterminals that must be left factored: <expr> ::= <term> + <expr> | <term> - <expr> | <term> <term> ::= <factor> * <term> | <factor> / <term> | <factor> Applying the transformation gives us: <expr> ::= <term> <expr > <expr > ::= + <expr> | - <expr> | <term> ::= <factor> <term > <term > ::= * <term> | / <term> |

Example: Sentential form

<goal> x - 2 * y <expr> x - 2 * y <term> <expr > x - 2 * y <factor> <term > <expr > x - 2 * y <id> <term > <expr > x - 2 * y <id> <term > <expr > x - 2 * y <id> <expr > x - 2 <id> - <expr> x - 2 * y <id> - <expr> x - 2 * y <id> - <term> <expr > x - 2 * y <id> - <factor> <term > <expr > x - 2 * y <id> - <num> <term > <expr > x - 2 * y <id> - <num> <term > <expr > x - 2 * y <id> - <num> * <term> <expr > x -2 * y <id> - <num> * <term> <expr > x -2 * y <id> - <num> * <factor> <term > <expr > x -2 * y <id> - <num> * <id> <expr > x -2 * y <id> - <num> * <id> <term > <expr > x -2 * y <id> - <num> * <id> <expr > x -2 * y <id> - <num> * <id> x -2 * y

Input

The next symbol determined each choice correctly.

CMSC 430

Lecture 4, Page 17

CMSC 430

Lecture 4, Page 19

Example Substituting back into the grammar yields 1 2 3 4 5 6 7 8 9 10 11 <goal> ::= <expr> <expr> ::= <term> <expr > <expr > ::= + <expr> | - <expr> | <term> ::= <factor> <term > <term > ::= * <term> | / <term> | <factor> ::= number | id

Generality Question: By eliminating left recursion and left factoring, can we transform an arbitrary context free grammar to a form where it can be predictively parsed with a single token lookahead? Answer: Given a context free grammar that doesnt meet our conditions, it is undecidable whether an equivalent grammar exists that does meet our conditions.

Many context free languages do not have such a grammar. Now, selection requires only a single token lookahead. Note: This grammar is still right-associative. {an0bn | n 1} {an1b2n | n 1}

CMSC 430

Lecture 4, Page 18

CMSC 430

Lecture 4, Page 20

The FIRST set For a string of grammar symbols , dene FIRST() as the set of terminal symbols that begin strings derived from if , then FIRST()
FIRST()

The FIRST construction

contains the set of tokens valid in the rst position of

To build FIRST(X): 1. if X is a terminal, FIRST(X) is {X} 2. if X ::= , then FIRST(X) 3. if X ::= Y1Y2 Yk then put FIRST(Y1) in FIRST(X) 4. if X is a non-terminal and X ::= Y1Y2 Yk , then a FIRST(X) if a FIRST(Yi) and FIRST(Yj ) for all 1 j < i (If FIRST(Y1), then FIRST(Yi) is irrelevant, for 1 < i)

rule 1 goal expr expr term term factor num num id id + + * * / /

2 3 num,id num,id +, num,id *,/ num,id

FIRST

{num,id} {num,id} { ,+,-} {num,id} { ,*,/} {num,id} {num} {id} {+} {-} {*} {/}

CMSC 430

Lecture 4, Page 21

CMSC 430

Lecture 4, Page 23

Our example grammar 1 2 3 4 5 6 7 8 9 goal expr expr ::= expr ::= term ::= + expr | - expr | ::= factor ::= * term | / term | term expr

The FOLLOW set For a non-terminal A, dene FOLLOW(A) as the set of terminals that can appear immediately to the right of A in some sentential form Thus, a non-terminals FOLLOW set species the tokens that can legally appear after it A terminal symbol has no FOLLOW set

term term

To build FOLLOW(X): 1. place eof in FOLLOW( goal ) 2. if A ::= B, then put {FIRST() } in FOLLOW(B) 3. if A ::= B then put FOLLOW(A) in FOLLOW(B) 10 factor ::= num 11 | id 4. if A ::= B and FIRST(), then put FOLLOW(A) in FOLLOW(B)

CMSC 430

Lecture 4, Page 22

CMSC 430

Lecture 4, Page 24

The FOLLOW construction

LL(1) grammars Features input parsed from left to right leftmost derivation one token lookahead

rule 1 2 3 4 FOLLOW goal eof {eof} expr eof {eof} expr eof {eof} term +, eof {eof,+,-} term eof,+, {eof,+,-} factor *,/ eof,+,- {eof,+,-,*,/}

Denition A grammar G is LL(1) if and only if, for all non-terminals A, each distinct pair of productions A ::= and A ::= satisfy the condition FIRST() FIRST() = A grammar G is LL(1) if and only if for each set of productions A ::= 1 | 2 | | n 1. FIRST(1), FIRST(2), , FIRST(n) are all pairwise disjoint 2. if i , then FIRST(j )
FOLLOW(A)

= , for all 1 j n, i =j.

If G is free, condition 1 is sucient.

Using FIRST and FOLLOW To build a predicative recursive-descent parser: For each production A ::= and lookahead token expand A using production if token FIRST() if FIRST() expand A using production if token FOLLOW(A) all other tokens return error If multiple choices, the grammar is not LL(1) (predicative).

LL(1) grammars Provable facts about LL(1) grammars: no left recursive grammar is LL(1) no ambiguous grammar is LL(1) LL(1) parsers operate in linear time an free grammar where each alternative expansion for A begins with a distinct terminal is a simple LL(1) grammar Not all grammars are LL(1) S ::= aS | a is not LL(1) FIRST(aS) = FIRST(a) = {a} S ::= aS S ::= aS | accepts the same language and is LL(1)

goal expr expr term term factor

id num + * / eof ge ge e te e te e +e e -e e t ft t ft t t t *t t /t t f id f num

CMSC 430

CMSC 430

Lecture 4, Page 28

LL grammars
LL(1)

grammars

may need to rewrite grammar (left recursion, left factoring) resulting grammar larger, less maintainable
LL(k)

grammars

k-token lookahead, more powerful than LL(1) grammars example: S ::= ac | abc is LL(2) Not all grammars are LL(k) example: S ::= aibj where i j

equivalent to dangling else problem problem - must choose production after k tokens of lookahead Bottom-up parsers avoid this problem
CMSC 430 Lecture 4, Page 29