Writing Compilers in Python

(with PLY)

Dave Beazley
http://www.dabeaz.com

October 12, 2006

 Overview
• Crash course on compilers
• Lex/yacc
• An introduction to PLY
• Blood and guts (Rated R)
• Various PLY features (more gore)
• Examples
 Disclaimer

• Compilers is an advanced topic

• Please stop me for questions!
 Motivation

• Writing a compiler is hard

• Writing Python code seems to be easy
• So why not write a compiler in Python?
 Compilers 101
# Some program
print “Hello World”
compiler
b = 3 + 4 * 5
for c in range(10): ??? Profit!
print c

• What is a compiler?
• A program that processes other programs
• Typically implements a programming lang.
• Examples:
• gcc, javac, SWIG, Doxygen, Python
 Compiler Design
in lexing parsing typecheck codegen out

• Compiler broken into stages

• Lexing/parsing related to reading input
• Type checking is error checking/validation
• Code generation does something
 Example
• Parse and generate code for the following:
b = 40 + 20*(2+3)/37.5
 Lexing
• Splits input text into tokens
• Makes sure the input uses right alphabet
b = 40 + 20*(2+3)/37.5

NAME = NUM + NUM * ( NUM + NUM ) / FLOAT

• Detects illegal symbols

b = 40 * $5

Illegal Character
 Parsing
• Makes sure input is structurally correct
b = 40 + 20*(2+3)/37.5

• Builds program structure (e.g., parse tree)

NAME = NUM + NUM * ( NUM + NUM ) / FLOAT
=
NAME +
NUM /
* FLOAT
NUM +

NUM NUM
 Parsing
• Detects syntax errors
b = 40 + “hello” (Syntax OK)
b = 3 * 4 7 / (Syntax error)

• If a program parses, it is at least well-formed

• Still don’t know if program is correct
b = 40 + “hello” (???)
 Type checking
• Enforces underlying semantics
b = 40 + 20*(2+3)/37.5 (OK)
c = 3 + “hello” (TYPE ERROR)
d[4.5] = 4 (BAD INDEX)

• Example: + operator
+
LHS RHS

1. LHS and RHS must be the same type

2. If different types, must be convertible to same type
 Code Generation
• Processing the parse tree in some way
• Usually a traversal of the parse tree
b = 40 + 20*(2+3)/37.5 +
40 /
* 37.5
20 +

2 3
 Code Generation
• Processing the parse tree in some way
• Usually a traversal of the parse tree
b = 40 + 20*(2+3)/37.5 +
40 /
LOAD R1, 40
* 37.5
20 +

2 3
 Code Generation
• Processing the parse tree in some way
• Usually a traversal of the parse tree
b = 40 + 20*(2+3)/37.5 +
40 /
LOAD R1, 40
LOAD R2, 20 * 37.5
20 +

2 3
 Code Generation
• Processing the parse tree in some way
• Usually a traversal of the parse tree
b = 40 + 20*(2+3)/37.5 +
40 /
LOAD R1, 40
LOAD R2, 20 * 37.5
LOAD R3, 2 20 +

2 3
 Code Generation
• Processing the parse tree in some way
• Usually a traversal of the parse tree
b = 40 + 20*(2+3)/37.5 +
40 /
LOAD R1, 40
LOAD R2, 20 * 37.5
LOAD R3, 2 20 +
LOAD R4, 3
2 3
 Code Generation
• Processing the parse tree in some way
• Usually a traversal of the parse tree
b = 40 + 20*(2+3)/37.5 +
40 /
LOAD R1, 40
LOAD R2, 20 * 37.5
LOAD R3, 2 20 +
LOAD R4, 3
ADD R3, R4, R3 ; R3 = (2+3) 2 3
 Code Generation
• Processing the parse tree in some way
• Usually a traversal of the parse tree
b = 40 + 20*(2+3)/37.5 +
40 /
LOAD R1, 40
LOAD R2, 20 * 37.5
LOAD R3, 2 20 +
LOAD R4, 3
ADD R3, R4, R3 ; R3 = (2+3) 2 3
MUL R2, R3, R2 ; R2 = 20*(2+3)
 Code Generation
• Processing the parse tree in some way
• Usually a traversal of the parse tree
b = 40 + 20*(2+3)/37.5 +
40 /
LOAD R1, 40
LOAD R2, 20 * 37.5
LOAD R3, 2 20 +
LOAD R4, 3
ADD R3, R4, R3 ; R3 = (2+3) 2 3
MUL R2, R3, R2 ; R2 = 20*(2+3)
LOAD R3, 37.5
 Code Generation
• Processing the parse tree in some way
• Usually a traversal of the parse tree
b = 40 + 20*(2+3)/37.5 +
40 /
LOAD R1, 40
LOAD R2, 20 * 37.5
LOAD R3, 2 20 +
LOAD R4, 3
ADD R3, R4, R3 ; R3 = (2+3) 2 3
MUL R2, R3, R2 ; R2 = 20*(2+3)
LOAD R3, 37.5
DIV R2, R3, R2 ; R2 = 20*(2+3)/37.5
 Code Generation
• Processing the parse tree in some way
• Usually a traversal of the parse tree
b = 40 + 20*(2+3)/37.5 +
40 /
LOAD R1, 40
LOAD R2, 20 * 37.5
LOAD R3, 2 20 +
LOAD R4, 3
ADD R3, R4, R3 ; R3 = (2+3) 2 3
MUL R2, R3, R2 ; R2 = 20*(2+3)
LOAD R3, 37.5
DIV R2, R3, R2 ; R2 = 20*(2+3)/37.5
ADD R1, R2, R1 ; R1 = 40+20*(2+3)/37.5
 Code Generation
• Processing the parse tree in some way
• Usually a traversal of the parse tree
b = 40 + 20*(2+3)/37.5 +
40 /
LOAD R1, 40
LOAD R2, 20 * 37.5
LOAD R3, 2 20 +
LOAD R4, 3
ADD R3, R4, R3 ; R3 = (2+3) 2 3
MUL R2, R3, R2 ; R2 = 20*(2+3)
LOAD R3, 37.5
DIV R2, R3, R2 ; R2 = 20*(2+3)/37.5
ADD R1, R2, R1 ; R1 = 40+20*(2+3)/37.5
STORE R1, “b”
 Comments

• Concept is mostly straightforward

• Omitting many horrible details
• More covered in a compilers course.
 Parsing (revisited)

• Parsing is probably most annoying problem

• Not a matter of simple text processing
• Not obvious
• Not fun
 Lex & Yacc
• Programming tools for writing parsers
• Lex - Lexical analysis (tokenizing)
• Yacc - Yet Another Compiler Compiler (parsing)
• History:
- Yacc : ~1973. Stephen Johnson (AT&T)
- Lex : ~1974. Eric Schmidt and Mike Lesk (AT&T)
- Both are standard Unix utilities
- GNU equivalents: flex and bison
- Part of IEEE POSIX 1003.2 standard
- Implementations available for most programming languages
Lex/Yacc Big Picture
scanner.l parser.y
token grammar
specification specification
Lex/Yacc Big Picture
scanner.l parser.y
token
/* scanner.l */ grammar
specification
%{ specification
#include “header.h”
int lineno = 1;
%}
%%
[ \t]* ; /* Ignore whitespace */
\n { lineno++; }
[0-9]+ { yylval.val = atoi(yytext);
return NUMBER; }
[a-zA-Z_][a-zA-Z0-9_]* { yylval.name = strdup(yytext);
return ID; }
\+ { return PLUS; }
- { return MINUS; }
\* { return TIMES; }
\/ { return DIVIDE; }
= { return EQUALS; }
%%
Lex/Yacc Big Picture
scanner.l parser.y
token
/* parser.y */ grammar
specification
%{ specification
#include “header.h”
%}
%union {
char *name;
int val;
}
%token PLUS MINUS TIMES DIVIDE EQUALS
%token<name> ID;
%token<val> NUMBER;
%%
start : ID EQUALS expr;
expr : expr PLUS term
| expr MINUS term
| term
;
...
Lex/Yacc Big Picture
scanner.l parser.y
token grammar
specification specification
Lex/Yacc Big Picture
scanner.l parser.y
token grammar
specification specification

lex

scanner.c
Lex/Yacc Big Picture
scanner.l parser.y
token grammar
specification specification

lex yacc

scanner.c parser.c
Lex/Yacc Big Picture
scanner.l parser.y
token grammar
specification specification

lex yacc

scanner.c parser.c

typecheck.c codegen.c otherstuff.c

Lex/Yacc Big Picture
scanner.l parser.y
token grammar
specification specification

lex yacc

scanner.c parser.c

typecheck.c codegen.c otherstuff.c

mycompiler
 Lex/Yacc Comments

• Code generators
• Create a parser from a specification
• Classic versions create C code.
• Variants target other languages
 PLY
• Python Lex-Yacc
• 100% Python version of lex/yacc toolset
• History:
- Late 90’s. “Write don’t you rewrite SWIG in Python?”
- 2000 : “No! Now stop bugging me about it!”
- 2001 : Dave teaches a compilers course at UofC. An experiment.
Students write a compiler in Python.
- 2001 : PLY-1.0 developed and released.
- 2002 - 2005 : Occasional maintenance and bug fixes.
- 2006 : Major update to PLY-2.x (in progress).

• This is the first talk about it

 PLY Overview
• Provide same functionality as lex/yacc
• Identical parsing algorithm (LALR(1))

• Extensive error checking.

• Comparable debugging features (sic)

• Keep it simple (ha!)

• Make use of Python features

 PLY Package
• PLY consists of two Python modules
ply.lex
ply.yacc

• You simply import the modules to use them

• However, PLY is not a code generator
• This is where it gets interesting
 lex.py example
import ply.lex as lex
tokens = [ ‘NAME’,’NUMBER’,’PLUS’,’MINUS’,’TIMES’,
’DIVIDE’, EQUALS’ ]
t_ignore = ‘ \t’
t_PLUS = r’\+’
t_MINUS = r’-’
t_TIMES = r’\*’
t_DIVIDE = r’/’
t_EQUALS = r’=’
t_NAME = r’[a-zA-Z_][a-zA-Z0-9_]*’

def t_NUMBER(t):
r’\d+’
t.value = int(t.value)
return t

lex.lex() # Build the lexer

 lex.py specification
• Tokens denoted by t_TOKEN declarations
• Tokens are defined by regular expressions
t_NAME = r’[a-zA-Z_][a-zA-Z0-9_]*’

def t_NUMBER(t):
r’\d+’
t.value = int(t.value)
return t

• May be a simple variable or a function

• For functions, regex is in docstring.
 lex.py construction
• lex() function is used to build the lexer
import ply.lex as lex
... token specifications ...

lex.lex() # Build the lexer

• Uses introspection to read spec

• Token information taken out of calling module
• Big difference between Unix lex and PLY
 lex.py construction
• lex() function is used to build the lexer
import ply.lex as lex
... token specifications ...

lex.lex() # Build the lexer

• Uses introspection to read spec

Sick introspection hack
try: raise RuntimeError

• Token information taken out of calling module

except RuntimeError:
e,b,t = sys.exc_info()
f = t.tb_frame

• Big difference between Unix lex and PLY

f = f.f_back
mdict = f.f_globals
 lex.py Validation
• lex.lex() performs extensive error checking
• Bad tokens, duplicate tokens, malformed
functions, etc.
t_NAME = r’[a-zA-Z_][a-zA-Z0-9_]*’
...
t_NAME = r’[a-zA-Z][a-zA-Z0-9]*’

calc.py:20 Rule t_NAME redefined.

Previously defined on line 14.

• Goal: informative debugging messages

 lex.py use
• Two functions: input(), token()
import ply.lex as lex
...
lex.lex() # Build the lexer
...
data = “x = 3*4+5-6”
lex.input(data) # Feed some text
while 1:
tok = lex.token() # Get next token
if not tok: break
print tok

• Call token() repeatedly to fetch tokens

Example
 yacc.py preliminaries
• yacc.py is a module for creating a parser
• Assumes you have defined a BNF grammar
assign : NAME EQUALS expr
expr : expr PLUS term
| expr MINUS term
| term
term : term TIMES factor
| term DIVIDE factor
| factor
factor : NUMBER
 yacc.py example
import ply.yacc as yacc
import mylexer # Import lexer information
tokens = mylexer.tokens # Need token list

def p_assign(p):
‘’’assign : NAME EQUALS expr’’’

def p_expr(p):
‘’’expr : expr PLUS term
| expr MINUS term
| term’’’
def p_term(p):
‘’’term : term TIMES factor
| term DIVIDE factor
| factor’’’
def p_factor(p):
‘’’factor : NUMBER’’’

yacc.yacc() # Build the parser

 yacc.py rules
• All rules defined by p_funcname(p) funcs
• Grammar specified in docstrings
def p_expr(p):
‘’’expr : expr PLUS term
| expr MINUS term
| term’’’

• Rules may be split apart or combined

def p_expr_plus(p):
‘expr : expr PLUS term’
def p_expr_minus(p):
‘expr : expr MINUS term’
def p_expr_term(p):
‘expr : term’
 yacc.py construction
• yacc() function builds the parser
import ply.yacc as yacc
... rule specifications ...

yacc.yacc() # Build the parser

• Uses introspection (as before)

• Generates parsing tables and diagnostics
% python myparser.py
yacc: Warning. no p_error() function is defined
yacc: Generating LALR parsing table...
 yacc.py performance

• yacc.yacc() is expensive (several seconds)

• Parsing tables written to file parsetab.py
• Only regenerated when grammar changes
• Avoids performance hit on repeated use
 yacc.py validation

• yacc.yacc() also performs validation

• Duplicate rules, malformed grammars,
infinite recursion, undefined symbols, bad
arguments, etc.
• Provides the same error messages provided
by Unix yacc.
 yacc.py parsing
• yacc.parse() function
yacc.yacc() # Build the parser
...
data = “x = 3*4+5*6”
yacc.parse(data) # Parse some text

• This implicitly feeds data into lexer

Example
 A peek inside

• PLY is based on LR-parsing. LALR(1)

• AKA: Shift-reduce parsing
• Widely used.
• Table driven.
• Speed is independent of grammar size
 LR Parsing
• Three basic components:
• A stack of grammar symbols and values.

• Two operators: shift, reduce

• An underlying state machine.

• Example
 LR Example: Step 1

stack input
X = 3 + 4 * 5 $end

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 1

stack input
X = 3 + 4 * 5 $end

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 1

stack input
NAME
= 3 + 4 * 5 $end
‘X’

Action: shift

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 1

stack Symbol type input

NAME
= 3 + 4 * 5 $end
‘X’
Symbol value
Action: shift

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 2

stack input
NAME
= 3 + 4 * 5 $end
‘X’

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 2

stack input
NAME
= 3 + 4 * 5 $end
‘X’

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 2

stack input
NAME EQUALS
3 + 4 * 5 $end
‘X’ ‘=’

Action: shift

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 3

stack input
NAME EQUALS
3 + 4 * 5 $end
‘X’ ‘=’

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 3

stack input
NAME EQUALS
3 + 4 * 5 $end
‘X’ ‘=’

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 3

stack input
NAME EQUALS NUMBER
+ 4 * 5 $end
‘X’ ‘=’ 3

Action: shift

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 3

stack input
NAME EQUALS NUMBER
+ 4 * 5 $end
‘X’ ‘=’ 3
def t_NUMBER(t):
Action: shift r’\d+’
t.value = int(t.value)
return t
Grammar PLY Rules
(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 4

stack input
NAME EQUALS NUMBER
+ 4 * 5 $end
‘X’ ‘=’ 3

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 4

stack input
NAME EQUALS NUMBER
+ 4 * 5 $end
‘X’ ‘=’ 3

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 4

stack input
NAME EQUALS NUMBER
+ 4 * 5 $end
‘X’ ‘=’ 3

Action: reduce using rule 8

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 4

stack input
NAME EQUALS factor
+ 4 * 5 $end
‘X’ ‘=’ None

Action: reduce using rule 8

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 4

stack input
NAME EQUALS factor
+ 4 * 5 $end
‘X’ ‘=’ None
This is None because
Action: reduce using rule 8
p_factor() didn’t do anything.
More later.

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) def p_factor(p):
term : term TIMES factor -> p_term(p)
(6) ‘’’factor : NUMBER’’’
| term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 5

stack input
NAME EQUALS factor
+ 4 * 5 $end
‘X’ ‘=’ None

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 5

stack input
NAME EQUALS factor
+ 4 * 5 $end
‘X’ ‘=’ None

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 5

stack input
NAME EQUALS factor
+ 4 * 5 $end
‘X’ ‘=’ None

Action: reduce using rule 7

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 5

stack input
NAME EQUALS term
+ 4 * 5 $end
‘X’ ‘=’ None

Action: reduce using rule 7

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 6

stack input
NAME EQUALS term
+ 4 * 5 $end
‘X’ ‘=’ None

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 6

stack input
NAME EQUALS term
+ 4 * 5 $end
‘X’ ‘=’ None

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 6

stack input
NAME EQUALS term
+ 4 * 5 $end
‘X’ ‘=’ None

Action: reduce using rule 4

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 6

stack input
NAME EQUALS expr
+ 4 * 5 $end
‘X’ ‘=’ None

Action: reduce using rule 4

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 7

stack input
NAME EQUALS expr
+ 4 * 5 $end
‘X” ‘=’ None

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 7

stack input
NAME EQUALS expr
+ 4 * 5 $end
‘X’ ‘=’ None

Action: ????

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 7

stack input
NAME EQUALS expr
+ 4 * 5 $end
‘X’ ‘=’ None

Action: shift

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 7

stack input
NAME EQUALS expr PLUS
4 * 5 $end
‘X’ ‘=’ None ‘+’

Action: shift

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 8

stack input
NAME EQUALS expr PLUS NUMBER
4 * 5 $end
‘X’ ‘=’ None ‘+’

Action: shift

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 9

stack input
NAME EQUALS expr PLUS NUMBER
* 5 $end
‘X’ ‘=’ None ‘+’ 4

Action: reduce using rule 8

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 9

stack input
NAME EQUALS expr PLUS factor
* 5 $end
‘X’ ‘=’ None ‘+’ None

Action: reduce using rule 7

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 10

stack input
NAME EQUALS expr PLUS term TIMES
* 5 $end
‘X’ ‘=’ None ‘+’ None

Action: shift

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 11

stack input
NAME EQUALS expr PLUS term TIMES NUMBER
5 $end
‘X’ ‘=’ None ‘+’ None ‘*’

Action: shift

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 12

stack input
NAME EQUALS expr PLUS term TIMES NUMBER
$end
‘X’ ‘=’ None ‘+’ None ‘*’ 5

Action: reduce using rule 8

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 13

stack input
NAME EQUALS expr PLUS term TIMES factor
$end
‘X’ ‘=’ None ‘+’ None ‘*’ None

Action: reduce using rule 5

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 14

stack input
NAME EQUALS expr PLUS term
$end
‘X’ ‘=’ None ‘+’ None

Action: reduce using rule 2

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 15

stack input
NAME EQUALS expr
$end
‘X’ ‘=’ None

Action: reduce using rule 1

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 16

stack input
assign
$end
None

Action: Done.

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 Yacc Rule Execution
• Rules are executed during reduction
def p_term_mul(p):
‘term : term TIMES factor’

• Parameter p refers to values on stack

stack: NAME EQUALS expr PLUS term TIMES factor
p[1] p[2] p[3]

reduce term : term TIMES factor

p[0]

stack: NAME EQUALS expr PLUS term

 Example: Calculator
def p_assign(p):
‘’’assign : NAME EQUALS expr’’’
print “Assigning”, p[1],”value”,p[3]

def p_expr_plus(p):
‘’’expr : expr PLUS term’’’
p[0] = p[1] + p[3]

def p_term_mul(p):
‘’’term : term TIMES factor’’’
p[0] = p[1] * p[3]

def p_factor(p):
‘’’factor : NUMBER’’’
p[0] = p[1]
 LR Example: Step 4

stack input
NAME EQUALS NUMBER
+ 4 * 5 $end
‘X’ ‘=’ 3

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 4

stack input
NAME EQUALS NUMBER
+ 4 * 5 $end
‘X’ ‘=’ 3

Action:

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 4

stack input
NAME EQUALS NUMBER
+ 4 * 5 $end
‘X’ ‘=’ 3

Action: reduce using rule 8

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 4

stack input
NAME EQUALS factor
+ 4 * 5 $end
‘X’ ‘=’ 3

Action: reduce using rule 8

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
(5) term : term TIMES factor -> p_term(p)
(6) | term DIVIDE factor
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 LR Example: Step 4

stack input
NAME EQUALS factor
+ 4 * 5 $end
‘X’ ‘=’ 3

This retains its value because

Action: reduce using rule 8
of assignment in p_factor().

Grammar PLY Rules

(1) assign : NAME EQUALS expr -> p_assign(p)
(2) expr : expr PLUS term -> p_expr(p)
(3) | expr MINUS term
(4) | term
def p_factor(p):
(5) term ‘’’factor
: term TIMES factor
: NUMBER’’’ -> p_term(p)
(6) | term
p[0] DIVIDE factor
= p[1]
(7) | factor
(8) factor : NUMBER -> p_factor(p)
 Example: Parse Tree
def p_assign(p):
‘’’assign : NAME EQUALS expr’’’
p[0] = (‘ASSIGN’,p[1],p[3])

def p_expr_plus(p):
‘’’expr : expr PLUS term’’’
p[0] = (‘+’,p[1],p[3])

def p_term_mul(p):
‘’’term : term TIMES factor’’’
p[0] = (‘*’,p[1],p[3])

def p_factor(p):
‘’’factor : NUMBER’’’
p[0] = (‘NUM’,p[1])
 Ambiguous Grammars
def p_assign(p):
‘’’assign : NAME EQUALS expr’’’

def p_expr(p):
‘’’expr : expr PLUS expr
| expr MINUS expr
| expr TIMES expr
| expr DIVIDE expr
| NUMBER’’’

3 + 4 * 5
? ?
+ *
3 * + 5
4 5 3 4
Ambiguous Grammars
• Multiple possible parse trees
• Is reported as a “shift/reduce conflict”
yacc: Generating LALR parsing table...
yacc: 16 shift/reduce conflicts

• May also get “reduce/reduce conflict”

• Probably most mysterious aspect of yacc
 Shift/Reduce Conflict Explained

stack input
NAME EQUALS expr PLUS expr * 5 $end

Grammar Possible Actions:

(1) assign : NAME EQUALS expr
(2) expr : expr PLUS expr
(3) | expr MINUS expr
(4) | expr TIMES expr
(5) | expr DIVIDE expr
(6) | NUMBER
 Shift/Reduce Conflict Explained

stack input
NAME EQUALS expr PLUS expr * 5 $end

Grammar Possible Actions:

(1) assign : NAME EQUALS expr
(2) expr : expr PLUS expr reduce using rule 2
(3) | expr MINUS expr
(4) | expr TIMES expr
(5) | expr DIVIDE expr
(6) | NUMBER
 Shift/Reduce Conflict Explained

stack input
NAME EQUALS expr PLUS expr * 5 $end

NAME EQUALS expr

NAME EQUALS expr TIMES
NAME EQUALS expr TIMES NUMBER
NAME EQUALS expr TIMES expr
NAME EQUALS expr

Grammar Possible Actions:

(1) assign : NAME EQUALS expr
(2) expr : expr PLUS expr reduce using rule 2
(3) | expr MINUS expr
(4) | expr TIMES expr
(5) | expr DIVIDE expr
(6) | NUMBER
 Shift/Reduce Conflict Explained

stack input
NAME EQUALS expr PLUS expr * 5 $end

NAME EQUALS expr

NAME EQUALS expr TIMES
NAME EQUALS expr TIMES NUMBER
NAME EQUALS expr TIMES expr
NAME EQUALS expr

Grammar Possible Actions:

(1) assign : NAME EQUALS expr
(2) expr : expr PLUS expr reduce using rule 2
(3)
(4)
| expr MINUS expr
| expr TIMES expr
shift TIMES
(5) | expr DIVIDE expr
(6) | NUMBER
 Shift/Reduce Conflict Explained

stack input
NAME EQUALS expr PLUS expr * 5 $end

NAME EQUALS expr NAME EQUALS expr PLUS expr TIMES

NAME EQUALS expr TIMES NAME EQUALS expr PLUS expr TIMES NUMBER
NAME EQUALS expr TIMES NUMBER NAME EQUALS expr PLUS expr TIMES expr
NAME EQUALS expr TIMES expr NAME EQUALS expr PLUS expr
NAME EQUALS expr NAME EQUALS expr

Grammar Possible Actions:

(1) assign : NAME EQUALS expr
(2) expr : expr PLUS expr reduce using rule 2
(3)
(4)
| expr MINUS expr
| expr TIMES expr
shift TIMES
(5) | expr DIVIDE expr
(6) | NUMBER
Shift/reduce resolution
• Default action is to always shift
• Can sometimes control with precedence
precedence = (
(‘left’,’PLUS’,’MINUS’),
(‘left’,’TIMES’,’DIVIDE’),
)
def p_assign(p):
‘’’assign : NAME EQUALS expr’’’

def p_expr(p):
‘’’expr : expr PLUS expr
| expr MINUS expr
| expr TIMES expr
| expr DIVIDE expr
| NUMBER’’’
Error handling/recovery
• Syntax errors first fed through p_error()
def p_error(p):
print “Syntax error”

• Then an ‘error’ symbol is shifted onto stack

• Stack is unwound until error is consumed
 Error recovery

stack input
NAME EQUALS expr PLUS expr
5 $end
‘X’ ‘=’ 3 ‘+’ 4

Grammar
(1) assign : NAME EQUALS expr
(2) | NAME EQUALS error
(3) expr : expr PLUS expr
(4) | expr MINUS expr
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Error recovery

stack input
NAME EQUALS expr PLUS expr
5 $end
‘X’ ‘=’ 3 ‘+’ 4
syntax error

Grammar
(1) assign : NAME EQUALS expr
(2) | NAME EQUALS error
(3) expr : expr PLUS expr
(4) | expr MINUS expr
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Error recovery

stack input
NAME EQUALS expr PLUS expr
5 $end
‘X’ ‘=’ 3 ‘+’ 4
syntax error

def p_error(p):
print “Syntax error”
Grammar
(1) assign : NAME EQUALS expr
(2) | NAME EQUALS error
(3) expr : expr PLUS expr
(4) | expr MINUS expr
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Error recovery

stack input
NAME EQUALS expr PLUS expr error
$end
‘X’ ‘=’ 3 ‘+’ 4

Grammar
(1) assign : NAME EQUALS expr
(2) | NAME EQUALS error
(3) expr : expr PLUS expr
(4) | expr MINUS expr
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Error recovery

stack input
NAME EQUALS expr PLUS error
$end
‘X’ ‘=’ 3 ‘+’

Grammar
(1) assign : NAME EQUALS expr
(2) | NAME EQUALS error
(3) expr : expr PLUS expr
(4) | expr MINUS expr
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Error recovery

stack input
NAME EQUALS expr error
$end
‘X’ ‘=’ 3

Grammar
(1) assign : NAME EQUALS expr
(2) | NAME EQUALS error
(3) expr : expr PLUS expr
(4) | expr MINUS expr
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Error recovery

stack input
NAME EQUALS error
$end
‘X’ ‘=’

Grammar
(1) assign : NAME EQUALS expr
(2) | NAME EQUALS error
(3) expr : expr PLUS expr
(4) | expr MINUS expr
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Error recovery

stack input
NAME EQUALS error
$end
‘X’ ‘=’

Grammar
(1) assign : NAME EQUALS expr def p_assign_err(p):
(2) | NAME EQUALS error ‘assign : NAME EQUALS error’
(3) expr : expr PLUS expr print “Bad assignment”
(4) | expr MINUS expr
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Error recovery

stack input
assign
$end

Grammar
(1) assign : NAME EQUALS expr def p_assign_err(p):
(2) | NAME EQUALS error ‘assign : NAME EQUALS error’
(3) expr : expr PLUS expr print “Bad assignment”
(4) | expr MINUS expr
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Debugging Output

• PLY creates a file parser.out

• Contains detailed debugging information
• Reading it involves voodoo and magic
• Useful if trying to track down conflicts
 Debugging Output
Grammar state 10

Rule 1 statement -> NAME = expression (1) statement -> NAME = expression .
Rule 2 statement -> expression (3) expression -> expression . + expression
Rule 3 expression -> expression + expression (4) expression -> expression . - expression
Rule 4 expression -> expression - expression (5) expression -> expression . * expression
Rule 5 expression -> expression * expression (6) expression -> expression . / expression
Rule 6 expression -> expression / expression
Rule 7 expression -> NUMBER $end reduce using rule 1 (statement -> NAME = expression .)
+ shift and go to state 7
Terminals, with rules where they appear - shift and go to state 6
* shift and go to state 8
* : 5 / shift and go to state 9
+ : 3
- : 4
/ : 6
= : 1 state 11
NAME : 1
NUMBER : 7 (4) expression -> expression - expression .
error : (3) expression -> expression . + expression
(4) expression -> expression . - expression
Nonterminals, with rules where they appear (5) expression -> expression . * expression
(6) expression -> expression . / expression
expression : 1 2 3 3 4 4 5 5 6 6
statement : 0 ! shift/reduce conflict for + resolved as shift.
! shift/reduce conflict for - resolved as shift.
Parsing method: LALR ! shift/reduce conflict for * resolved as shift.
! shift/reduce conflict for / resolved as shift.
state 0 $end reduce using rule 4 (expression -> expression - expression .)
+ shift and go to state 7
(0) S' -> . statement - shift and go to state 6
(1) statement -> . NAME = expression * shift and go to state 8
(2) statement -> . expression / shift and go to state 9
(3) expression -> . expression + expression
(4) expression -> . expression - expression ! + [ reduce using rule 4 (expression -> expression - expression .) ]
(5) expression -> . expression * expression ! - [ reduce using rule 4 (expression -> expression - expression .) ]
(6) expression -> . expression / expression ! * [ reduce using rule 4 (expression -> expression - expression .) ]
(7) expression -> . NUMBER ! / [ reduce using rule 4 (expression -> expression - expression .) ]

NAME shift and go to state 1

NUMBER shift and go to state 2

expression shift and go to state 4

statement shift and go to state 3

state 1

(1) statement -> NAME . = expression

= shift and go to state 5

 Debugging Output
...Grammar state 10
state 11
Rule 1 statement -> NAME = expression (1) statement -> NAME = expression .
Rule 2 statement -> expression (3) expression -> expression . + expression
Rule 3 expression -> expression + expression (4) expression -> expression . - expression
(4) expression -> expression
Rule 4 expression -> expression - expression - expression . (5) expression -> expression . * expression
Rule 5 expression -> expression * expression (6) expression -> expression . / expression
(3) expression -> expression
Rule 6 expression -> expression / expression . + expression
Rule 7 expression -> NUMBER $end reduce using rule 1 (statement -> NAME = expression .)
(4) expression -> expression . - expression + shift and go to state 7
Terminals, with rules where they appear - shift and go to state 6
(5) expression -> expression . * expression * shift and go to state 8
* : 5 / shift and go to state 9
+ (6) expression : 3 -> expression . / expression
- : 4
/ : 6
= : 1 state 11
! shift/reduce conflict for + resolved as shift.
NAME : 1
NUMBER : 7 (4) expression -> expression - expression .
! shift/reduce conflict for - resolved as shift.
error : (3) expression -> expression . + expression
(4) expression -> expression . - expression
!Nonterminals,
shift/reducewith rules whereconflict
they appear for * resolved as shift.
(5) expression -> expression . * expression
(6) expression -> expression . / expression
!expression
shift/reduce : 1 2 3 conflict
3 4 4 5 5 6 6 for / resolved as shift.
statement : 0 ! shift/reduce conflict for + resolved as shift.
$end reduce using rule 4 (expression ->conflict
! shift/reduce expression - shift.
for - resolved as expression .)
Parsing method: LALR ! shift/reduce conflict for * resolved as shift.
+ shift and go to state 7 ! shift/reduce conflict for / resolved as shift.
state 0 $end reduce using rule 4 (expression -> expression - expression .)
- shift and go to state 6 + shift and go to state 7
(0) S' -> . statement - shift and go to state 6
*(1) statement -> . NAME = expression
shift and go to state 8 * shift and go to state 8
(2) statement -> . expression / shift and go to state 9
/ (3) expression -> . expression shift and go to state 9
+ expression
(4) expression -> . expression - expression ! + [ reduce using rule 4 (expression -> expression - expression .) ]
(5) expression -> . expression * expression ! - [ reduce using rule 4 (expression -> expression - expression .) ]
(6) expression -> . expression / expression ! * [ reduce using rule 4 (expression -> expression - expression .) ]
! + (7) expression -> . NUMBER [ reduce using rule 4 (expression
! / -> [ expression
reduce using rule 4 - expression
(expression -> expression - .) ]
expression .) ]

! -NAME shift and go to[state

reduce
1 using rule 4 (expression -> expression - expression .) ]
NUMBER shift and go to state 2
! * [ reduce using rule 4 (expression -> expression - expression .) ]
! /expression [shift
reduce using
and go to state 4 rule 4 (expression -> expression - expression .) ]
statement shift and go to state 3
...
state 1

(1) statement -> NAME . = expression

= shift and go to state 5

 Advanced PLY

• PLY supports more advanced yacc features

• Empty productions
• Error handling/recovery
• Inherited attributes
• Embedded actions
 Advanced PLY (cont)

• Some Python specific features

• Lexers/parsers can be defined as classes
• Support for multiple lexers and parsers
• Support for optimized mode (-O)
 Class Example
import ply.yacc as yacc

class MyParser:
def p_assign(self,p):
‘’’assign : NAME EQUALS expr’’’
def p_expr(self,p):
‘’’expr : expr PLUS term
| expr MINUS term
| term’’’
def p_term(self,p):
‘’’term : term TIMES factor
| term DIVIDE factor
| factor’’’
def p_factor(self,p):
‘’’factor : NUMBER’’’
def build(self):
self.parser = yacc.yacc(object=self)
 Summary

• This has been a quick tour of PLY/yacc

• Have skipped a lot of subtle details.
 Why use PLY?

• Standard lex/yacc well known and used

• Suitable for large grammars
• Decent performance
• Very extensive error checking/validation
 PLY Usage
• Thousands of downloads over five years
• Some applications (that I know of)
• Teaching compilers
• numbler.com (Carl Shimer)
• Parsing Ada source code.
• Parsing molecule descriptions
• Reading configuration files
 Resources
• PLY homepage
http://www.dabeaz.com/ply

• Mailing list/group
http://groups.google.com/group/ply-hack
 Reduce/Reduce Conflict Explained

stack input
NAME EQUALS NUMBER $end

Grammar Possible Actions:

(1) assign : NAME EQUALS expr
(2) | NAME EQUALS NUMBER
(3) expr : expr PLUS expr
(4) | expr MINUS expr
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Reduce/Reduce Conflict Explained

stack input
NAME EQUALS NUMBER $end

Grammar Possible Actions:

(1) assign : NAME EQUALS expr
(2) | NAME EQUALS NUMBER reduce using rule 2
(3) expr : expr PLUS expr
(4) | expr MINUS expr
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Reduce/Reduce Conflict Explained

stack input
assign $end

Grammar Possible Actions:

(1) assign : NAME EQUALS expr
(2) | NAME EQUALS NUMBER reduce using rule 2
(3) expr : expr PLUS expr
(4) | expr MINUS expr
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Reduce/Reduce Conflict Explained

stack input
assign $end
NAME EQUALS NUMBER

Grammar Possible Actions:

(1) assign : NAME EQUALS expr
(2) | NAME EQUALS NUMBER reduce using rule 2
(3) expr
(4)
: expr PLUS expr
| expr MINUS expr
reduce using rule 7
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER
 Reduce/Reduce Conflict Explained

stack input
assign $end
NAME EQUALS expr

Grammar Possible Actions:

(1) assign : NAME EQUALS expr
(2) | NAME EQUALS NUMBER reduce using rule 2
(3) expr
(4)
: expr PLUS expr
| expr MINUS expr
reduce using rule 7
(5) | expr TIMES expr
(6) | expr DIVIDE expr
(7) | NUMBER