Language Translation

Principles
Part 1: Language Specification
 Attributes of a language
• Syntax: rules describing use of language
tokens
• Semantics: logical meaning of
combinations of tokens
• In a programming language, “tokens”
include identifiers, keywords, and
punctuation
 Linguistic correctness
• A syntactically correct program is one in
which the tokens are arranged so that the
code can be successfully translated into a
lower-level language
• A semantically correct program is one
that produces correct results
 Language translation tools
• Parser: scans source code, compares with
established syntax rules
• Code generator: replaces high level
source code with semantically equivalent
low level code
Techniques to describe syntax of a
language
• Grammars: specify how you combine atomic
elements of language (characters) to form legal
strings (words, sentences)
• Finite State Machines: specify syntax of a
language through a series of interconnected
diagrams
• Regular Expressions: symbolic representation of
patterns describing strings; applications include
forming search queries as well as language
specification
 Elements of a Language
• Alphabet: finite, non-empty set of characters
– not precisely the same thing we mean when we
speak of natural language alphabet
– for example, the alphabet of C++ includes the upper-
and lowercase letters of the English alphabet, the
digits 0-9, and the following punctuation symbols:
{,},[,],(,),+,-,*,/,%,=,>,<,!,&,|,’,”,,,.,:,;,_,\
– Pep/8 alphabet is similar, but uses less punctuation
– Language of real numbers has its own alphabet; the
set of characters {0,1,2,3,4,5,6,7,8,9,+,-,.}
 Language as ADT
• A language is an example of an Abstract
Data Type (ADT)
• An ADT has these characteristics:
– Set of possible values (an alphabet)
– Set of operations on those values
• One of the operations on the set of values
in a language is concatenation
 Concatenation
• Concatenation is the joining of two or more
characters to form a string
• Many programming language tokens are formed
this way; for example:
– > and = form >=
– & and & form &&
– 1, 2, 3 and 4 form 1234
• Concatenation always involves two operands –
either one can be a string or a single character
 String characteristics
• The number of characters in a string is the
string’s length
• An empty string is a string with length 0;
we denote the empty string with the
symbol є
• The є is the identity element for
concatenation; if x is string, then:
єx = xє = x
 Closure of an alphabet
• The set of all possible strings that can formed by
concatenating elements from an alphabet is the
alphabet’s closure, denoted T* for some
alphabet T
• The closure of an alphabet includes strings that
are not valid tokens in the language; it is not a
finite set
• For example, if R is the real number alphabet,
then R* includes:
-0.092 and 563.18 but also
.0.0.- and 2-4-2.9..-5.
 Languages & Grammars
• A language is a subset of the closure of an
alphabet
• A grammar specifies how to concatenate
symbols from an alphabet to form legal
strings in a language
 Parts of a grammar
• N: a nonterminal alphabet; each element
of N represents a group of characters
from:
– T: a terminal alphabet
– P: a set of rules for string production; uses
nonterminals to describe language structure
– S: the start symbol, an element of N
Terminal vs. non-terminal symbols
• A non-terminal symbol is used to describe
or represent a set of terminal symbols
• For example, the following standard data
types are terminal symols in C++ and
Java: int, double, float, char
• The non-terminal symbol <type-specifier>
could be used to represent any or all of
these
 Valid strings
• S (the start symbol) is a single symbol, not
a set
• Given S and P (rules for production), you
can decide whether a set of symbols is a
valid string in the language
• Conversely, starting from S, if you can
generate a string of terminal symbols
using P, you can create a valid string
 Productions

Aw
a non-terminal a string of terminals &
non-terminals
produces
 Derivations
• A grammar specifies a language through
the derivation process:
– begin with the start symbol
– substitute for non-terminals using rules of
production until you get a string of terminals
 Example: a grammar for identifiers
(a toy example)
• N = {<identifier>, <letter>, <digit>}
• T = {a, b, c, 1, 2, 3}
• P = the productions: ( means “produces”)
1. <identifier><letter>
2. <identifier><identifier><letter>
3. <identifier><identifier><digit>
4. <letter>  a
5. <letter>  b
6. <letter>  c
7. <digit>  1
8. <digit>  2
9. <digit>  3
• S = <identifier>
 Example: deriving a12bc:
<identifier>  <identifier><letter> (rule 2)
 <identifier>c (rule 6)
 means  <identifier><letter>c (rule 2)
derives in one  <identifier>bc (rule 5)
step  <identifier><digit>bc (rule 3)
 <identifier>2bc (rule 8)
 <identifier><digit>2bc (rule 3)
 <identifier>12bc (rule 7)
 <letter>12bc
 a12bc
 Closure of derivation
• The symbol * means “derives in 0 or more
steps”
• A language specified by a grammar consists of
all strings derivable from the start symbol using
the rules of production
– provides operational test for membership in the
language
– if a string can’t be derived using production rules, it
isn’t in the language
 Example: attempting to derive 2a
<identifier>  <identifier><letter>
 <identifier>a
• Since there is no <identifier><digit>
combination in the production rules, we can’t
proceed any further
• This means that 2a isn’t a valid string in our
language
 A grammar for signed integers
• N = {I, F, M}
– I means integer
– F means first symbol; optional sign
– M means magnitude
• T = {+,-,d} (d means digit 0-9)
• P = the productions:
1. I  FM
2. F +
3. F -
4. F є (means +/- is optional)
5. M  dM
6. Md
• S=I
 Examples
• Deriving 14: Deriving -7:
I  FM I  FM
 єM  -M
 dM  -d
 dd  -7
 14
 Recursive rules
• Both of the previous examples (identifiers,
integers) have rules in which a
nonterminal is defined in terms of itself:
– <identifier>  <identifier><letter> and
– M  dM
• Such rules produce languages with infinite
sets of legal sentences
 Context-sensitive grammar
• A grammar in which the production rules
may contain more than one non-terminal
on the left side
• The opposite (all of the examples we have
seen thus far), have production rules
restricted to a single non-terminal on the
left: these are known as context-free
grammars
 Example
• N = {A,B,C}
• T = {a,b,c}
• P = the productions:
1. A  aABC
2. A  abC
3. CB  BC
4. bB  bb
5. bC  bc This rule is context-sensitive: C can be
substituted with c only if C is immediately
preceded by b
6. cC  cc
• S=A
 Context-sensitive grammar
• N = {A, B, C} Example:
aaabbbcc is a valid string by:
• T = {a, b, c} A => aABC (1)
=> aaABCBC (1)
• P = the productions => aaabCBCBC (2)
1. A --> aABC => aaabBCCBC (3)
=> aaabBCBCC (3)
2. A --> abC => aaabBBCCC (3)
3. CB --> BC => aaabbBCCC (4)
4. bB --> bb => aaabbbCCC (4)
=> aaabbbcCC (5)
5. bC --> bc
Here, we substituted c for C;
6. cC --> cc this is allowable only if C has
• S=A b in front of it
=> aaabbbccC (6)
=> aaabbbccc (6)
 Valid & invalid strings from
previous example:
• Valid: • Invalid:
– abc – aabc
– aabbcc – cba
– aaabbbccc – bbbccc
– aaaabbbbcccc

The grammar describes a language consisting of strings that start

with a number of a’s, followed by an equal number of b’s and c’s;
this language can be defined mathematically as:

L = {anbncn | n > 0}

Note: an means the concatenation of n a’s

 A grammar for expressions
N = {E, T, F} where:
E: expression
T: term – T = {+, *, (, ), a}
F: factor
P: the productions:
1. E -> E + T
2. E -> T
3. T -> T * F
4. T -> F
5. F -> (E)
6. F -> a
S=E
 Applying the grammar
• You can’t reach a valid conclusion if you don’t
have a valid string, but the opposite is not true
• For example, suppose we want to parse the string
(a * a) + a using the grammar we just saw
• First attempt:
E => T (by rule 2)
=> F (by rule 4)
… and, we’re stuck, because F can only produce (E) or
a; so we reach a dead end, even though the string is
valid
 Applying the grammar
• Here’s a parse that works for (a*a)+a:
E => E+T (rule 1)
=> T+T (rule 2)
=> F+T (rule 4)
=> (E)+T (rule 5)
=> (T)+T (rule 2)
=> (T*F)+T (rule 3)
=> (T*a)+T (rule 6)
=> (F*a)+F (rule 4 applied twice)
=> (a*a) + a (rule 6 applied twice)
 Deriving a valid string from a
grammar
• Arbitrarily pick a nonterminal on right side of
current intermediate string & select rules for
substitution until you get a string of terminals
• Automatic translators have more difficult
problem:
– given string of terminals, determine if string is valid,
then produce matching object code
– only way to determine string validity is to derive it
from the start string of the grammar – this is called
parsing
 The parsing problem
• Automatic translators aren’t at liberty to
pick rules randomly (as illustrated by the
first attempt to translate the preceding
expression)
• Parsing algorithm must search for the right
sequence of substitutions to derive a
proposed string
• Translator must also be able to prove that
no derivation exists if proposed string is
not valid
 Syntax tree
• A parse routine can be represented as a
tree
– start symbol is the root
– interior nodes are nonterminal symbols
– leaf nodes are terminal symbols
– children of an interior node are symbols from
right side of production rule substituted for
parent node in derivation
Syntax tree for (a*a)+a
 Grammar for a programming
language
• A grammar for a subset of the C++
language is laid out on pages 340-341 of
the textbook
• A sampling (suitable for either C++ or
Java) is given on the next couple of slides
 Rules for declarations
<declaration> -> <type-specifier><declarator-list>;
<type-specifier> -> char | int | double
(remember, this is subset of actual language)
<declarator-list> -> <identifier> |
<declarator-list> , <identifier>
<identifier> -> <letter> |
<identifier><letter> |
<identifier><digit>
<letter> -> a|b|c| … |z|A|B|…|Z
<digit> -> 0|1|2|3|4|5|6|7|8|9
 Rules for control structures
<selection-statement> ->
if (<expression>) <statement> |
if (<expression>) <statement>
else <statement>
<iteration-statement> ->
while (<expression>) <statement> |
do <statement> while (<expression>) ;
 Rules for expressions
<expression-statement> -> <expression> ;
<expression> -> <relational-expression>
| <identifier> = <expression>
<relational-expression> ->
<additive-expression> |
<relational expression> < <additive-expression> |
<relational expression> > <additive-expression> |
<relational expression> <= <additive-expression> |
<relational expression> >= <additive-expression>
etc.
 Backus-Naur Form (BNF)
• BNF is the standardized form for
specification of a programming language
by its rules of production
• In BNF, the -> operator is written ::=
• ALGOL-60 first popularized the form
 BNF described in terms of itself
(from Wikipedia)
<syntax>::= <rule> | <rule> <syntax>

<rule> ::= <opt-whitespace> "<" <rule-name> ">" <opt-whitespace> "::="

<opt-whitespace> <expression> <line-end>

<opt-whitespace> ::= " " <opt-whitespace> | ""

<!-- "" is empty string, i.e. no whitespace -->

<expression> ::= <list> | <list> "|" <expression>

<line-end> ::= <opt-whitespace> <EOL> | <line-end> <line-end>

<list> ::= <term> | <term> <opt-whitespace> <list>

<term> ::= <literal> | "<" <rule-name> ">“

<literal> ::= '"' <text> '"' | "'" <text> "'"

<!-- actually, the original BNF did not use quotes -->