Introduction to Syntax

CS 585, Fall 2018

Introduction to Natural Language Processing
http://people.cs.umass.edu/~miyyer/cs585/

Mohit Iyyer
College of Information and Computer Sciences
University of Massachusetts Amherst

(Slides and Lecture by Tu Vu)

some slides adapted from Michael Collins, Marine Carpuat, Wei Xu, and Rebecca Hwa
10/16/18 1
 A Reminder!
• Project proposal
• due on October 19, 2018 (this Friday) at 11:59 PM
• Midterm
• will be held in this room on October 25, 2018 (next Thursday)
• will cover text classification, word representations, language
modeling, sequence labeling, and machine translation
• will not cover today’s lecture and next lectures
• 20% multiple choice, 80% short answer/computational questions
• 1-page “cheat sheet” allowed, must be hand-written
• Reading for the next lecture
• JM 12

10/16/18 2
 Overview
Ø An Introduction to Syntax
Ø Constituency
Ø Context-Free Grammars (CFGs)
Ø English Grammar in a Nutshell

10/16/18 3
 Overview
Ø An Introduction to Syntax
• Syntax
• Syntax and Grammar
• Syntax vs. Semantics
• Syntax in NLP applications
• Syntactic Structure
Ø Constituency
Ø Context-Free Grammars (CFGs)
Ø English Grammar in a Nutshell

10/16/18 4
 Syntax
• Sýntaxis (setting out together or arrangement)
• The ordering of words and how they group into phrases

10/16/18 5
 Syntax
• Sýntaxis (setting out together or arrangement)
• The ordering of words and how they group into phrases

- [[students][[cook and serve][grandparents]]]

- [[students][[cook][and][serve grandparents]]]
10/16/18 6
 Syntax and Grammar
• Goal of syntactic theory
• “explain how people combine words to form sentences and
how children attain knowledge of sentence structure”
• Grammar
• implicit knowledge of a native speaker
• acquired without explicit instruction
• minimally able to generate all and only the possible
sentences of the language

Colin Phillips, “Syntax”, 2003

10/16/18 7
 Syntax vs. Semantics

“Colorless green ideas sleep furiously.”

— Noam Chomsky
(1957)

Contrast with: “sleep green furiously ideas colorless”

10/16/18 8
 Syntax in NLP application
• Syntactic analysis is often a key component in
many applications
• Grammar checkers
• Dialogue systems
• Question answering
• Information extraction
• Machine translation
• …

10/16/18 9
An Example: Machine Translation
• English word order is
• subject – verb – object
• Japanese word order is
• subject – object – verb

English: IBM bought Lotus

Japanese: IBM Lotus bought

English: Sources said that IBM bought Lotus yesterday

Japanese: Sources yesterday IBM Lotus bought that said

10/16/18 10
 Another Example: Paraphrasing

Credit: Wei Xu

10/16/18 11
 Syntactic Structure
• Constituency (phrase structure)
• Phrase structure organizes words in nested constituents

10/16/18 12
 Syntactic Structure (cont.)
• Dependency structure
• Shows which words depend on (modify or are arguments
of) which on other words

10/16/18 13
 Overview
Ø An Introduction to Syntax
Ø Constituency
• Constituency
• Grammars and Constituency
Ø Context-Free Grammars (CFGs)
Ø English Grammar in a Nutshell

3
10/16/18 14
 Constituency
• Basic idea
• Groups of words behaving as single units, or constituents

• Constituents form coherent classes that

behave similarly
• With respect to their internal structure: e.g., at the core of a
noun phrase is a noun
• With respect to other constituents: e.g., noun phrases
generally occur before verbs

10/16/18 15
 Constituency (cont.)
• Examples
• Noun phrases

• Prepositional phrases
on September seventeenth

10/16/18 16
 Constituency (cont.)
• Examples
• Noun phrases

• Prepositional phrases
on September seventeenth

What evidence do we have that these groups of words behave

as single units (or “form constituents”)?

10/16/18 17
 Constituency (cont.)
• One piece of evidence
• They can all appear in similar syntactic environments, e.g.,
before a verb

10/16/18 18
 Constituency (cont.)
• One piece of evidence
• They can all appear in similar syntactic environments, e.g.,
before a verb

• This is true for the entire phrase but not true of each of the
individual words that make up the phrase

(*) marks fragments that are not grammatical English sentences

10/16/18 19
 Constituency (cont.)
• Another piece of evidence
• They can be placed in a number of different locations, e.g.,
at the beginning (preposed) or at the end (postposed) of a
sentence

10/16/18 20
 Constituency (cont.)
• Another piece of evidence
• They can be placed in a number of different locations, e.g.,
at the beginning (preposed) or at the end (postposed) of a
sentence

• Again, the entire phrase can be placed differently, but the

individual words that make up the phrase cannot be

10/16/18 21
 Grammars and Constituency
• For a particular language:
• What are the “right” set of constituents?
• What rules govern how they combine?

10/16/18 22
 Grammars and Constituency (cont.)
• For a particular language:
• What are the “right” set of constituents?
• What rules govern how they combine?

• Answer: not obvious and difficult

• A significant part of developing a grammar involves
discovering the inventory of constituents present in the
language
• That’s why there are many different theories of grammar
and competing analyses of the same data!

10/16/18 23
 Grammars and Constituency (cont.)
• Some standard grammar formalisms:
• Context-Free Grammar (CFG)
• Lexical-Functional Grammar (LFG)
• Head-Driven Phrase Structure Grammar (HPSG),
• Tree-Adjoining Grammar (TAG),
• Combinatory Categorial Grammar (CCG)

• While CFG emphasizes phrase-structure rules,

the other approaches share the common
theme of making better use of the lexicon

10/16/18 24
 Overview
Ø An Introduction to Syntax
Ø Constituency
Ø Context-Free Grammars (CFGs)
• The Chomsky Hierarchy
• Context-Free Grammars (CFGs)
• Formal Definition of Context-Free Grammar
• Syntactic Parsing
• Examples of ambiguous structures
Ø English Grammar in a Nutshell

10/16/18 25
 The Chomsky Hierarchy
• You’ve already seen one class of grammars:
regular expressions
• A pattern like ^[a-z][0-9]$ corresponds to a grammar which
accepts (matches) some strings but not others.

• Q: Can regular languages define infinite languages?

10/16/18 26
 The Chomsky Hierarchy (cont.)
• You’ve already seen one class of grammars:
regular expressions
• A pattern like ^[a-z][0-9]$ corresponds to a grammar which
accepts (matches) some strings but not others.

• Q: Can regular languages define infinite languages?

Yes, e.g. a*

10/16/18 27
 The Chomsky Hierarchy (cont.)
• You’ve already seen one class of grammars:
regular expressions
• A pattern like ^[a-z][0-9]$ corresponds to a grammar which
accepts (matches) some strings but not others.

• Q: Can regular languages define infinite languages?

Yes, e.g. a*
• Q: Can regular languages define arbitrarily complex
languages?

10/16/18 28
 The Chomsky Hierarchy (cont.)
• You’ve already seen one class of grammars:
regular expressions
• A pattern like ^[a-z][0-9]$ corresponds to a grammar which
accepts (matches) some strings but not others.

• Q: Can regular languages define infinite languages?

Yes, e.g. a*
• Q: Can regular languages define arbitrarily complex
languages?
No. Cannot match all strings with matched
parentheses or in anbn forms in general
(recursion/arbitrary nesting).

10/16/18 https://en.wikipedia.org/wiki/Pumping_lemma_for_regular_languages 29
 The Chomsky Hierarchy (cont.)
• Hierarchy of classes of formal languages
• One grammar is of greater generative power or
complexity than another if it can define a language
that other cannot define.

• Context-free grammars are more powerful than

regular grammars, and can account for much of
the syntactic structure of English.

10/16/18 30
 Context-Free Grammars
• Context-Free Grammars (CFGs)
• Aka Phrase Structure Grammars
• Aka Backus-Naur Form (BNF)
• The most widely used formal system for modeling
constituent structure in English and other natural languages
• Good enough for most NLP applications!
The idea of basing a grammar on constituent structure dates
back to Wilhelm Wundt (1900) but was not formalized until
Chomsky (1956) and, independently, Backus (1959)

• Consist of
• Rules or productions
• Terminals
• Non-terminals
10/16/18 31
 Context-Free Grammars (cont.)

• Rules or productions
each rule can express
• the ways that symbols of the language can be grouped and
ordered together

• a lexicon of words and symbols

10/16/18 32
 Context-Free Grammars (cont.)

• Terminals
• Words in the language, e.g., “the”, “flight”

• Non-terminals
• The constituents in the language, e.g., noun phrases (NP),
verb phrases (VP)
• Express abstractions over terminals

10/16/18 33
 Context-Free Grammars (cont.)

• A grammar with examples for each rule

“|” indicates that a non-terminal has alternate possible expansions

10/16/18 34
 Context-Free Grammars (cont.)

• A lexicon

“|” indicates that a non-terminal has alternate possible expansions

10/16/18 35
 Context-Free Grammars (cont.)

• The form of a context-free rule A → β

• β is an ordered list of one or more terminals and non-
terminals
• A is a single non-terminal symbol expressing some cluster
or generalization.
• In the lexicon, β is a word and A is its lexical category, or
POS

• Two view of a CFG

• As a device for generating sentences
• As a device for assigning a structure to a given sentence

10/16/18 36
 Context-Free Grammars (cont.)

• CFG as a generator
• We can read the rule A → β as “rewrite the symbol A on the
left with string of symbols in β on the right”.

10/16/18 37
 An example
Rule expansions Rules used
S

10/16/18 38
 An example (cont.)
Rule expansions Rules used
S S → NP VP

10/16/18 39
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP

10/16/18 40
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro

10/16/18 41
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP

10/16/18 42
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP

10/16/18 43
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP

10/16/18 44
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I

10/16/18 45
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP

10/16/18 46
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer

10/16/18 47
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP

10/16/18 48
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP NP → Det Nom

10/16/18 49
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP NP → Det Nom
I prefer Det Nom

10/16/18 50
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP NP → Det Nom
I prefer Det Nom Det → a

10/16/18 51
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP NP → Det Nom
I prefer Det Nom Det → a
I prefer a Nom

10/16/18 52
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP NP → Det Nom
I prefer Det Nom Det → a
I prefer a Nom Nom → Nom Noun

10/16/18 53
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP NP → Det Nom
I prefer Det Nom Det → a
I prefer a Nom Nom → Nom Noun
I prefer a Nom Noun

10/16/18 54
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP NP → Det Nom
I prefer Det Nom Det → a
I prefer a Nom Nom → Nom Noun
I prefer a Nom Noun Nom → Noun

10/16/18 55
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP NP → Det Nom
I prefer Det Nom Det → a
I prefer a Nom Nom → Nom Noun
I prefer a Nom Noun Nom → Noun
I prefer a Noun Noun

10/16/18 56
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP NP → Det Nom
I prefer Det Nom Det → a
I prefer a Nom Nom → Nom Noun
I prefer a Nom Noun Nom → Noun
I prefer a Noun Noun Noun → flight

10/16/18 57
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP NP → Det Nom
I prefer Det Nom Det → a
I prefer a Nom Nom → Nom Noun
I prefer a Nom Noun Nom → Noun
I prefer a Noun Noun Noun → flight
I prefer a Noun flight

10/16/18 58
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP NP → Det Nom
I prefer Det Nom Det → a
I prefer a Nom Nom → Nom Noun
I prefer a Nom Noun Nom → Noun
I prefer a Noun Noun Noun → flight
I prefer a Noun flight Noun → morning

10/16/18 59
 An example (cont.)
Rule expansions Rules used
S S → NP VP
NP VP NP → Pro
Pro VP VP → Verb NP
Pro Verb NP Pro → I
I Verb NP Verb → prefer
I prefer NP NP → Det Nom
I prefer Det Nom Det → a
I prefer a Nom Nom → Nom Noun
I prefer a Nom Noun Nom → Noun
I prefer a Noun Noun Noun → flight
I prefer a Noun flight Noun → morning
I prefer a morning flight

10/16/18 60
 An example (cont.)
• Some Terminologies Sequence of
• Each grammar must have one designated start rule expansions
symbol, S S

• We say the string “I prefer a morning flight” can be NP VP

Pro VP
derived from S and the sequence of rule expansions
Pro Verb NP
is called a derivation of the string I Verb NP
• A CFG ! can be used to generate a set of strings. I prefer NP
This set of strings is called the formal language I prefer Det Nom
defined by ! I prefer a Nom
I prefer a Nom Noun
• Sentences that can be derived by ! are called I prefer a Noun Noun
grammatical sentences in the formal language I prefer a Noun flight
defined by ! I prefer a morning flight
• Sentences that cannot be derived by ! are called
ungrammatical sentences in the formal language
defined by !
10/16/18 61
 An example (cont.)
• We can represent the derivation by a parse tree
or in bracketed notation

Q: What information is conveyed by a parse tree?

10/16/18 62
 Formal Definition of Context-Free Grammar

• A context-free grammar G is defined by four

parameters: !, ", #, $

• The Kleene star means “zero or more occurrences of the

immediately previous character or regular expression”

10/16/18 63
 Syntactic Parsing
• The problem of mapping from a sentence (a
string of words) to its parse tree

10/16/18 64
 Syntactic Parsing (cont.)
• The problem with parsing: ambiguity
• Each string in the language defined by a CFG may
have more than one derivation (“ambiguity”)

10/16/18 65
 Syntactic Parsing (cont.)
• Sources of ambiguity
• Lexical ambiguity, e.g., multiple word senses, multiple
parts-of-speech
• Structural ambiguity

10/16/18 66
 Examples of ambiguous structures (cont.)

• Example 1:
“I saw her duck with a telescope”

10/16/18 67
 Examples of ambiguous structures (cont.)

• Example 1:
“I saw her duck with a telescope”

• Part-of-Speech ambiguity
NN → duck
Vi → duck

10/16/18 68
 Examples of ambiguous structures (cont.)
• Example 1:
“I saw her duck with a telescope”

10/16/18 69
 Examples of ambiguous structures (cont.)
• Example 2:
“I drove down the road in the car”

10/16/18 70
 Examples of ambiguous structures (cont.)
• Example 2:

10/16/18 71
 Examples of ambiguous structures (cont.)
• Example 3:
“the fast car mechanic”

10/16/18 72
 Examples of ambiguous structures (cont.)
• Example 3:
• Noun premodifiers

10/16/18 73
 Overview
Ø An Introduction to Syntax
Ø Constituency
Ø Context-Free Grammars (CFGs)
Ø English Grammar in a Nutshell
• Some Grammar Rules
• Treebanks

10/16/18 74
 Some Grammar Rules
• Sentence-level Constructions
• Declaratives
• Imperatives
• Yes-no questions
• Wh-questions

10/16/18 75
 Some Grammar Rules (cont.)
• Declaratives
• Form: S → NP VP I prefer a morning flight
• Have a great number of uses

• Imperatives
• Form: S → VP Show the lowest fare
• Used for commands and suggestions

10/16/18 76
 Some Grammar Rules (cont.)
• Yes-no questions
• Form: S → Aux NP VP Do any of these flights have stops?
• Often used to ask questions

• Wh-questions
• Wh-subject-questions
− Form: S → Wh-NP VP What airlines fly from Burbank to Denver?
− Identical to the declarative structure, except that the first NP contains
some wh-word
• Wh-no-subject-questions
− Form: S → Wh-NP Aux NP VP
What flights do you have from Burbank to Tacoma Washington?

10/16/18 77
 Some Grammar Rules (cont.)
• Clauses and Sentences
• The S rules are intended to account for entire
sentences that stand alone as fundamental units of
discourse
• S can also occur on the right-hand side of grammar
rules and can be embedded within larger sentences
• The S rules are some sense complete (i.e., forming a
complete thought). They correspond to the notion of
clause.

10/16/18 78
 Some Grammar Rules (cont.)
• Noun Phrases
• Can be complicated
− Determiners
− Pre-modifiers
− Post-modifiers

10/16/18 79
 Some Grammar Rules (cont.)
• Determiners
• Noun phrases can begin with determiners …
• Determiners can be
− simple lexical items a, the, this, those, any, some, etc.
− simple possessives John’s car
− complex recursive versions of that
John’s sister’s husband’s son’s car

10/16/18 80
 Some Grammar Rules (cont.)
• Premodifiers
• Come before the head
• Examples
− Cardinal numbers one, two, three
− Ordinal numbers first, next, other
− Quantifiers many, (a) few, several
− Adjectives first-class, longest, non-stop
• Ordering constraints
three large cars vs. large three cars

10/16/18 81
 Some Grammar Rules (cont.)
• Postmodifiers
• Come after the head
• Three kinds
− Prepositional phrases from Seattle
− Non-finite clauses arriving before noon
− Relative clauses that serve breakfast
• Similar recursive rules to handle these:
− Nominal → Nominal PP
− Nominal → Nominal GerundVP
− Nominal → Nominal RelClause

10/16/18 82
 Some Grammar Rules (cont.)
• Agreement Issues
• Agreement: constraints that hold among various constituents
• For example, subjects must agree with their verbs on person
and number:
I am cold. You are cold. He is cold.
* I are cold. * You is cold. * He am cold.
• Requires separate productions for each combination in CFG:
− S → NP1stPersonSing VP1stPersonSing
− S → NP2ndPersonSing VP2ndPersonSing
− NP1stPersonSing → …
− VP1stPersonSing → …
− NP2ndPersonSing → …
− VP2ndPersonSing → …
10/16/18 83
 Some Grammar Rules (cont.)
• Other agreement Issues
• Pronouns have case (e.g. nominative, accusative) that must
agree with their syntactic position.
I gave him the book. * I gave he the book.
He gave me the book. * Him gave me the book.

• Many languages have gender agreement.

Los Angeles * Las Angeles
Las Vegas * Los Vegas

10/16/18 84
 Some Grammar Rules (cont.)
• Verb Phrases
• English verb phrases consists of
− Head verb
− Zero or more following constituents (called arguments)
• Sample rules
− VP → Verb disappear
− VP → Verb NP prefer a morning flight
− VP → Verb NP PP leave Boston in the morning
− VP → Verb PP leave on Thursday

10/16/18 85
 Some Grammar Rules (cont.)
• Subcategorization Issues
• Specific verbs take some types of arguments but not others
− Transitive verb: “found” requires a direct object
John found the ring. * John found.
− Intransitive verb: “disappeared” cannot take one
John disappeared. * John disappeared the ring.
− “gave” takes both a direct and indirect object
John gave Mary the ring. * John gave Mary. *John gave the ring
− “want” takes an NP, or non-finite VP or S
John wants a car. John wants to buy a car.
John wants Mary to take the ring. * John wants.

10/16/18 86
 Some Grammar Rules (cont.)
• Subcategorization frames
• Specify the range of argument types that a given verb can take.

10/16/18 87
 Treebanks
• Data for parsing experiments
• Penn WSJ Treebank = 50,000 sentences with associated trees
• Usual set-up: 40,000 training sentences, 2400 test sentences
• Example tree

10/16/18 88
 Treebanks (cont.)
• Penn Treebank
• Treebanks implicitly define a grammar for the language
• Penn Treebank has 4500 different rules for VPs, including…
− VP → BD PP
− VP → VBD PP PP
− VP → VBD PP PP PP
− VP → VBD PP PP PP PP

10/16/18 89
 exercise!

10/16/18 90