Natalie Parde, Ph.D.

Department of Computer
Science
University of Illinois at
Chicago

CS 421: Natural Language

Processing
Fall 2019

Many slides adapted from Jurafsky and Martin

(https://web.stanford.edu/~jurafsky/slp3/) and

Syntactic and
Ben-Gurion University’s NLP course
(https://www.cs.bgu.ac.il/~michaluz/seminar/CK
Y1.pdf ).

Dependency Parsing
 What is The process of automatically recognizing
syntactic sentences and assigning syntactic

parsing?
(grammatical) structure to them.

9/17/19 Natalie Parde - UIC CS 421 2

Why is syntactic parsing useful?

• Lots of reasons!
• Grammar checking
• Sentences that can’t be parsed may be grammatically incorrect (or at least hard
to read)
• Semantic analysis
• Downstream applications
• Question answering
• Information extraction

What courses were taught by UIC CS assistant professors in 2019?

Subject = courses …don’t return a list of UIC CS assistant professors!

9/17/19 Natalie Parde - UIC CS 421 3

 Parsing algorithms are one of the core
tools for analyzing natural language.
• Parsing algorithms automatically describe the
syntactic structure of sentences in terms of
context-free grammars (such as those
discussed last week)
• This can be viewed as a search problem:
• Given the set of all possible parse trees, find
the correct parse tree for this sentence.

9/17/19 Natalie Parde - UIC CS 421 4

 • Recognition: Deciding whether a sentence
belongs to the language specified by a formal
grammar.
• Parsing: Producing a parse tree for the
sentence based on that formal grammar.

Recognition • Both tasks are necessary for generating correct

syntactic parses!
vs. Parsing • Failure to accurately recognize whether a
sentence can be parsed will lead to
misparses, which will in turn lead to
additional errors in downstream
applications.
• Parsing is more “difficult” (greater time
complexity) than recognition

9/17/19 Natalie Parde - UIC CS 421 5

Remember, language is ambiguous!
Input sentences may have many possible parses

S S

NP VP NP VP

PRP VP NP PRP VP PP

I VBP NP PP I VP NP PP NP

eat NNS PP NP VBP NNS IN NNS

spaghetti IN NNS eat spaghetti with chopsticks

with chopsticks

9/17/19 Natalie Parde - UIC CS 421 6

 Top-Down Parsing:
There are • Goal-driven
many • Builds parse tree from the start
symbol down to the terminal nodes
ways to
generate Bottom-Up Parsing:
parse
• Data-driven
trees. • Builds parse tree from the terminal
nodes up to the start symbol

9/17/19 Natalie Parde - UIC CS 421 7

 Dynamic programming
Naïve approach: approach: Save partial
Enumerate all possible solutions in a table, and
solutions use this information to
reduce search time

These approaches can be implemented naïvely,

or using more advanced techniques.
9/17/19 Natalie Parde - UIC CS 421 8
 • Assume that the input can be derived by the
designated start symbol S
• Find the tops of all trees that can start with S
• Look for all production rules with S on the
left-hand side
Top-Down • Find the tops of all trees that can start with
Parsing those constituents
• (Repeat recursively until POS categories at
bottom of tree are reached)
• Trees whose leaves fail to match all words in
the input sentence can be rejected, leaving
behind trees that represent successful parses

9/17/19 Natalie Parde - UIC CS 421 9

Top-Down Parsing: Example
Input Sentence: Grammar: Lexicon:

Book that flight. S → NP VP Det → that | this | a

S → Aux NP VP Noun → book | flight | meal | money
S → VP Verb → book | include | prefer
NP → Pronoun Pronoun → I | she | me
NP → Proper-Noun Proper-Noun → Houston | NWA
NP → Det Nominal Aux → does
Nominal → Noun Preposition → from | to | on | near | through
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb
VP → Verb NP
VP → Verb NP PP
VP → Verb PP
VP → VP PP
PP → Preposition NP
9/17/19 Natalie Parde - UIC CS 421 10
Top-Down Parsing: Example
Book that flight.
S S S
S → NP VP
S → Aux NP VP
S → VP
NP → Pronoun
NP → Proper-Noun
NP → Det Nominal
Nominal → Noun
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb
VP → Verb NP
VP → Verb NP PP
VP → Verb PP
VP → VP PP
PP → Preposition NP
9/17/19 Natalie Parde - UIC CS 421 11
Top-Down Parsing: Example
Book that flight.
S S S
S → NP VP
S → Aux NP VP NP VP Aux NP VP VP
S → VP
NP → Pronoun
NP → Proper-Noun
NP → Det Nominal
Nominal → Noun
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb
VP → Verb NP
VP → Verb NP PP
VP → Verb PP
VP → VP PP
PP → Preposition NP
9/17/19 Natalie Parde - UIC CS 421 12
Top-Down Parsing: Example
Book that flight.
S S S
S → NP VP
S → Aux NP VP NP VP NP VP NP VP
S → VP
NP → Pronoun Pronoun Verb Proper-Noun Verb NP Det Nominal Verb NP PP
NP → Proper-Noun
NP → Det Nominal
Nominal → Noun
Nominal → Nominal Noun
Nominal → Nominal PP S S S S
VP → Verb
VP → Verb NP NP VP NP VP NP VP NP VP
VP → Verb NP PP
VP → Verb PP Pronoun Verb PP Pronoun VP PP Det Nominal Verb Pronoun Verb NP PP
VP → VP PP
PP → Preposition NP
…and many more!
9/17/19 Natalie Parde - UIC CS 421 13
Top-Down Parsing: Example
Book that flight.
S S S
S → NP VP
S → Aux NP VP NP VP Aux NP VP VP
S → VP
NP → Pronoun Pronoun Verb Det Nominal Verb Verb NP
NP → Proper-Noun
NP → Det Nominal Noun Det Nominal
Nominal → Noun
Nominal → Nominal Noun Noun
Nominal → Nominal PP
VP → Verb
VP → Verb NP
VP → Verb NP PP
VP → Verb PP
VP → VP PP
PP → Preposition NP
…and many, many more not shown!
9/17/19 Natalie Parde - UIC CS 421 14
Top-Down Parsing: Example
Book that flight.
S S S
S → NP VP
S → Aux NP VP NP VP Aux NP VP VP
S → VP
NP → Pronoun Pronoun Verb Det Nominal Verb Verb NP
NP → Proper-Noun
NP → Det Nominal Noun Det Nominal
Nominal → Noun
Nominal → Nominal Noun Det → that | this | a
Noun → book | flight | meal | money Noun
Nominal → Nominal PP
VP → Verb Verb → book | include | prefer
VP → Verb NP Pronoun → I | she | me
VP → Verb NP PP Proper-Noun → Houston | NWA
VP → Verb PP Aux → does
VP → VP PP Preposition → from | to | on | near | through
PP → Preposition NP
…and many, many more not shown!
9/17/19 Natalie Parde - UIC CS 421 15
Top-Down Parsing: Example
Book that flight.
S S S
S → NP VP
S → Aux NP VP NP VP Aux NP VP VP
S → VP
NP → Pronoun Pronoun Verb Det Nominal Verb Verb NP
NP → Proper-Noun
NP → Det Nominal Noun Det Nominal
Nominal → Noun
Nominal → Nominal Noun Det → that | this | a
Noun → book | flight | meal | money Noun
Nominal → Nominal PP
VP → Verb Verb → book | include | prefer
VP → Verb NP Pronoun → I | she | me
VP → Verb NP PP Proper-Noun → Houston | NWA
VP → Verb PP Aux → does
VP → VP PP Preposition → from | to | on | near | through
PP → Preposition NP
…and many, many more not shown!
9/17/19 Natalie Parde - UIC CS 421 16
Top-Down Parsing: Example
Book that flight.
S S S
S → NP VP
S → Aux NP VP NP VP Aux NP VP VP
S → VP
NP → Pronoun Pronoun Verb Det Nominal Verb Verb NP
NP → Proper-Noun
NP → Det Nominal Noun Det Nominal
Nominal → Noun
Nominal → Nominal Noun Det → that | this | a
Noun → book | flight | meal | money Noun
Nominal → Nominal PP
VP → Verb Verb → book | include | prefer
VP → Verb NP Pronoun → I | she | me
VP → Verb NP PP Proper-Noun → Houston | NWA
VP → Verb PP Aux → does
VP → VP PP Preposition → from | to | on | near | through
PP → Preposition NP
…and many, many more not shown!
9/17/19 Natalie Parde - UIC CS 421 17
Top-Down Parsing: Example
Book that flight.
S S S
S → NP VP
S → Aux NP VP NP VP Aux NP VP VP
S → VP
NP → Pronoun Pronoun Verb Det Nominal Verb Verb NP
NP → Proper-Noun
NP → Det Nominal Noun Det Nominal
Nominal → Noun
Nominal → Nominal Noun Det → that | this | a
Noun → book | flight | meal | money Noun
Nominal → Nominal PP
VP → Verb Verb → book | include | prefer
VP → Verb NP Pronoun → I | she | me
VP → Verb NP PP Proper-Noun → Houston | NWA
VP → Verb PP Aux → does
VP → VP PP Preposition → from | to | on | near | through
PP → Preposition NP
…and many, many more not shown!
9/17/19 Natalie Parde - UIC CS 421 18
Top-Down Parsing: Example
Book that flight.
S S S
S → NP VP
S → Aux NP VP NP VP Aux NP VP VP
S → VP
NP → Pronoun Pronoun Verb Det Nominal Verb Verb NP
NP → Proper-Noun
NP → Det Nominal Noun Det Nominal
Nominal → Noun
Nominal → Nominal Noun Det → that | this | a
Noun → book | flight | meal | money Noun
Nominal → Nominal PP
VP → Verb Verb → book | include | prefer
VP → Verb NP Pronoun → I | she | me
VP → Verb NP PP Proper-Noun → Houston | NWA
VP → Verb PP Aux → does
VP → VP PP Preposition → from | to | on | near | through
PP → Preposition NP
…and many, many more not shown!
9/17/19 Natalie Parde - UIC CS 421 19
Top-Down Parsing: Example
Book that flight.
S S S
S → NP VP
S → Aux NP VP NP VP Aux NP VP VP
S → VP
NP → Pronoun Pronoun Verb Det Nominal Verb Verb NP
NP → Proper-Noun
NP → Det Nominal Noun Det Nominal
Nominal → Noun
Nominal → Nominal Noun Det → that | this | a
Noun → book | flight | meal | money Noun
Nominal → Nominal PP
VP → Verb Verb → book | include | prefer
VP → Verb NP Pronoun → I | she | me
VP → Verb NP PP Proper-Noun → Houston | NWA
VP → Verb PP Aux → does
VP → VP PP Preposition → from | to | on | near | through
PP → Preposition NP
…and many, many more not shown!
9/17/19 Natalie Parde - UIC CS 421 20
 • Earliest known parsing algorithm!
• Starts with the words in the input sentence,
and tries to build trees from those words up
by applying rules from the grammar one at
Bottom-Up a time
• Looks for places in the in-progress
Parsing parse where the righthand side of a
production rule might fit
• Success = parser builds a tree rooted in the
start symbol S that covers all of the input
words

9/17/19 Natalie Parde - UIC CS 421 21

Bottom-Up Parsing: Example
Input Sentence: Grammar: Lexicon:

Book that flight. S → NP VP Det → that | this | a

S → Aux NP VP Noun → book | flight | meal | money
S → VP Verb → book | include | prefer
NP → Pronoun Pronoun → I | she | me
NP → Proper-Noun Proper-Noun → Houston | NWA
NP → Det Nominal Aux → does
Nominal → Noun Preposition → from | to | on | near | through
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb
VP → Verb NP
VP → Verb NP PP
VP → Verb PP
VP → VP PP
PP → Preposition NP
9/17/19 Natalie Parde - UIC CS 421 22
Bottom-Up Parsing: Example
Noun Det Noun Verb Det Noun
Book that flight.
book that flight book that flight

Det → that | this | a

Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Houston | NWA
Aux → does
Preposition → from | to | on | near | through

9/17/19 Natalie Parde - UIC CS 421 23

Bottom-Up Parsing: Example
Book that flight.

S → NP VP
S → Aux NP VP
Nominal Nominal Nominal
S → VP
NP → Pronoun
Noun Det Noun Verb Det Noun
NP → Proper-Noun
NP → Det Nominal book that flight book that flight
Nominal → Noun
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb
VP → Verb NP
VP → Verb NP PP
VP → Verb PP
VP → VP PP
PP → Preposition NP
9/17/19 Natalie Parde - UIC CS 421 24
Bottom-Up Parsing: Example
Book that flight.

S → NP VP NP NP
S → Aux NP VP
Nominal Nominal VP Nominal Nominal
S → VP
NP → Pronoun
Noun Det Noun Verb Det Noun Verb Det Noun
NP → Proper-Noun
NP → Det Nominal book that flight book that flight book that flight
Nominal → Noun
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb
VP → Verb NP
VP → Verb NP PP
VP → Verb PP
VP → VP PP
PP → Preposition NP
9/17/19 Natalie Parde - UIC CS 421 25
Bottom-Up Parsing: Example
Book that flight.

S → NP VP VP
S → Aux NP VP
S → VP NP NP NP
NP → Pronoun
NP → Proper-Noun Nominal Nominal VP Nominal Nominal
NP → Det Nominal
Nominal → Noun Noun Det Noun Verb Det Noun Verb Det Noun
Nominal → Nominal Noun
Nominal → Nominal PP book that flight book that flight book that flight
VP → Verb
VP → Verb NP
VP → Verb NP PP
VP → Verb PP
VP → VP PP
PP → Preposition NP
9/17/19 Natalie Parde - UIC CS 421 26
Bottom-Up Parsing: Example
Book that flight.
S

S → NP VP VP
S → Aux NP VP
S → VP NP NP NP
NP → Pronoun
NP → Proper-Noun Nominal Nominal VP Nominal Nominal
NP → Det Nominal
Nominal → Noun Noun Det Noun Verb Det Noun Verb Det Noun
Nominal → Nominal Noun
Nominal → Nominal PP book that flight book that flight book that flight
VP → Verb
VP → Verb NP
VP → Verb NP PP
VP → Verb PP
VP → VP PP
PP → Preposition NP
9/17/19 Natalie Parde - UIC CS 421 27
Bottom-Up Parsing: Example
Book that flight.
S

S → NP VP VP
S → Aux NP VP
S → VP NP NP NP
NP → Pronoun
NP → Proper-Noun Nominal Nominal VP Nominal Nominal
NP → Det Nominal
Nominal → Noun Noun Det Noun Verb Det Noun Verb Det Noun
Nominal → Nominal Noun
Nominal → Nominal PP book that flight book that flight book that flight
VP → Verb
VP → Verb NP
VP → Verb NP PP
VP → Verb PP
VP → VP PP
PP → Preposition NP
9/17/19 Natalie Parde - UIC CS 421 28
 Top-Down Parsing

• Pros:
• Never wastes time exploring trees that cannot result in a
sentence
• Never explores subtrees that cannot fit into a larger valid
(i.e., results in a sentence) tree
Top-Down • Cons:
• Spends considerable effort on trees that are not
vs. consistent with the input
Bottom-Up
Parsing Bottom-Up Parsing

• Pros:
• Never suggests trees that are inconsistent with the input
• Cons:
• Generates many trees and subtrees that cannot result in
a valid sentence (according to production rules specified
by the grammar)

9/17/19 Natalie Parde - UIC CS 421 29

 • Structural Ambiguity: Occurs when a
grammar allows for more than one possible
parse for a given sentence
• Two Forms:
• Attachment Ambiguity: Occurs when a
More about constituent can be attached to a parse
tree at more than one place
ambiguity…. • I eat spaghetti with chopsticks.
• Coordination Ambiguity: Occurs when
different sets of phrases can be
conjoined by a conjunction
• I grabbed a muffin from the table marked
“nut-free scones and muffins,” hoping I’d
parsed the sign correctly.

9/17/19 Natalie Parde - UIC CS 421 30

 Remember …local ambiguity can also
exist.

• Det → that | this | a

• Noun → book | flight | meal |
money
Noun Det Noun Verb Det Noun • Verb → book | include | prefer
• Pronoun → I | she | me
book that flight book that flight • Proper-Noun → Houston | NWA
• Aux → does
• Preposition → from | to | on |
near | through

9/17/19 Natalie Parde - UIC CS 421 31

All of this ambiguity can lead to really
complex search spaces!
• Backtracking approaches expand the search space incrementally,
systematically exploring one state at a time
• When they arrive at trees inconsistent with the input, they return to an
unexplored alternative
• However, in doing so they tend to discard valid subtrees …this means that
time-consuming work needs to be repeated
• More efficient approach?
• Dynamic programming

9/17/19 Natalie Parde - UIC CS 421 32

 • Tables store subtrees for constituents as
Dynamic they are discovered
Programming • Solves:
Parsing • Re-parsing problem
• (Partially) ambiguity problem, since the
Methods table implicitly stores all possible parses

9/17/19 Natalie Parde - UIC CS 421 33

 Dynamic • Most widely used methods:
Programming • Cocke-Kasami-Younger (CKY) algorithm
Parsing • Earley algorithm
• Chart parsing
Methods

9/17/19 Natalie Parde - UIC CS 421 34

CKY Algorithm
• One of the earliest recognition and parsing algorithms
• Bottom-up dynamic programming
• Standard version can only recognize CFGs in Chomsky Normal Form
(CNF)

9/17/19 Natalie Parde - UIC CS 421 35

 Chomsky Normal Form
• Grammars are restricted to production rules of the form:
• A→BC
• A→w
• This means that the righthand side of each rule must expand to either two non-
terminals or a single terminal
• Any CFG can be converted to a corresponding CNF grammar that accepts exactly
the same set of strings as the original grammar!

9/17/19 Natalie Parde - UIC CS 421 36

How does this conversion work?
• Three situations we need to address:
1. Production rules that mix terminals and non-terminals on the righthand side
2. Production rules that have a single non-terminal on the righthand side (unit
productions)
3. Production rules that have more than two non-terminals on the righthand side
• Situation #1: Introduce a dummy non-terminal that covers only the original terminal
• INF-VP → to VP could be replaced with INF-VP → TO VP and TO → to
• Situation #2: Replace the non-terminals with the non-unit production rules to which they
eventually lead
• A → B and B → w could be replaced with A → w
• Situation #3: Introduce new non-terminals that spread longer sequences over multiple
rules
• A → B C D could be replaced with A → B X1 and X1 → C D

9/17/19 Natalie Parde - UIC CS 421 37

CNF
Conversion: Original CNF

Example S → NP VP S → NP VP
S → Aux NP VP S → X1 VP
• S → NP VP
• S → Aux NP VP X1 → Aux NP
• S → VP
• NP → Pronoun S → VP S → book | include | prefer
• NP → Proper-Noun
• NP → Det Nominal
S → Verb NP
• Nominal → Noun
S → X2 PP
• Nominal → Nominal Noun
• Nominal → Nominal PP X2 → Verb NP
• VP → Verb
• VP → Verb NP S → Verb PP
• VP → Verb NP PP
• VP → Verb PP S → VP PP
• VP → VP PP
• PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 38

CKY Algorithm
• With the grammar in CNF, each non-terminal node above the POS level of
the parse tree will have exactly two children
• Thus, a two-dimensional matrix can be used to encode the tree structure
• For sentence of length n, work with upper-triangular portion of (n+1) x
(n+1) matrix
• Each cell [i,j] contains a set of non-terminals that represent all constituents
spanning positions i through j of the input
• Cell that represents the entire input resides in position [0,n]

9/17/19 Natalie Parde - UIC CS 421 39

CKY Algorithm
• Non-terminal entries: For each constituent [i,j], there is a position, k, where
the constituent can be split into two parts such that i < k < j
• [i,k] must lie to the left of [i,j] somewhere along row i, and [k,j] must lie
beneath it along column j
• To fill in the parse table, we proceed in a bottom-up fashion so when we fill
a cell [i,j], the cells containing the parts that could contribute to this entry
have already been filled

9/17/19 Natalie Parde - UIC CS 421 40

 Book the flight through Chicago

CKY
Algorithm:
Example S → NP VP
S → VP
NP → Pronoun
NP → Proper-Noun
NP → Det Nominal
Nominal → Noun
Nominal → Nominal Noun Det → that | this | a | the
Nominal → Nominal PP Noun → book | flight | meal | money
VP → Verb Verb → book | include | prefer
VP → Verb NP Pronoun → I | she | me
VP → Verb PP Proper-Noun → Chicago | Dallas
VP → VP PP Aux → does
PP → Preposition NP Preposition → from | to | on | near | through

9/17/19 Natalie Parde - UIC CS 421 41

 Book the flight through Chicago
CKY
Algorithm: 1

Example
2
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas
Aux → does 3
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me
NP → Proper-Noun → Chicago | Dallas
4
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer
VP → Verb NP
5
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 42

 Book the flight through Chicago
CKY
Algorithm: 1
Noun,
Verb

Example
2
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas
Aux → does 3
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me
NP → Proper-Noun → Chicago | Dallas
4
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer
VP → Verb NP
5
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 43

 Book the flight through Chicago
CKY
Algorithm: 1
Noun,
Verb

Example
2 Det
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas
Aux → does 3
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me
NP → Proper-Noun → Chicago | Dallas
4
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer
VP → Verb NP 5
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 44

 Book the flight through Chicago
CKY
Algorithm: 1
Noun,
Verb

Example
2 Det
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas
Aux → does 3 Noun
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me
NP → Proper-Noun → Chicago | Dallas
4
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer
VP → Verb NP
5
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 45

 Book the flight through Chicago
CKY
Algorithm: 1
Noun,
Verb

Example
2 Det
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas
Aux → does 3 Noun
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep.
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer
VP → Verb NP
5
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 46

 Book the flight through Chicago
CKY
Algorithm: 1
Noun,
Verb

Example
2 Det
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas
Aux → does 3 Noun
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me
NP → Proper-Noun → Chicago | Dallas
4 Prep.
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer
VP → Verb NP 5 PropN
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 47

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP

Example
2 Det
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas
Aux → does 3 Noun
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep.
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer
VP → Verb NP
5 PropN
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 48

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP

Example
2 Det
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep.
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer
VP → Verb NP
5 PropN
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 49

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP

Example
2 Det
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep.
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 50

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP

Example
2 Det
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep.
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 51

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP

Example
2 Det
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep.
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 52

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP

Example
2 Det NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep.
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 53

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP

Example
2 Det NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep.
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 54

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP

Example
2 Det NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep.
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 55

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP

Example
2 Det NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep. PP
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 56

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP

Example
2 Det NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep. PP
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 57

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP
S, VP

Example
2 Det NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep. PP
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 58

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP
S, VP

Example
2 Det NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep. PP
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 59

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP
S, VP

Example
2 Det NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep. PP
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 60

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP
S, VP

Example
2 Det NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3 Nominal
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep. PP
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 61

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP
S, VP

Example
2 Det NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3 Nominal
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep. PP
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 62

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP
S, VP

Example
2 Det NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3 Nominal
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep. PP
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 63

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP
S, VP

Example
2 Det NP NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3 Nominal
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep. PP
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 64

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP
S, VP

Example
2 Det NP NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3 Nominal
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep. PP
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 65

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP
S, VP S, VP

Example
2 Det NP NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3 Nominal
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep. PP
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP
Note that we can arrive at S in a NP
VP → Verb PP
VP → VP PP couple different ways! Each way is an
PP → Preposition NP
alternative parse.
9/17/19 Natalie Parde - UIC CS 421 66
CKY Algorithm
• The example we just saw functions as a recognizer …for it to succeed (i.e., find
a valid sentence according to this grammar), is simply needs to find an S in cell
[0,n]
• To return all possible parses, we need to make two changes to the algorithm:
• Pair each non-terminal with pointers to the table entries from which it was
derived
• Permit multiple versions of the same non-terminal to be entered into the table
• Then, we can choose an S from cell [0,n] and recursively retrieve its component
constituents from the table

9/17/19 Natalie Parde - UIC CS 421 67

 Book the flight through Chicago
CKY Noun,

Algorithm: 1
Verb, S,
Nominal,
VP
S, VP
S1, S2,
VP

Example
2 Det NP NP
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer
Pronoun → I | she | me
Proper-Noun → Chicago | Dallas Noun,
Aux → does 3 Nominal
Nominal
Preposition → from | to | on | near | through

S → NP VP
S → VP → Verb → book | include | prefer
NP → Pronoun → I | she | me 4 Prep. PP
NP → Proper-Noun → Chicago | Dallas
NP → Det Nominal
Nominal → Noun → book | flight | meal | money
Nominal → Nominal Noun
Nominal → Nominal PP
VP → Verb → book | include | prefer PropN,
5
VP → Verb NP NP
VP → Verb PP
VP → VP PP
PP → Preposition NP

9/17/19 Natalie Parde - UIC CS 421 68

 CKY Time Complexity: Space Complexity:
Complexity O(n3) O(n2)

9/17/19 Natalie Parde - UIC CS 421 69

How can we improve upon CKY?

• Fill table in a single sweep over the input words, using a top-down
approach
• Table is length n+1, where n is equivalent to the number of words
• Table entries contain three types of information:
• A subtree corresponding to a single grammar rule
• Information about the progress made in completing the subtree
• The position of the subtree with respect to the input
• This is called Earley parsing

9/17/19 Natalie Parde - UIC CS 421 70

 • States include structures called dotted
In Earley rules
parsing, • A • within the righthand side of a state’s
grammar rule indicates the progress made
table entries towards recognizing it

are known • A state’s position with respect to the

input is represented by two numbers,
as states. indicating (1) where the state begins, and
(2) where its dot lies

9/17/19 Natalie Parde - UIC CS 421 71

 • Input: Book that flight.
• S → • VP, [0,0]
• Top-down prediction for this particular kind of S
• First 0: Constituent predicted by this state should
begin at the start of the input

Example • Second 0: Dot lies at the start of the input as well

• NP → Det • Nominal, [1,2]
States • NP begins at position 1
• Det has been successfully parsed
• Nominal is expected next
• VP → V NP •, [0,3]
• Successful discovery of a tree corresponding to
a VP that spans the entire input

9/17/19 Natalie Parde - UIC CS 421 72

 • An Earley parser moves through the n+1 sets
of states in a chart in a left-to-right fashion,
processing the states within each set in order
• At each step, one of three operators is applied
to each state depending on its status
• Predictor
• Scanner
• Completer
Earley Algorithm • This results in the addition of new states to the
end of either the current or next set of states
in the chart
• States are never removed
• The algorithm never backtracks
• The presence of S → 𝛼 •, [0,n] in the list of
states in the last chart entry indicates a
successful parse

9/17/19 Natalie Parde - UIC CS 421 73

 Predictor

• Creates new states representing top-down

expectations
• Applied to any state that has a non-terminal
immediately to the right of its dot (as long as
the non-terminal is not a POS category)
Earley • New states are placed into the same chart
entry as the generating state

Operators: • They begin and end at the same point in the

input where the generating state ends

Predictor S → • VP, [0,0]

• VP → • Verb, [0,0]
• VP → • Verb NP, [0,0]
• VP → • Verb NP PP, [0,0]
• VP → • Verb PP, [0,0]
• VP → • VP PP, [0,0]

9/17/19 Natalie Parde - UIC CS 421 74

Earley Operators: Scanner
• Used when a state has a POS category to the right of the dot
• Examines input and incorporates a state corresponding to the prediction of
a word with a particular POS into the chart
• Does so by creating a new state from the input state with the dot advanced
over the predicted input category
• VP → • Verb NP, [0,0]
• Since category following the dot is a part of speech (Verb)….
• Verb → book •, [0,1]

9/17/19 Natalie Parde - UIC CS 421 75

Earley Operators: Completer
• Applied to a state when its dot has reached the right end of the rule
• Indicates that the parser has successfully discovered a particular grammatical category
over some span of input
• Purpose: Find and advance all previously created states that were looking for this
grammatical category at this position in the input
• New states are created by copying the older state, advancing the dot over the expected
category, and installing the new state in the current chart entry
• NP → Det Nominal •, [1,3]
• What incomplete states end at position 1 and expect an NP?
• VP → Verb • NP, [0,1]
• VP → Verb • NP PP, [0,1]
• So, add VP → Verb NP •, [0,3] and the new incomplete VP → Verb NP • PP, [0,3] to
the chart

9/17/19 Natalie Parde - UIC CS 421 76

 Chart State Rule Start, End Operator
0 S0 𝛾→•S 0, 0 Start State

Earley Algorithm: 0
0
S1
S2
S → • NP VP
S → • VP
0, 0
0, 0
Predictor
Predictor

Example 0
0
S3
S4
NP → • Det Nominal
VP → • Verb
0, 0
0, 0
Predictor
Predictor
0 S5 VP → • Verb NP 0, 0 Predictor

Det → that | this | a | the

Noun → book | flight | meal | money
Verb → book | include | prefer

S → NP VP
S → VP
NP → Det Nominal
Nominal → Noun
Nominal → Nominal Noun
VP → Verb
VP → Verb NP

9/17/19 Natalie Parde - UIC CS 421 77

 Chart State Rule Start, End Operator
0 S0 𝛾→•S 0, 0 Start State

Earley Algorithm: 0
0
S1
S2
S → • NP VP
S → • VP
0, 0
0, 0
Predictor
Predictor

Example 0
0
S3
S4
NP → • Det Nominal
VP → • Verb
0, 0
0, 0
Predictor
Predictor
0 S5 VP → • Verb NP 0, 0 Predictor
1 S6 Verb → book • 0, 1 Scanner
1 S7 VP → Verb • 0, 1 Completer
1 S8 VP → Verb • NP 0, 1 Completer
1 S9 S → VP • 0, 1 Completer
1 S10 NP → • Det Nominal 1, 1 Predictor
Det → that | this | a | the
Noun → book | flight | meal | money
Verb → book | include | prefer

S → NP VP
S → VP
NP → Det Nominal
Nominal → Noun
Nominal → Nominal Noun
VP → Verb
VP → Verb NP

9/17/19 Natalie Parde - UIC CS 421 78

 Chart State Rule Start, End Operator
0 S0 𝛾→•S 0, 0 Start State

Earley Algorithm: 0
0
S1
S2
S → • NP VP
S → • VP
0, 0
0, 0
Predictor
Predictor

Example 0
0
S3
S4
NP → • Det Nominal
VP → • Verb
0, 0
0, 0
Predictor
Predictor
0 S5 VP → • Verb NP 0, 0 Predictor
1 S6 Verb → book • 0, 1 Scanner
1 S7 VP → Verb • 0, 1 Completer
1 S8 VP → Verb • NP 0, 1 Completer
1 S9 S → VP • 0, 1 Completer
1 S10 NP → • Det Nominal 1, 1 Predictor
Det → that | this | a | the
2 S11 Det → that • 1, 2 Scanner
Noun → book | flight | meal | money
Verb → book | include | prefer 2 S12 NP → Det • Nominal 1, 2 Completer
2 S13 Nominal → • Noun 2, 2 Predictor
2 S14 Nominal → • Nominal Noun 2, 2 Predictor
S → NP VP
S → VP
NP → Det Nominal
Nominal → Noun
Nominal → Nominal Noun
VP → Verb
VP → Verb NP

9/17/19 Natalie Parde - UIC CS 421 79

 Chart State Rule Start, End Operator
0 S0 𝛾→•S 0, 0 Start State

Earley Algorithm: 0
0
S1
S2
S → • NP VP
S → • VP
0, 0
0, 0
Predictor
Predictor

Example 0
0
S3
S4
NP → • Det Nominal
VP → • Verb
0, 0
0, 0
Predictor
Predictor
0 S5 VP → • Verb NP 0, 0 Predictor
1 S6 Verb → book • 0, 1 Scanner
1 S7 VP → Verb • 0, 1 Completer
1 S8 VP → Verb • NP 0, 1 Completer
1 S9 S → VP • 0, 1 Completer
1 S10 NP → • Det Nominal 1, 1 Predictor
Det → that | this | a | the
2 S11 Det → that • 1, 2 Scanner
Noun → book | flight | meal | money
Verb → book | include | prefer 2 S12 NP → Det • Nominal 1, 2 Completer
2 S13 Nominal → • Noun 2, 2 Predictor
2 S14 Nominal → • Nominal Noun 2, 2 Predictor
S → NP VP
3 S15 Noun → flight • 2, 3 Scanner
S → VP
NP → Det Nominal 3 S16 Nominal → Noun • 2, 3 Completer
Nominal → Noun 3 S17 NP → Det Nominal • 1, 3 Completer
Nominal → Nominal Noun
3 S18 Nominal → Nominal • Noun 2, 3 Completer
VP → Verb
VP → Verb NP 3 S19 VP → Verb NP • 0, 3 Completer
3 S20 S → VP • 0, 3 Completer

9/17/19 Natalie Parde - UIC CS 421 80

 Chart State Rule Start, End Operator
0 S0 𝛾→•S 0, 0 Start State
0 S1 S → • NP VP 0, 0 Predictor
0 S2 S → • VP 0, 0 Predictor
0 S3 NP → • Det Nominal 0, 0 Predictor
0 S4 VP → • Verb 0, 0 Predictor

Which 0
1
S5
S6
VP → • Verb NP
Verb → book •
0, 0
0, 1
Predictor
Scanner

states 1
1
S7
S8
VP → Verb •
VP → Verb • NP
0, 1
0, 1
Completer
Completer

participate 1
1
S9
S10
S → VP •
NP → • Det Nominal
0, 1
1, 1
Completer
Predictor

in the final
2 S11 Det → that • 1, 2 Scanner
2 S12 NP → Det • Nominal 1, 2 Completer
2 S13 Nominal → • Noun 2, 2 Predictor

parse? 2
3
S14
S15
Nominal → • Nominal Noun
Noun → flight •
2, 2
2, 3
Predictor
Scanner
3 S16 Nominal → Noun • 2, 3 Completer
3 S17 NP → Det Nominal • 1, 3 Completer
3 S18 Nominal → Nominal • Noun 2, 3 Completer
3 S19 VP → Verb NP • 0, 3 Completer
3 S20 S → VP • 0, 3 Completer

9/17/19 Natalie Parde - UIC CS 421 81

 As with • We can retrieve parse trees by adding a
field to store information about the
CKY, the completed states that generated
constituents
example • How to do this?
algorithm • Have the Completer operator add a
pointer to the previous state onto a list of
acted as a constituent states for the new state
• When an S is found in the final chart,
recognizer. just follow pointers backward

9/17/19 Natalie Parde - UIC CS 421 82

 Chart State Rule Start, End Operator (Backward Pointer)
0 S0 𝛾→•S 0, 0 Start State
0 S1 S → • NP VP 0, 0 Predictor
0 S2 S → • VP 0, 0 Predictor
0 S3 NP → • Det Nominal 0, 0 Predictor
0 S4 VP → • Verb 0, 0 Predictor

Which 0
1
S5
S6
VP → • Verb NP
Verb → book •
0, 0
0, 1
Predictor
Scanner

states 1
1
S7
S8
VP → Verb •
VP → Verb • NP
0, 1
0, 1
Completer
Completer

participate 1
1
S9
S10
S → VP •
NP → • Det Nominal
0, 1
1, 1
Completer
Predictor

in the final
2 S11 Det → that • 1, 2 Scanner
2 S12 NP → Det • Nominal 1, 2 Completer
2 S13 Nominal → • Noun 2, 2 Predictor

parse? 2
3
S14
S15
Nominal → • Nominal Noun
Noun → flight •
2, 2
2, 3
Predictor
Scanner
3 S16 Nominal → Noun • 2, 3 Completer (S15)
3 S17 NP → Det Nominal • 1, 3 Completer (S11, S15)
3 S18 Nominal → Nominal • Noun 2, 3 Completer
3 S19 VP → Verb NP • 0, 3 Completer (S6, S17)
3 S20 S → VP • 0, 3 Completer (S19)

9/17/19 Natalie Parde - UIC CS 421 83

 • Syntactic parsing is the process of
automatically determining the grammatical
structure of an input sentence
• Language is ambiguous, so sentences can
Summary: have multiple grammatically-correct
parses
Syntactic • Parsing can be performed using either a
Parsing top-down or bottom-up approach
• Common algorithms for syntactic parsing
include:
• CKY
• Earley

9/17/19 Natalie Parde - UIC CS 421 84

 • Automatically determining directed
What is grammatical (and semantic!)
relationships between words in a
dependency sentence.
parsing? • Syntactic: Focused on sentence structure
• Semantic: Focused on meaning

9/17/19 Natalie Parde - UIC CS 421 85

 • CFGs are used to automatically
generate constituent-based
representations
• Noun phrases, verb phrases, etc.
• Dependency grammars ignore
How are dependency phrase-structure rules, and instead
define sentence structure in terms of
grammars different the relationships between individual
from CFGs? words
• Nominal subject, direct object,
etc.
• For both, labels are still drawn from
a fixed inventory of grammatical
relations

9/17/19 Natalie Parde - UIC CS 421 86

 Morphologically rich: More inflections (changes to
words that influence meaning or grammatical relation)
Dependency
grammars can
deal with
languages that Free word order: Words can be moved around in a
are sentence but the overall meaning will remain the same
morphologically (syntax is less important)
rich and have a
relatively free
word order. Typically, there is a trade-off between morphological
richness and importance of syntax

9/17/19 Natalie Parde - UIC CS 421 87

Typed Dependency Structure

dobj nmod
root

det

nsubj nmod case

I prefer the morning flight through Dallas

9/17/19 Natalie Parde - UIC CS 421 88

 Comparison with Syntactic Parse
dobj nmod S
root

det

nsubj nmod case NP VP

I prefer the morning flight through Dallas

Pronoun Verb NP

prefer I prefer Nominal

Det
vs.
the
I flight Nominal PP

Nominal Noun Prep. NP

the morning Dallas
Noun flight through PropN

through morning Dallas

9/17/19 Natalie Parde - UIC CS 421 89
 • Dependency parsing provides an
approximation of the semantic
relationships between different
words in a sentence and their
Why is dependency arguments
parsing useful? • This information is useful for many
NLP applications, including:
• Coreference resolution
• Question answering
• Information extraction

9/17/19 Natalie Parde - UIC CS 421 90

 • Two components:
• Head
• Dependent
• Heads are linked to the words that are
Dependency immediately dependent on them
Relations • Relation types describe the dependent’s
role with respect to its head
• Subject
• Direct object
• Indirect object

9/17/19 Natalie Parde - UIC CS 421 91

 • Relation types tend to correlate with
sentence position and constituent type in
English, but there is not an explicit
connection between these elements
Dependency
• In more flexible languages (e.g., those with
Relations relatively free word order), the information
encoded in these relation types often
cannot be estimated otherwise (e.g., by
using a constituent tree)

9/17/19 Natalie Parde - UIC CS 421 92

 These are often referred to as dependency treebanks
Just like with
CFGs, there A few of the most popular dependency treebanks
are a variety include:

of taxonomies • Stanford dependencies

that can be • CoNLL dependencies
• Universal dependencies
used to label Just like with other corpora we've discussed so far,
dependencies these treebanks are typically created by either:

between • Having human annotators manually create

dependency structures for a collection of sentences
words. • Automatically creating initial dependency structures
and then having human annotators manually correct
those structures

9/17/19 Natalie Parde - UIC CS 421 93

Recently, most • Universal dependencies can be broken into
two sets
researchers • Clausal Relations: Describe syntactic
have moved roles with respect to predicates (the
toward using part(s) of the sentence that say
something about the subject)
universal • Modifier Relations: Describe the ways
dependencies. that words can modify their heads

9/17/19 Natalie Parde - UIC CS 421 94

Clausal Relations

dobj nmod
root

det

nsubj nmod case

I prefer the morning flight through Dallas

9/17/19 Natalie Parde - UIC CS 421 95

Modifier Relations

dobj nmod
root

det

nsubj nmod case

I prefer the morning flight through Dallas

9/17/19 Natalie Parde - UIC CS 421 96

 Universal Dependencies
Structural categories of dependent
Nominals Clauses Modifier Words Function Words
Functional categories w.r.t. head

Core Arguments nsubj csubj

of Clausal obj ccomp
Predicates iobj xcomp
Non-Core obl
aux
Dependents of vocative advmod
advcl cop
Clausal expl discourse
mark
Predicates dislocated
nmod
Dependents of det
appos acl amod
Nominals case
nummod

Other miscellaneous dependency relations (see https://universaldependencies.org/u/dep/index.html for details):

conj, cc, fixed, flat, compound, list, parataxis, orphan, goeswith, reparandum, punct, root, dep

9/17/19 Natalie Parde - UIC CS 421 97

 Universal Dependencies
Structural categories of dependent
Nominals Clauses Modifier Words Function Words
Functional categories w.r.t. head

Core Arguments nsubj csubj

Natalie wrote a dissertation.
of Clausal obj ccomp
nsubj(wrote, Natalie)
Predicates iobj xcomp
Non-Core obl
Natalie wrote a dissertation. aux
Dependents of vocative advmod
advcl dissertation)
obj(wrote, cop
Clausal expl discourse
mark
Predicates dislocated
nmod Natalie wrote him a dissertation.
Dependents of iobj(wrote, him) det
appos acl amod
Nominals case
nummod

Other miscellaneous dependency relations (see https://universaldependencies.org/u/dep/index.html for details):

conj, cc, fixed, flat, compound, list, parataxis, orphan, goeswith, reparandum, punct, root, dep

9/17/19 Natalie Parde - UIC CS 421 98

 Universal Dependencies
Structural categories of dependent
Nominals Clauses Modifier Words Function Words
Functional categories w.r.t. head

Core Arguments nsubj csubj

Natalie wrote a dissertation for him.
of Clausal obj ccomp
obl(wrote, him)
Predicates iobj xcomp
Non-Core obl
Natalie, read my dissertation! aux
Dependents of vocative advmod
advcl
vocative(read, Natalie)discourse cop
Clausal expl
mark
Predicates dislocated
nmod It is clear that her dissertation is a masterpiece.
Dependents of expl(clear, det
appos acl it) amod
Nominals case
nummod
You must not eat it, the dissertation.
dislocated(eat, dissertation)
Other miscellaneous dependency relations (see https://universaldependencies.org/u/dep/index.html for details):
conj, cc, fixed, flat, compound, list, parataxis, orphan, goeswith, reparandum, punct, root, dep

9/17/19 Natalie Parde - UIC CS 421 99

 Universal Dependencies
Structural categories of dependent
Nominals Clauses Modifier Words Function Words
Functional categories w.r.t. head

Core Arguments nsubj csubj of this dissertation is to determine an optimal

The purpose
of Clausal obj ccomp
strategy for selecting fantasy football teams.
Predicates iobj xcomp
nmod(purpose, dissertation)
Non-Core obl
My advisor, Natalie, is boring me. aux
Dependents of vocative advmod
advcl
appos(advisor, Natalie) cop
Clausal expl discourse
mark
Predicates dislocated
nmod Yesterday she wrote five dissertations.
Dependents of nummod(dissertations, five) det
appos acl amod
Nominals case
nummod

Other miscellaneous dependency relations (see https://universaldependencies.org/u/dep/index.html for details):

conj, cc, fixed, flat, compound, list, parataxis, orphan, goeswith, reparandum, punct, root, dep

9/17/19 Natalie Parde - UIC CS 421 100

 Universal Dependencies
Structural categories of dependent
Nominals Clauses Modifier Words Function Words
Functional categories w.r.t. head

Core Arguments nsubj csubj What she said about only choosing
of Clausal obj ccomp fantasy football players from northern
Predicates iobj xcomp teams makes sense.
Non-Core obl csubj(makes, said)
aux
Dependents of vocative advmod
advcl She copdissertation
said to read her second
Clausal expl discourse
for more tips. mark
Predicates dislocated
ccomp(said, read)
nmod
Dependents of det
appos acl amod
I consider her a genius. case
Nominals
nummod xcomp(consider, genius)

Other miscellaneous dependency relations (see https://universaldependencies.org/u/dep/index.html for details):

conj, cc, fixed, flat, compound, list, parataxis, orphan, goeswith, reparandum, punct, root, dep

9/17/19 Natalie Parde - UIC CS 421 101

 Universal Dependencies
Structural categories of dependent
Nominals Clauses Modifier Words Function Words
Functional categories w.r.t. head

Core Arguments nsubj csubj He was upset when she read her
of Clausal obj ccomp dissertation to him.
Predicates iobj xcomp advcl(upset, read)
Non-Core obl
aux
Dependents of vocative advmod
advcl cop
Clausal expl discourse
mark
Predicates dislocated
nmod
Dependents of det
appos acl amod
Nominals case
nummod

Other miscellaneous dependency relations (see https://universaldependencies.org/u/dep/index.html for details):

conj, cc, fixed, flat, compound, list, parataxis, orphan, goeswith, reparandum, punct, root, dep

9/17/19 Natalie Parde - UIC CS 421 102

 Universal Dependencies
Structural categories of dependent
Nominals Clauses Modifier Words Function Words
Functional categories w.r.t. head

Core Arguments nsubj csubj

of Clausal obj ccomp
Predicates iobj xcomp
Non-Core obl She added an appendix discussing
aux
Dependents of vocative fantasy bachelor.
advmod
advcl cop
Clausal expl acl(appendix,
discourse discussing)
mark
Predicates dislocated
nmod
Dependents of det
appos acl amod
Nominals case
nummod

Other miscellaneous dependency relations (see https://universaldependencies.org/u/dep/index.html for details):

conj, cc, fixed, flat, compound, list, parataxis, orphan, goeswith, reparandum, punct, root, dep

9/17/19 Natalie Parde - UIC CS 421 103

 Universal Dependencies
Structural categories of dependent
Nominals Clauses Modifier Words Function Words
Functional categories w.r.t. head

Core Arguments nsubj csubj

He was deeply offended by the
of Clausal obj ccomp
generalization from football to reality show.
Predicates iobj xcomp
advmod(offended, deeply)
Non-Core obl
aux
Dependents of vocative advmod
advcl cop
Clausal expl discourse
mark
Predicates dislocated
nmod
Dependents of She said, “Well, I have otheracl
dissertations that explain det
appos amod this further.”
Nominals discourse(explain, well) case
nummod

Other miscellaneous dependency relations (see https://universaldependencies.org/u/dep/index.html for details):

conj, cc, fixed, flat, compound, list, parataxis, orphan, goeswith, reparandum, punct, root, dep

9/17/19 Natalie Parde - UIC CS 421 104

 Universal Dependencies
Structural categories of dependent
Nominals Clauses Modifier Words Function Words
Functional categories w.r.t. head

Core Arguments nsubj csubj

of Clausal obj ccomp
Predicates iobj xcomp
Non-Core He satoblwith rapt attention as she
began reading an excerpt from her aux
Dependents of vocative advmod
third dissertation. advcl cop
Clausal expl discourse
amod(attention, rapt) mark
Predicates dislocated
nmod
Dependents of det
appos acl amod
Nominals case
nummod

Other miscellaneous dependency relations (see https://universaldependencies.org/u/dep/index.html for details):

conj, cc, fixed, flat, compound, list, parataxis, orphan, goeswith, reparandum, punct, root, dep

9/17/19 Natalie Parde - UIC CS 421 105

 Universal Dependencies
Structural categories of dependent
Nominals Clauses Modifier Words Function Words
Functional categories w.r.t. head

Core Arguments nsubj csubj

of Clausal She had included case
obj ccompstudies.
Predicates aux(included, had)
iobj xcomp
Non-Core obl
The studies demonstrated that both fantasy games aux
Dependents of vocative advmod
were essentially theadvcl
same. cop
Clausal expl discourse
cop(same, were) mark
Predicates dislocated
nmod
He knew that this would change life as he knew it.
Dependents of det
appos
mark(change, that) acl amod
Nominals case
nummod

Other miscellaneous dependency relations (see https://universaldependencies.org/u/dep/index.html for details):

conj, cc, fixed, flat, compound, list, parataxis, orphan, goeswith, reparandum, punct, root, dep

9/17/19 Natalie Parde - UIC CS 421 106

 Universal Dependencies
Structural categories of dependent
Nominals Clauses Modifier Words Function Words
Functional categories w.r.t. head

Core Arguments nsubj csubj

of Clausal obj She told him that was the point.
ccomp
Predicates iobj det(point,
xcomp the)
Non-Core obl
aux
Dependents of vocative advmod
advcl cop
Clausal expl discourse
Then she waved goodbye and went to her office. mark
Predicates dislocated
case(office, to)
nmod
Dependents of det
appos acl amod
Nominals case
nummod

Other miscellaneous dependency relations (see https://universaldependencies.org/u/dep/index.html for details):

conj, cc, fixed, flat, compound, list, parataxis, orphan, goeswith, reparandum, punct, root, dep

9/17/19 Natalie Parde - UIC CS 421 107

 Modifier Function
Nominals Clauses
Words Words

Core
In-Class Arguments
of Clausal
nsubj
obj
csubj
ccomp

Exercise Predicates
iobj xcomp

• Assign universal
dependency relations to the
following sentences: Non-Core obl
• Time flies like an aux
Dependents vocative advmod
arrow. advcl cop
of Clausal expl discourse
• Fruit flies like a mark
banana. Predicates dislocated

https://www.google.com/searc nmod
h?q=timer Dependents det
appos acl amod
of Nominals case
nummod

9/17/19 Natalie Parde - UIC CS 421 108

 In-Class Exercise

Time flies like an arrow

Fruit flies like a banana

nsubj nmod case det compound obj

9/17/19 Natalie Parde - UIC CS 421 109
 • Dependency structures are directed graphs
• G = (V, A)
• V is a set of vertices
• A is a set of ordered pairs of vertices, or
arcs
• V corresponds to the words in a sentence
• May also include punctuation
Dependency • In morphologically complex languages,
may include stems and affixes
Formalisms • Arcs capture the grammatical relationships
between those words
• According to most grammatical theories (including
those followed in this course), dependency
structures:
• Must be connected
• Must have a designated root node
• Must be acyclic

9/17/19 Natalie Parde - UIC CS 421 110

 • Directed graphs (such as those
we’ve seen already) that satisfy the
following constraints:
• Single designated root node
Dependency Trees • No incoming arcs to the root!
• All vertices except the root node
have exactly one incoming arc
• There is a unique path from the
root node to each vertex

9/17/19 Natalie Parde - UIC CS 421 111

 Two steps:

How to • Identify all head-dependent relations in

the constituent tree
translate • Identify the correct dependency
from relations for those relations

constituent One algorithm for doing this:

to • Mark the head child of each node in a
dependency phrase structure, based on a set of
predetermined rules
structures? • In the dependency structure, make the
head of each non-head child depend
on the head of the head child

9/17/19 Natalie Parde - UIC CS 421 112

 However, • Most noun phrases do not have much (or
doing this any) internal structure
can • Morphological information has little to no
presence in phrase structure trees
produce • For low resource languages in particular,
most dependency treebanks are developed
results that manually by human annotators rather than
are far from attempting to automatically translate from
constituent to dependency parse
perfect!

9/17/19 Natalie Parde - UIC CS 421 113

Types of Dependency Parsers

Transition-based
Transition • Build a single tree in a left-to-right (assuming a left-to-right language)
sweep over the input sentence

Graph-based
Graph • Search through the space of possible trees for a given sentence, and try
to find the tree that maximizes some score

9/17/19 Natalie Parde - UIC CS 421 114

 • Earliest transition-based approach:
shift-reduce parsing
• Input tokens are successively
shifted onto a stack
• The two top elements of the
Transition-based stack are matched against a set
Dependency of possible relations provided by
Parsing some knowledge source
• When a match is found, a head-
dependent relation between the
matched elements is asserted
• Goal is to find a final parse that
accounts for all words

9/17/19 Natalie Parde - UIC CS 421 115

Shift-Reduce Parsing

Oracle

Dependency Relations Input Buffer

Stack
9/17/19 Natalie Parde - UIC CS 421 116
 • We can build upon shift-reduce
parsing by defining a set of
transition operators to guide the
parser’s decisions
Transition-based • Transition operators work by
Parsing producing new configurations:
• Stack
• Input buffer of words
• Set of relations representing a
dependency tree

9/17/19 Natalie Parde - UIC CS 421 117

 Initial configuration:

• Stack contains the ROOT node

• Word list is initialized with all words
in the sentence, in order
Transition- • Empty set of relations represents
the parse
based
Parsing Final configuration:

• Stack should be empty

• Word list should be empty
• Set of relations represents the
parse

9/17/19 Natalie Parde - UIC CS 421 118

 • The operators used in transition-
based parsing then perform the
following tasks:
• Assign the current word as the
head of some other word that
Operators has already been seen
• Assign some other word that has
already been seen as the head of
the current word
• Do nothing with the current word

9/17/19 Natalie Parde - UIC CS 421 119

 • More formally, these operators are defined as:
• LeftArc: Asserts a head-dependent relation
between the word at the top of the stack and the
word directly beneath it (the second word), and
removes the second word from the stack
• Cannot be applied when ROOT is the second
element in the stack
• Requires two elements on the stack

Operators • RightArc: Asserts a head-dependent relation

between the second word and the word at the
top of the stack, and removes the word at the top
of the stack
• Requires two elements on the stack
• Shift: Removes a word from the front of the
input buffer and pushes it onto the stack
• These operators implement the arc standard
approach to transition-based parsing

9/17/19 Natalie Parde - UIC CS 421 120

 Notable characteristics:

Arc • Transition operators only assert

relations between elements at the
Standard top of the stack
• Once an element has been assigned
Approach to its head, it is removed from the stack
Transition- • Not available for further processing!

based Benefits:
Parsing • Reasonably effective
• Simple to implement

9/17/19 Natalie Parde - UIC CS 421 121

Formal Algorithm: Arc Standard
Approach
state ← {[root], [words], []}
while state not final:
# Choose which transition operator to apply
transition ← oracle(state)

# Apply the operator and create a new state

state ← apply(transition, state)

9/17/19 Natalie Parde - UIC CS 421 122

 When does • When all words in the sentence have been
consumed
the process • When the ROOT node is the only element
end? remaining on the stack

9/17/19 Natalie Parde - UIC CS 421 123

 • No! 😮
• The arc standard approach is a greedy
Is this another algorithm
• Oracle provides a single choice at each
example of step
dynamic • Parser proceeds with that choice
programming? • No other options explored
• No backtracking
• Single parse returned at the end

9/17/19 Natalie Parde - UIC CS 421 124

Arc Standard: Example
Input Buffer book me the morning flight

Stack root

Relations

9/17/19 Natalie Parde - UIC CS 421 125

Arc Standard: Example
Input Buffer me the morning flight Only one item in the stack!

Shift book from the input

Stack book root buffer to the stack

Relations

9/17/19 Natalie Parde - UIC CS 421 126

Arc Standard: Example
Input Buffer the morning flight Valid options: Shift, RightArc

Oracle selects Shift

Stack me book root
Shift me from the input
buffer to the stack

Relations

9/17/19 Natalie Parde - UIC CS 421 127

Arc Standard: Example
Input Buffer the morning flight Valid options: Shift,
RightArc, LeftArc

Stack book root Oracle selects RightArc

Remove me from the stack

Add relation (book → me) to

the set of relations

Relations (book → me)

9/17/19 Natalie Parde - UIC CS 421 128

Arc Standard: Example
Input Buffer morning flight Valid options: Shift, RightArc

Oracle selects Shift

Stack the book root
Shift the from the input
buffer to the stack

Relations (book → me)

9/17/19 Natalie Parde - UIC CS 421 129

Arc Standard: Example
Input Buffer flight Valid options: Shift,
RightArc, LeftArc

Stack morning the book root Oracle selects Shift

Shift morning from the input

buffer to the stack

Relations (book → me)

9/17/19 Natalie Parde - UIC CS 421 130

Arc Standard: Example
Input Buffer Valid options: Shift,
RightArc, LeftArc

Stack flight morning the book root Oracle selects Shift

Shift flight from the input

buffer to the stack

Relations (book → me)

9/17/19 Natalie Parde - UIC CS 421 131

Arc Standard: Example
Input Buffer Valid options: RightArc,
LeftArc

Stack flight the book root Oracle selects LeftArc

Remove morning from the

stack

Add relation (flight →

morning) to the set of
(book → me) relations
Relations
(flight → morning)

9/17/19 Natalie Parde - UIC CS 421 132

Arc Standard: Example
Input Buffer Valid options: RightArc,
LeftArc

Stack flight book root Oracle selects LeftArc

Remove the from the stack

Add relation (flight → the) to

the set of relations

(book → me)
Relations (flight → morning)
(flight → the)

9/17/19 Natalie Parde - UIC CS 421 133

Arc Standard: Example
Input Buffer Valid options: RightArc,
LeftArc

Stack book root Oracle selects RightArc

Remove flight from the

stack

Add relation (book → flight)

(book → me) to the set of relations
(flight → morning)
Relations
(flight → the)
(book → flight)

9/17/19 Natalie Parde - UIC CS 421 134

Arc Standard: Example
Input Buffer Valid options: RightArc

Oracle selects RightArc

Stack root
Remove book from the
stack

Add relation (root → book)

to the set of relations
(book → me)
(flight → morning)
Relations (flight → the)
(book → flight)
(root → book)

9/17/19 Natalie Parde - UIC CS 421 135

Arc Standard: Example
Input Buffer Valid options: None

State is final
Stack root

(book → me)
(flight → morning)
Relations (flight → the)
(book → flight) book me the morning flight
(root → book)

9/17/19 Natalie Parde - UIC CS 421 136

 • We assumed in the previous example that
our oracle was always correct …this is not
necessarily (or perhaps not even likely) the
A few things case!
worth • Incorrect choices lead to incorrect
parses since the algorithm cannot
noting…. perform any backtracking
• Alternate sequences may also lead to
equally valid parses

9/17/19 Natalie Parde - UIC CS 421 137

How do we get actual dependency
labels?
• Parameterize LeftArc and RightArc
• LeftArc(nsubj), RightArc(obj), etc.
iobj(book → me)
• Of course, this makes the oracle’s job compound(flight → morning)
det(flight → the)
more difficult (much larger set of obj(book → flight)
operators from which to choose!) root(root → book)

9/17/19 Natalie Parde - UIC CS 421 138

 How does the oracle know what to
choose?

• State of the art systems use supervised machine

learning for this task
• This requires a training set of configurations labeled with
correct transition operators
• The person designing the system needs to decide what
types of features should be extracted from these
configurations to best train the oracle (a machine learning
model)
• The oracle will then learn which transitions to predict for
previously-unseen configurations based on the extracted
features and associated labels for configurations in the
training set

9/17/19 Natalie Parde - UIC CS 421 139

What Commonly:
types of
machine • Logistic regression
learning • Support vector machines
models
are used Recently:
as
oracles? • Neural networks

9/17/19 Natalie Parde - UIC CS 421 140

 • Search through the space of
possible trees for a given sentence,
attempting to maximize some score
• This score is generally a function of
the scores of individual subtrees
Graph-based within the overall tree
Dependency • Edge-factored approaches
Parsing determine scores based on the
scores of the edges that comprise
the tree
• overall_score(t) = ∑$∈& 𝑠𝑐𝑜𝑟𝑒(𝑒)
• Letting t be a tree for a given
sentence, and e be its edges

9/17/19 Natalie Parde - UIC CS 421 141

 Why use • Transition-based methods tend to have high
accuracy on shorter dependency relations,
graph- but that accuracy declines as the distance
between the two words increases
based • This is largely due to the fact that transition-
methods for based methods are greedy …they can be
fooled by seemingly-optimal local solutions
dependency • Graph-based methods score entire trees,
parsing? thereby avoiding that issue

9/17/19 Natalie Parde - UIC CS 421 142

 Given an input sentence, construct a
fully-connected, weighted, directed
graph

• Vertices are input words

Maximum • Directed edges represent all possible head-
Spanning dependent assignments
• Weights reflect the scores for each possible
Tree head-dependent assignment, predicted by a
supervised machine learning model

A maximum spanning tree represents

the preferred dependency parse for
the sentence, as determined by the
weights

9/17/19 Natalie Parde - UIC CS 421 143

Maximum Spanning Tree: Example

4
4
8
12 flight
5 7
that
6
root book

9/17/19 Natalie Parde - UIC CS 421 144

Maximum Spanning Tree: Example

4
4
8
12 flight
5 7
that
6
root book

9/17/19 Natalie Parde - UIC CS 421 145

 Two • Every vertex in a spanning tree has exactly
one incoming edge
intuitions to • Absolute values of the edge scores are not
keep in critical
• Relative weights of the edges entering a
mind…. vertex are what matter!

9/17/19 Natalie Parde - UIC CS 421 146

 • Given a fully-connected graph G =
(V, E), a subgraph T = (V, F) is a
How do we know spanning tree if:
that we have a • It has no cycles
spanning tree? • Each vertex (except the root) has
exactly one edge entering it

9/17/19 Natalie Parde - UIC CS 421 147

 • If the greedy selection process produces
a spanning tree, then that tree is the
How do we maximum spanning tree

know that • However, the greedy selection process may

select edges that result in cycles
we have a • If this happens, an alternate graph can be
created that collapses cycles into new
maximum nodes, with edges that previously entered
or exited the cycle now entering or exiting
spanning the new node
tree? • The greedy selection process is then
recursively applied to the new graph until a
(maximum) spanning tree is found

9/17/19 Natalie Parde - UIC CS 421 148

Formal Algorithm
F ← []
T ← []
score’ ← []
for each v in V do:
bestInEdge ← argmax 𝑠𝑐𝑜𝑟𝑒[𝑒]
$3(4,6)∈7

F ← F ∪ bestInEdge
for each 𝑒 = 𝑢, 𝑣 ∈ 𝐸 do:
score’[e] ← score[e] - score[bestInEdge]

if T=(V,F) is a spanning tree:

return T
else:
C ← a cycle in F
G’ ← collapse(G, C)
T’ ← maxspanningtree(G’, root, score’) # Recursively call the current function
T ← expand(T’, C)
return T

9/17/19 Natalie Parde - UIC CS 421 149

Maximum Spanning Tree: Updated
Example

4
4
8
12 flight
5 7
that
6
root book

9/17/19 Natalie Parde - UIC CS 421 150

Maximum Spanning Tree: Updated
Example

4
4
8
12 flight
5 7
that
6
root book

9/17/19 Natalie Parde - UIC CS 421 151

Maximum Spanning Tree: Updated
Example

-4
-3
0
flight
0 8
-2 0
that
7
book -6
root
12
-7

-1

9/17/19 Natalie Parde - UIC CS 421 152

Maximum Spanning Tree: Updated
Example

-4
-3

0
-2 that-
flight
book -6 -1
root
12
-7

-1

9/17/19 Natalie Parde - UIC CS 421 153

Maximum Spanning Tree: Updated
Example

-4
-3

0
-2 that-
flight
book -6 -1
root
12
-7

-1

9/17/19 Natalie Parde - UIC CS 421 154

Maximum Spanning Tree: Updated
Example

-4
-3
0
flight
0 8
-2 0
that
7
book -6
root
12
-7

-1

9/17/19 Natalie Parde - UIC CS 421 155

Maximum Spanning Tree: Updated
Example

-4
-3
0
flight
0 8
-2 0
that
7
book -6
root
12
-7

-1

9/17/19 Natalie Parde - UIC CS 421 156

Maximum Spanning Tree: Updated
Example

-4
-3
0
flight
0 8
-2 0
that
7
book -6
root
12
-7

-1

9/17/19 Natalie Parde - UIC CS 421 157

 • Similar approach to training the oracle in a
How do we transition-based parser
train our • Common features can include:
• Words, lemmas, parts of speech
model to • Corresponding features from contexts
predict before and after words
• Word embeddings
edge • Dependency relation type
• Dependency relation direction
weights? • Distance from head to dependent

9/17/19 Natalie Parde - UIC CS 421 158

 • Dependency parsing is the process of
automatically determining directed relationships
between words in a source sentence
• Many dependency taxonomies exist, but the most
common taxonomy for English text is the set of
Summary: universal dependencies
• Dependency parsers can be transition-based or
Dependency graph-based

Parsing • A popular transition-based method is the arc

standard approach
• A popular graph-based method is the maximum
spanning tree approach
• Both make use of supervised machine learning to
aid the decision-making process

9/17/19 Natalie Parde - UIC CS 421 159