Lecture Notes On Syntactic Processing
Lecture Notes On Syntactic Processing
Syntactic Processing
Syntactic processing is widely used in applications such as question answering systems, information extraction,
sentiment analysis, grammar checking etc. In this module, you learnt that there are three broad levels of syntactic
processing - (Parts-of-Speech) POS tagging, constituency parsing, and dependency parsing. And, POS tagging is a
crucial task in syntactic processing and is used as a preprocessing step in many NLP applications.
Parts-of-Speech Tags
You learnt about most commonly used tags in Penn Treebank of NLTK. Here is the list of the commonly
used tags:
Noun
NNS Noun, singular or tag of plural forms of common nouns Cats buses,
mass animals
NNP Proper noun, Tag of noun names, places and things India, Rahul,
singular Eric
NNPS Proper noun, plural Names of nations and nationalities, Personal Germans,
names Indians, three
Billys
VBD Verb, past form Past tense verbs to express action/state of Learnt, ate,
the past studied
VBN Verb, past Verbs in the past form when used with - Have died
participle has/have/had/modal verb. Common - Had waited
scenarios are: -Could have
Have + ‘verb’ written
Might + have + ‘verb’ - were excited
were/is + ‘verb’
VBP Verb, non-3rd Non 3rd person verb to express I like this
person singular routines/habits, facts/truth and thoughts and game.
present feelings (used without word ‘to’) I go for a walk
daily
VBZ Verb, 3rd person Verbs that are used for 3rd person singular Argues,
singular present entities catches,
-ends with ‘s’, ‘es’ replies
Next, you learnt about the four main techniques used for POS tagging:
• Lexicon-based approach uses the following simple statistical algorithm: for each word, it assigns the
POS tag that most frequently occurs for that word in some training corpus.
For example, it will assign the tag "verb" to any occurrence of the word "run" if "run" is used as a
verb more often than any other tag.
• Rule-based taggers first assign the tag using the lexicon method and then apply predefined rules.
Some examples of rules are: Change the tag to VBG for words ending with ‘-ing’, Changes the tag to
VBD for words ending with ‘-ed’, etc.
You also learnt to implement the lexicon and rule-based tagger on the Treebank corpus of NLTK. Next, you
learnt about:
• Probabilistic (or stochastic) techniques don't naively assign the highest frequency tag to each word,
instead, they look at slightly longer parts of the sequence and often use the tag(s) and the word(s)
appearing before the target word to be tagged. You learnt about the commonly used probabilistic
algorithm for POS tagging - the Hidden Markov Model (HMM)
Deep-learning based POS tagging: Recurrent Neural Networks (RNNs) are used for sequential modeling
processes. In this module, you got a basic overview on how RNNs are used for POS tagging.
In probabilistic method, you learnt about Markov processes and HMMs. Markov processes are commonly
used to model sequential data, such as text and speech. You learnt that the first-order Markov assumption
states that the probability of an event (or state) depends only on the previous state.
The Hidden Markov Model, an extension to the Markov process, is used to model phenomena where the
states are hidden and they emit observations. The transition and the emission probabilities specify the
probabilities of transition between states and emission of observations from states, respectively. In POS
tagging, the states are the POS tags while the words are the observations. To summarise, a Hidden Markov
Model is defined by the initial state, emission, and the transition probabilities. Refer to following image to
revise transition, emission probabilities:
So, the probability of a tag sequence (T , T , T , …, T ) for a given the word sequence (W , W , W , …, W ) can
1 2 3 n 1 2 3 n
be defined as:
P(T|W) = (P(W |T ) * P(T |start)) * (P(W |T ) * P(T |T )) * ...* (P(W |T ) * P(T |T ))
1 1 1 2 2 2 1 n n n n-1
You learnt that for a sequence of n words and t tags, a total of t tag sequences are possible. The Penn
n
Treebank dataset in NLTK itself has 36 POS tags, so for a sentence of length say 10, there are 36 possible 10
tag sequences.
Viterbi Heuristic
Next, you studied how Viterbi Heuristic can deal with this problem by taking a greedy approach. The basic
idea of the Viterbi algorithm is as follows - given a list of observations (words) O1,O2....On to be tagged,
rather than computing the probabilities of all possible tag sequences, you assign tags sequentially, i.e.
assign the most likely tag to each word using the previous tag.
More formally, you assign the tag T to each word W such that it maximises the likelihood:
i i
where T is the tag assigned to the previous word. The probability of a tag T is assumed to be dependent
i-1 i
only on the previous tag T , and hence the term P(T |T ) - Markov Assumption.
i-1 i i-1
Next you learnt that the Viterbi algorithm is an example of a dynamic programming algorithm. In general,
algorithms which break down a complex problem into subproblems and solve each subproblem optimally
are called dynamic programming algorithms.
Next, you learnt to compute the emission & transition probabilities from a tagged corpus. This process of
learning the probabilities from a tagged corpus is called training an HMM model. The emission and the
transition probabilities can be learnt as follows:
You learnt how to build a POS tagger using Viterbi Heuristic. For training the HMM, i.e., for learning the
model parameters, you used the NLTK Treebank corpus. After learning the model parameters, you find the
best possible state (tag) sequence for each given sentence. For that, you used the Viterbi algorithm - for
every word w in the sentence, a tag t is assigned to w such that it maximises the likelihood of the
occurrence of P(tag|word).
In other words, the tag t is assigned to the word w which has the max P(tag|word).
The assigned tags and words are then stored as a list of tuples. As you move to the next word in the list,
each tag to be assigned will use the tag of the previous word.
You saw that the Viterbi algorithm gave ~87% accuracy. The 13% loss of accuracy was majorly because of
the fact that when the algorithm hit an unknown word (i.e. not present in the training set), it naively
assigned the first tag in the list of tags that we have created.
Next, you got a brief overview of how you can build POS taggers using RNNs. Recurrent Neural Networks (RNNs)
have empirically proven to outperform many conventional sequence models for tasks such as POS tagging, entity
recognition, dependency parsing etc. You’ll learn RNNs in detail later in the Neural Network course.
Constituency Parsing
Next, you studied why shallow parsing is not sufficient. Shallow parsing, as the name suggests, refers to
fairly shallow levels of parsing such as POS tagging, chunking, etc. But such techniques would not be able
to check the grammatical structure of the sentence, i.e. whether a sentence is grammatically correct, or
understand the dependencies between words in a sentence.
So, you learnt the two most commonly used paradigms of parsing - constituency parsing and dependency
parsing, which would help to check the grammatical structure of the sentence.
In constituency parsing, you learnt the basic idea of constituents as grammatically meaningful groups of
words, or phrases, such as noun phrase, verb phrase etc. You also learnt the idea of context-free grammars
or CFGs which specify a set of production rules. Any production rule can be written as A -> B C, where A is a
non-terminal symbol (NP, VP, N etc.) and B and C are either non-terminals or terminal symbols (i.e. words
in vocabulary such as flight, man etc.). Example a CFG is:
S -> NP VP
NP -> DT N| N| N PP
VP -> V| V NP
N -> ‘man’| ‘bear’
V -> ‘ate’
DT -> ‘the’| ‘a’
You also learnt how to build both these types of parsed structures in Python.
Since natural languages are inherently ambiguous (at least for computers to understand), there are often
cases where multiple parse trees are possible. In such cases, we need a way to make the algorithms figure
out the most likely parse tree. Probabilistic Context-Free Grammars (PCFGs) are used when we want to
find the most probable parsed structure of the sentence. PCFGs are grammar rules, similar to what you
have seen before, along with probabilities associated with each production rule. An example production
rule is as follows:
NP -> Det N (0.5) | N (0.3) |N PP (0.2)
It means that the probability of an NP breaking down to a ‘Det N’ is 0.50, to an 'N' is 0.30 and to an ‘N PP’ is
0.20. Note that the sum of probabilities is 1.00.
Overall probability for a parsed structure of the sentence is probabilities of all rules used in that parsed
structure. The parsed tree with maximum probability is best possible interpretation of the sentence. You
also learnt to implement PCFG in Python.
Dependency Parsing
After constituency parsing, you learnt about Dependency Parsing. In dependency grammar, constituencies
(such as NP, VP etc.) do not form the basic elements of grammar, but rather dependencies are established
between the words themselves.
Next, you learnt about universal dependencies. Apart from dependencies defined in the form of subject-
verb-object, there's a non-exhaustive list of dependency relationships, which are called universal
dependencies.
Dependencies are represented as labelled arcs of the form h → d (l) where 'h' is called the “head” of the
dependency, 'd' is the “dependent” and l is the “label” assigned to the arc. In a dependency parse, we start
from the root of the sentence, which is often a verb. And then start to establish dependencies between
root and other words.
Information Extraction
In this session, you learnt to build an information extraction (IE) system which can extract entities relevant
for booking flights (such as source and destination cities, time, date, budget constraints etc.) in a
structured format from unstructured user-generated queries. IE is used in many applications such as
conversational chatbots, extracting information from encyclopedias (such as Wikipedia), etc. In this
session, you learnt to use the ATIS dataset for IE.
A generic IE pipeline is as follows:
1. Preprocessing
1. Sentence Tokenization: sequence segmentation of text.
You learnt about IOB labeling. IOB (or BIO) method tags each token in the sentence with one of the three
labels: I - inside (the entity), O- outside (the entity) and B - beginning (of entity). You saw that IOB labeling
is especially helpful if the entities contain multiple words. For example: words like ‘Air India’, ‘New Delhi’,
etc, are single entities.
Next, you learnt in detail about Rule-based method: Chunking. Chunking is a commonly used shallow
parsing technique used to chunk words that constitute some meaningful phrase in the sentence. A noun
phrase chunk (NP chunk) is commonly used in NER tasks to identify groups of words that correspond to
some 'entity'.
Sentence: He bought a new car from the Maruti Suzuki showroom.
Noun phrase chunks - a new car, the Maruti Suzuki showroom
The idea of chunking in the context of entity recognition is simple - most entities are nouns and noun
phrases, so rules can be written to extract these noun phrases and hopefully extract a large number of
named entities. Example of chunking done using regular expressions:
Sentence: Ram booked the flight.
Noun phrase chunks: 'Ram', 'the flight'
Grammar: 'NP_chunk: {<DT>?<NN>}'
© Copyright 2018. UpGrad Education Pvt. Ltd. All rights reserved
Probabilistic method for NER
Next, you learnt the following two probabilistic models to get the most probable IOB tags for word:
1. Unigram chunker computes the unigram probabilities P(IOB label | pos) for each word and assigns
the label that is most likely for the POS tag.
1. Bigram chunker works similar to a unigram chunker, the only difference being that now the
probability of a POS tag having an IOB label is computed using the current and the previous POS
tags, i.e. P(label | pos, prev_pos).
Gazetteer Lookup
Another way to identify named entities (like cities and states) is to look up a dictionary or a gazetteer. A
gazetteer is a geographical directory which stores data regarding the names of geographical entities (cities,
states, countries) and some other features related to the geographies.
You studied that just like machine learning classification models, you can have features for sequence labelling task.
Features could be the morphology (or shape) of the word such as whether the word is upper/lowercase, POS tags of
the words in the neighbourhood, whether the word is present in the gazetteer (i.e. word_is_city, word_is_state),
etc. And using these features, you learnt to build a Naive Bayes classifier and Decision Tree classifier.
HMMs can be used for any sequence classification task, such as NER. However, many NER tasks and
datasets are far more complex than tasks such as POS tagging, and therefore, more sophisticated sequence
models have been developed and widely accepted in the NLP community. One of these models is
Conditional Random Fields (CRFs).
CRFs are used in a wide variety of sequence labelling tasks across various domains - POS tagging, speech
recognition, NER, and even in computational biology for modelling genetic patterns etc.
Next, you studied the architecture of CRFs. CRFs model the conditional probability P(Y|X), where Y is the
vector of output sequence (IOB labels here) and X is the input sequence (words to be tagged), which are
similar to Logistic Regression classifier. Broadly, there are two types of classifiers in ML:
1. Discriminative classifiers learn the boundary between classes by modelling the conditional
probability distribution P(y|x), where y is the vector of class labels and x represents the input
features. Examples are Logistic Regression, SVMs etc.
1. Generative classifiers model the joint probability distribution P(x,y). Examples of generative
classifiers are Naive Bayes, HMMs etc.
Next, you learnt about CRFs’ feature functions. CRFs use ‘feature functions’ rather than the input word
sequence x itself. The idea is similar to how features are extracted for building the naive Bayes and
decision tree classifiers in a previous section. Some example ‘word-features’ (each word has these
features) are:
• Word and POS tag based features: word_is_city, word_is_digit, pos, previous_pos, etc.
• Label-based features: previous_label
The feature function returns 1 only if both the conditions are satisfied, i.e. when the word is a city name
and is tagged as ‘I-location’ (e.g. Chicago/I-location).
Every feature function fi has a weight wi associated with it, which represents the ‘importance’ of that
feature function. This is almost exactly the same as logistic regression where coefficients of features
represent their importance. Training a CRF means to compute the optimal weight vector w which best
represents the observed sequences y for the given word sequences x. In other words, we want to find the
set of weights w which maximises P(y|x,w).
In CRFs, the conditional probabilities P(y|x,w) are modeled using a scoring function. If there are k feature
functions (and thus k weights), for each word i in the sequence x, a scoring function for a word is defined
as follows:
and overall sequence score for the sentence can be defined as:
The probability of observing the label sequence y given the input sequence x is given by:
By taking log and simplifying the equations, the final equation comes out as:
The final equation after taking the gradient of the log-likelihood function is:
Prediction using CRF: the inference task to assign the label sequence y to x which maximises the score of
*
The naive way to get y* is by calculating w.f(x,y) for every possible label sequence , and then choose the
label sequence that has maximum (w.f(x,y)) value. However, there are an exponential number of possible
labels (tn for a tag set of size t and a sentence of length n), and this task is computationally heavy. You
learnt how to derive the best possible path using Viterbi algorithm.
You also learnt the Python implementation of CRF. CRFs outperformed the rule-based and ML-classification
algorithms.
• You can download this document from the website for self-use only.
• Any copies of this document, in part or full, saved to disc or to any other storage medium may only be used
for subsequent, self-viewing purposes or to print an individual extract or copy for non-commercial personal
use only.
• Any further dissemination, distribution, reproduction, copying of the content of the document herein or the
uploading thereof on other websites or use of content for any other commercial/unauthorized purposes in
any way which could infringe the intellectual property rights of UpGrad or its contributors, is strictly
prohibited.
• No graphics, images or photographs from any accompanying text in this document will be used separately
for unauthorised purposes.
• No material in this document will be modified, adapted or altered in any way.
• No part of this document or UpGrad content may be reproduced or stored in any other web site or included
in any public or private electronic retrieval system or service without UpGrad’s prior written permission.
• Any rights not expressly granted in these terms are reserved.