SRI KRISHNA INSTITUTE OF TECHNOLOGY

(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

Module-4

Semantic Parsing

1.​ Introduction

•​ Two approaches have emerged in the NLP for language understanding.

•​ In the first approach, a specific, rich meaning representation is created for a
limited domain for use by application that are restricted to that domain, such as
travel reservations, football game simulations, or querying a geographic
database.
•​ In the second approach, a related set of intermediate-specific meaning representation
is created, going from low-level analysis to a middle analysis, and the bigger
understanding task is divided into multiple, smaller pieces that are more manageable,
such as word sense disambiguation followed by predicate-argument structure
recognition.
•​ Here two types of meaning representations: a domain-dependent, deeper
representation and a set of relatively shallow but general-purpose, low-level, and
intermediate representation.
•​ The task of producing the output of the first type is often called deep semantic
parsing, and the task of producing the output of the second type is often called
shallow semantic parsing.
•​ The first approach is so specific that porting to every new domain can require
anywhere from a few modifications to almost reworking the solution from scratch.

•​ In other words, the reusability of the representation across domains is very limited.
•​ The problem with second approach is that it is extremely difficult to construct a
general-purpose ontology and create symbols that are shallow enough to be learnable
but detailed enough to be useful for all possible applications.

•​ Ontology means
1.​ The branch of metaphysics dealing with the nature of being.
2.​ a set of concepts and categories in a subject area or domain that shows their properties
and the relations between them.
"what's new about our ontology is that it is created automatically from large datasets"

•​ Therefore, an application specific translation layer between the more general

representation and the more specific representation becomes necessary.

2.​ Semantic Interpretation

Semantic parsing can be considered as part of Semantic interpretation, which

involves various components that together define a representation of text that can be
fed into a computer to allow further computations manipulations and search, which
are prerequisite for any language understanding system or application. Here we
discuss the structure of semantic theory.

A Semantic theory should be able to:

​ Explain sentence having ambiguous meaning: The bill is large is ambiguous in the
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

sense that is could represent money or the beak of a bird.

 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

​ Resolve the ambiguities of words in context. The bill is large but need not be paid, the
theory should be able to disambiguate the monetary meaning of bill.
​ Identify meaningless but syntactically well-formed sentence: Colorless green ideas
sleep furiously.
​ Identify syntactically or transformationally unrelated paraphrasers of concept having
the same semantic content.

​ Here we look at some requirements for achieving a semantic representation.

2.1​Structural Ambiguity

​ Structure means syntactic structure of sentences.

​ The syntactic structure means transforming a sentence into its underlying syntactic
representation and in theory of semantic interpretation refer to underlying syntactic
representation.

2.2​Word Sense

​ In any given language, the same word type is used in different contexts and with
different morphological variants to represent different entities or concepts in the
world.
​ For example, we use the word nail to represent a part of the human anatomy and also
to represent the generally metallic object used to secure other objects.

2.3​Entity and Event Resolution

​ Any discourse consists of a set of entities participating in a series of explicit or

implicit events over a period of time.
​ So, the next important component of semantic interpretation is the identification of
various entities that are sparkled across the discourse using the same or different
phrases.
​ The predominant tasks have become popular over the years: named entity
recognition and coreference resolution.
​ Coreference resolution is the task of finding all expressions that refer to the same
entity in a text.
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

2.4​Predicate Argument Structure

 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

​ Once we have the word-sense, entities and events identified, another level of
semantics structure comes into play: identifying the participants of the entities in
these events.
​ Resolving the argument structure of predicate in the sentence is where we identify
which entities play what part in which event.
​ A word which functions as the verb is called a predicate and words which function as
the nouns arecalled arguments. Here are some other predicates and arguments:

2.5​Meaning Representation

​ The final process of the semantic interpretation is to build a semantic

representation or meaning representation that can then be manipulated by
algorithms to various application ends.
​ This process is sometimes called the deep representation.

The following two examples

3.​ System Paradigms

●​ It is important to get a perspective on the various primary dimensions on which the

 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

problem of semantic interpretation has been tackled.

●​ The approaches generally fall into the following three categories: 1.System architecture
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

2.​ Scope 3. Coverage.

1.​System Architectures
a.​ Knowledge based: These systems use a predefined set of rules or a knowledge base
to obtain a solution to a new problem.
b.​ Unsupervised: Thesesystemstendto requireminimal humanintervention to be
functional by using existing resources that can be bootstrapped for a particular
application or problem domain.
c.​ Supervised: these systems involve the manual annotation of some
phenomena that appear in a sufficient quantity of data so that machine
learning algorithms can be applied.
d.​ Semi-Supervised: manual annotation is usually very expensive and does not
yield enough data to completely capture a phenomenon. In such instances,
researches can automatically expand the data set on which their models are
trained either by employing machine-generated output directly or by
bootstrapping off an existing model by having humans correct its output.

2.​Scope:
​ Domain Dependent: These systems are specific to certain domains, such as air travel
reservations or simulated football coaching.
​ Domain Independent: These systems are general enough that the techniques can be
applicable to multiple domains without little or no change.

3.​ Coverage
a.​ Shallow: These systems tend to produce an intermediate representation that can then be
converted to one that a machine can base its action on.
b.​ Deep: These systems usually create a terminal representation that is directly consumed by a
machine or application.

4.​ Word Sense

​ Word Sense Disambiguation is an important method of NLP by which the meaning
of a word is determined, which is used in a particular context.
​ In a compositional approach to semantics, where the meaning of the whole is
composed on the meaning of parts, the smallest parts under consideration in textual
discourse are typically the words themselves: either tokens as they appear in the text
or their lemmatized forms.
​ Words sense has been examined and studied for a very long time.
​ Attempts to solve this problem range from rule based and knowledge based to
completely unsupervised, supervised, and semi-supervised learning methods.
​ Very early systems were predominantly rule based or knowledge based and used
dictionary definitions of senses of words.
​ Unsupervised word sense induction or disambiguation techniques try to induce the
senses or word as it appears in various corpora.
​ These systems perform either a hard or soft clustering of words and tend to allow the
tuning of these clusters to suit a particular application.
​ Most recent supervised approaches to word sense disambiguation, usually
application- independent-level of granularity (including small details). Although the
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

output of supervised approaches can still be amendable to generating a ranking,

 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

or distribution, of membership sense.

​ Word sense ambiguities can be of three principal types: i.homonymy ii.polysemy
iii.categorial ambiguity.
​ Homonymy defined as the words having same spelling or same form but having
different and unrelated meaning. For example, the word “Bat” is a homonymy word
because bat can be an implement to hit a ball or bat is a nocturnal flying mammal
also
​ Polysemy is a Greek word, which means “many signs”. polysemy has the same
spelling but different and related meaning.
​ Both polysemy and homonymy words have the same syntax or spelling. The main
difference between them is that in polysemy, the meanings of the words are related
but in homonymy, the meanings of the words are not related.
​ For example: Bank Homonymy: financial bank and river bank
Polysemy: financial bank, bank of clouds and book bank: indicate collection of
things.
​ Categorial ambiguity: the word book can mean a book which contain the chapters or
police register which is used to enter the charges against someone.
​ In the above note book, text book belongs to the grammatical category of noun and
book is verb.
​ Distinguishing between these two categories effectively helps disambiguate these two
senses.
​ Therefore, categorical ambiguity can be resolved with syntactic information (part
of speech) alone, but polyseme and homonymy need more than syntax.
​ ​Traditionally, in English, word senses have been annotated for each part of speech
separately, whereas in Chinese, the sense annotation has been done per lemma.

Resources:
​ As with any language understanding task, the availability of resources is key factor in
the disambiguation of the word senses in corpora.
​ Early work on word sense disambiguation used machine readable dictionaries or
thesaurus as knowledge sources.
​ Two prominent sources were the Longman dictionary of contemporary English
(LDOCE) and Roget’s Thesaurus.
​ The biggest sense annotation corpus OntoNotes released through Linguistic Data
Consortium (LDC).
​ The Chinese annotation corpus is HowNet.

Systems:
Researchers have explored various system architectures to address the sense disambiguation
problem.
We can classify these systems into four main categories: i. rules based or knowledge
ii. Supervised iii.unsupervised iv. Semisupervised

Rule Based:
​ The first-generation of word sense disambiguation systems was primarily based on
dictionary sense definitions.
​ Much of this information is historical and cannot readily be translated and made
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

available for building systems today. But some of techniques and algorithms are still
available.
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

​ The simplest and oldest dictionary-based sense disambiguation algorithm was

introduced by Lesk.
The core of the algorithm is that the dictionary senses whose terms most closely overlap with
the terms in the context.

Another dictionary-based algorithm was suggested Yarowsky.

This study used Roget’s Thesaurus categories and classified unseen words into one of these
1042 categories based on a statistical analysis of 100 word concordances for each member of
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

each category.
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

The method consists of three steps, as shown in Fig below.

●​ The first step is a collection of contexts.
●​ The second step computes weights for each of the salient words.
●​ P(w|Rcat) is the probability of a word w occurring in the context of a Roget’s
Thesaurus category Rcat.
●​ P(w|Rcat) |Pr(w) , the probability of a word (w) appearing in the context of a Roget
category divided by its overall probability in the corpus.
●​ Finally, in third step, the unseen words in the test set are classified into the category
that has the maximum weight.

Supervised:
•​ The simpler form of word sense disambiguating systems the supervised approach,
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

which tends to transfer all the complexity to the machine learning machinery while
still requiring hand annotation tends to be superior to unsupervised and performs best
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

when tested on annotated data.

•​ These systems typically consist of a machine learning classifier trained on various
features extracted for words that have been manually disambiguated in a given
corpus and the application of the resulting models to disambiguating words in the
unseen test sets.
•​ A good feature of these systems is that the user can incorporate rules and knowledge
in the form of features.
Classifier:
Probably the most common and high performing classifiers are support vector machine (SVMs) and
maximum entropy classifiers.
Features: Here we discuss a more commonly found subset of features that have been useful in
supervised learning of word sense.
Lexical context: The feature comprises the words and lemma of words occurring in the entire
paragraph or a smaller window of usually five words.
Parts of speech: the feature comprises the surrounding the word that is being sense tagged. Bag of
words context: this feature comprises using an unordered set of words in the context window.
Local Collocations: Local collocations are an ordered sequence of phrases near the target word that
provide semantic context for disambiguation. Usually, a very small window of about three tokens on
each side of the target word, most often in contiguous pairs or triplets, are added as a list of features.
Syntactic relations: if the parse of the sentence containing the target word is available, then we can
use syntactic features.
Topic features: The board topic, or domain, of the article that word belongs to is also a good
indicator of what sense of the word might be most frequent.
SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

Unsupervised:
SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

Figure: Conceptual Density

Semi Supervised:
SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

(WSD) in Natural Language Processing (NLP), semi-supervised learning techniques can be

quite beneficial because labeled data for WSD is often limited and expensive to obtain. Here's
an overview of a semi-supervised learning algorithm for WSD:

Self-Training for WSD:

Self-training is a popular semi-supervised learning approach that can be adapted for WSD. In
self-training for WSD, you start with a small set of labeled examples and a larger set of
unlabeled examples. The process involves iterative steps:
 SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

Motivation and concept of Yorowsky algorithm

SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT
SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT
SRI KRISHNA INSTITUTE OF TECHNOLOGY
(Accredited by NAAC Approved by A.I.C.T.E. New Delhi, Recognized by Govt. of Karnataka & Affiliated to V.T U., Belagavi)
#57, Chimney Hills, Hesaraghatta Main Road, Chikkabanavara Post, Bengaluru- 560090
ARTIFICIAL AND MACHINE LEARNING DEPARTMENT

Figure: Yorowsky algorithm