Natural Language Understanding (NLU)

Understanding involves the following tasks −

 Mapping the given input in natural language into useful representations.

 Analyzing different aspects of the language.
Difficulties in NLU
NL has an extremely rich form and structure.
It is very ambiguous. There can be different levels of ambiguity −

 Lexical ambiguity − It is at very primitive level such as word-level.

 For example, treating the word “board” as noun or verb?
 Syntax Level ambiguity − A sentence can be parsed in different ways.
 For example, “He lifted the beetle with red cap.” − Did he use cap to lift the beetle or he lifted a
beetle that had red cap?
 Referential ambiguity − Referring to something using pronouns. For example, Rima went to
Gauri. She said, “I am tired.” − Exactly who is tired?
 One input can mean different meanings.
 Many inputs can mean the same thing
A subfield of NLP called natural language understanding (NLU) has begun to rise in popularity because
of its potential in cognitive and AI applications. NLU goes beyond the structural understanding of
language to interpret intent, resolve context and word ambiguity, and even generate well-formed human
language on its own. NLU algorithms must tackle the extremely complex problem of semantic
interpretation – that is, understanding the intended meaning of spoken or written language, with all the
subtleties, context and inferences that we humans are able to comprehend.

The evolution of NLP toward NLU has a lot of important implications for businesses and consumers
alike. Imagine the power of an algorithm that can understand the meaning and nuance of human language
in many contexts, from medicine to law to the classroom. As the volumes of unstructured information
continue to grow exponentially, we will benefit from computers’ tireless ability to help us make sense of
it all.

How does Natural Language Processing Works?

NLP entails applying algorithms to identify and extract the natural language rules such that the
unstructured language data is converted into a form that computers can understand.

When the text has been provided, the computer will utilize algorithms to extract meaning associated with
every sentence and collect the essential data from them.
Sometimes, the computer may fail to understand the meaning of a sentence well, leading to obscure results.

For example, a humorous incident occurred in the 1950s during the translation of some words between the
English and the Russian languages.

Here is the biblical sentence that required translation:

“The spirit is willing, but the flesh is weak.”

Here is the result when the sentence was translated to Russian and back to English:

“The vodka is good, but the meat is rotten.”

19.4.2.2 Morphological analysis
Morphological analysis (MA) is a method for identifying, structuring and investigating the total set of
possible relationships contained in a given multidimensional problem complex. MA allows small groups
of subject specialists to define, link, and internally evaluate the parameters of complex problem spaces,
creating a solution space and a flexible inference model [46, 47]. MA has been applied successfully in
strategic planning and decision support in various domains, such as governance of technological
development and modeling the Bioethics of drug redevelopment, which were reported more
comprehensively by Ritchey [46].
MA provides an “if-then” laboratory within which drivers, and certain conditions, can be assumed and a
range of associated solutions found, in order to test various inputs against possible outputs [48].
Generally, MA is intended to broaden the space of alternatives by systematic search for combinations of
attributes and systematically narrow them down through the results. The result of MA is called a
morphological field. Morphological fields describe the total problem complex. MA consists of the steps
given below [48].
Why Part-of-Speech tagging?
Part-of-Speech tagging in itself may not be the solution to any particular NLP problem. It is
however something that is done as a pre-requisite to simplify a lot of different problems. Let us
consider a few applications of POS tagging in various NLP tasks.

What are POS tags used for?

POS tags make it possible for automatic text processing tools to take into account which
part of speech each word is. This facilitates the use of linguistic criteria in addition to
statistics.

For languages where the same word can have different parts of speech, e.g. work in
English, POS tags are used to distinguish between the occurrences of the word when used
as a noun or verb.
POS tags are also used to search for examples of grammatical or lexical patterns without
specifying a concrete word, e.g. to find examples of any plural noun not preceded by an
article.

Uses of part-of-speech tagging algorithms vary depending on the field(s) where they are
applied. What is wonderful and mind-blowing on a research paper may sometimes be
unrealistic in a business environment. This is generally due to the scaling of the
computational resources required to perform an algorithm for potentially millions of daily
requests by following a paper that was centered around a very small sample of requests.

However, the main benefits of POS tagging would be to do some sort of information
retrieval, indexing, and classification. By tagging the various parts of a textual piece of
content you can very easily spot its noun phrases, verbal phrases, and therefore deduct the
main objects, subjects, as well as actions. By fine-tuning your application logic, you can then
clean up stop words and extract the essence of a piece of text — be it a tweet, a research
paper, a video transcribed, or the entire Internet in Google’s case.

Once you have tagged parts of speech, you may then decide to compute a dependency tree
to see what root words get the most attention, and what qualifiers and modifiers get used
to most with them, and why.

Part-of-speech tagging is only a first necessary step in understanding what a text is about.
Tagging is not very useful as is for business purposes, but it’s from a tagged text that you
can work your way up or down a more actionable business purpose (indexing, information
retrieval using noun phrases
PoS tagging allows you to do all sorts of useful things in NLP. For example, if you know
which words are adjectives, which are nouns, and which a re prepositions, you can find all
the technical terms (TTs) in a text using the TERMS algorithm of Justeson and Katz. Knowing
the verb in a sentence allows you to essentially find out what action(s) the sentence is
talking about, and many NLP systems concentrate on the verb when trying to figure out
sentential meaning. PoS tagging allows "chunking" of a sentence into the various parts that
fit into a parse tree, such as when you are trying to find the "functional" structure of a
sentence (i.e. Subject, Object and Verb - SVO structure.) If you know which words are
adverbs, you can essentially ignore them (unless you want the very fine detail of meaning).
Knowing which words/phrases are named entities (NEs) is also extremely useful. And so on!
In fact, PoS tagging is so useful that many systems don't even need to use a full parser, as
PoS tagging can get the job done without you having to construct a full parse tree (a
difficult and time-consuming process.) In fact, so useful is PoS tagging that you can even do
good NLP using a "crude" tagger with only 90 - 95% accuracy (the best PoS taggers
achieving up to 98% accuracy, by which I mean words correctly tagged.) So, PoS tagging is
VERY useful!