INTRODUCTION TO NATURAL LANGUAGE PROCESSING

The following examples show how NLP has made an impact in various fields
Chatbots in healthcare
Chatbots like Molly, Eva, Ginger, Replika, Florence, and Izzy are widely used in
healthcare. These bots provide the following support by
• Providing end to end suggestions to patients from their diagnosis to treatment
options
• Managing medication
• Helping in emergency situations or with first aid
• Offering a solution for simpler medical issues
Chatbots for mental health support
Bots like Wysa and Woebot are designed in such a way that they can provide support
like a life coach. They are so good at asking right probing questions that can help the
user to share their emotions and feelings after a hard day.
Chatbots in banks
Banks like Bank of America, JP Morgan Chase and Wells Fargo have chatbots to
provide financial advice and digital support.
Basic tasks like balance inquiry, bank account details, loan queries, etc. can be handled
by these bots efficiently, so that the customer service representatives of the banks can
invest their time for handling complex issues where human expertise is crucial.
Chatbots for legal advice
Lawyers can use bots like DonotPay, LISA, Ross, and BillyBot to accelerate their work
and provide better client experiences.
NLP is also playing an increasing role in areas like Legal research by
• Finding relevant information to make legal decisions
• Finding the relevance of documents to information requests
• Contract review to check if a contract is complete
• Document automation to generate routine legal documents
• Questions and answer dialogs to provide tailored legal advice
NLP is used across different domains for Brand Sentiment Analysis, Recruitment, and
Optimizing call center operations.
Brand sentiment analysis
Brand sentiment analysis helps in understanding the emotional tone of consumers via
social posts in the public domain. Knowing the opinions and having a real-time view of
the customer’s pulse is a critical element of brand marketing.

Recruitment
Companies can use a semantic search to filter resumes that seem like a good fit for
open positions.

Call center operations

High volumes of consumer interaction create the need for a critical capability to
prioritize which tasks to act upon first. Using voice to text, NLP and machine learning
can quickly deliver insights to the most important customer inquiries.

Other NLP applications

In Smart keyboards like Swiftkey, the software automatically completes your sentences
by predicting the next word and corrects your spelling mistakes.

Applications like Grammarly can automatically correct your spelling and grammar and
assists you in writing better essays or emails.

AMBIGUITY
Lexical ambiguity
Take a look at the following sentences:
John bagged two silver medals.
Mary made a silver speech.
Roger’s worries had silvered his hair.
The word silver is used as a noun, an adjective, and a verb.
The word silver in isolation is mostly associated with the metal and considered as a
noun. However, in other sentences, the context gives the word silver different meanings
and also different parts of speech like adjectives and verbs. This ambiguity is called
lexical ambiguity.
Syntactic ambiguity
Take a look at the sentence given below
“Old men and women were taken to safe locations”
This sentence has a syntactic ambiguity where the scope of the adjective “old” needs to
be resolved.
In this sentence, we may not know if the adjective applies only to men or to both men
and women.
Semantic ambiguity
Semantic ambiguity refers to ambiguity in the meaning.
For example, the sentence
“Alice loves her mother and so does Jacob.”
The ambiguity here is, we may not know if Jacob loves his own mother or Alice’s mother.
Anaphoric ambiguity
In the below paragraph
“The horse ran up the hill. It was very steep. It soon got tired.”
In this paragraph, the pronoun ‘it’ is used to refer to the hill first and then to the horse. To
interpret this sentence, we need to have knowledge of the world and context. These
ambiguities are called anaphoric ambiguities.
Pragmatic Ambiguity
The hardest kind of ambiguity to resolve is the pragmatic ambiguity. This kind of
ambiguity arises from the inability to process the intention or sentiment or world belief.
For example, in the below conversation,
My wife said: "Please go to the store and buy a carton of milk and if they have eggs, get
six."
I came back with 6 cartons of milk
She said, "why did you buy six cartons of milk?"
I replied, "They had eggs"
As you can see here, the ambiguity is in understanding the intention of the speaker.
Having seen a few popular applications of NLP like speech recognition, machine
translation and sentiment analysis let us now take a look at the various techniques that
are commonly used in building these systems.
Speech to text conversion (Automatic Speech Recognition)
This task is required in systems that usually have a voice user interface. The primary
responsibility of systems that are built for this task is to accurately convert the audio
waves (speech) into text representations. Several speech recognition systems use a
combination of acoustic models that correlate audio waves to phonemes (how different
words sound) and language models that are statistical distributions over word
sequence used to provide context and distinguish between words/phrases that could
sound similar (like hear vs here).
Speech recognition is a broad area of study and research by itself and recent use of
machine learning techniques (especially deep learning) have contributed to more
reliable and accurate speech recognition systems
Handling/processing text
Text processing comprises several techniques including tokenization, stop word
removal, part of speech tagging, stemming, lemmatization, etc. Lets look at a few of
these techniques:
• Tokenization
In tokenization, the whole text is split into tokens. Sometimes we can assume
that a token is a word, however that is not always necessary. During the process
of tokenization, one may choose to also remove special characters appearing in
punctuation or otherwise. Each token then can be used for several tasks in NLP.
For example, if you were to create a translator you can construct a token based
translation (although this may not work as good as you want a translator to).

1. Example:
2. input - "He did not try to navigate after the first bold flight, for the reaction
had taken something out of his soul."
3. ouptut - ['He', 'did', 'not', 'try', 'to', 'navigate', 'after', 'the', 'first', 'bold', 'flight',
',', 'for', 'the', 'reaction', 'had', 'taken', 'something', 'out', 'of', 'his', 'soul', '.']

4.

• Stop word removal

Stop words are often considered to be words that frequently appear across the
entire corpus of text used for analysis. For example, in English the words 'the',
 'and', 'to', etc. are quite common. Stop word removal is not necessary fro all
applications. For example, we might remove stop words when we are clustering
similar documents but we might choose to retain them when we are doing
machine translation.

1. Example:
2. input - ['He', 'did', 'not', 'try', 'to', 'navigate', 'after', 'the', 'first', 'bold', 'flight', ',',
'for', 'the', 'reaction', 'had', 'taken', 'something', 'out', 'of', 'his', 'soul', '.']
3. output - ['try', 'navigate', 'first', 'bold', 'flight', ',', 'reaction', 'taken',

'something', 'soul', '.']

• Part of speech (POS) tagging

Part of speech tagging involves identifying the right form of the word (verb, noun,
etc.). Identifying the correct part of speech being used is important to several
NLP applications such as machine translation, text summarization,
understanding meaning and context in speech, sentiment analysis etc.
1. Example:
2. input - ['And', 'from', 'their', 'high', 'summits', ',', 'one', 'by', 'one', ',', 'drop',
'everlasting', 'dews', '.']
3. output - [('And', 'CC'),
4. ('from', 'IN'),
5. ('their', 'PRP$'),
6. ('high', 'JJ'),
7. ('summits', 'NNS'),
8. (',', ','),
9. ('one', 'CD'),
10. ('by', 'IN'),
11. ('one', 'CD'),
12. (',', ','),
13. ('drop', 'NN'),
14. ('everlasting', 'VBG'),
15. ('dews', 'NNS'),
 16. ('.', '.')]
17.
18. where,
19. JJ: Adjective
20. RB: Adverb
21. CC: Conjunction
22. DT: Determiner
23. MD: Modal Verb
24. NN: Noun
25. CD: Numeral
26. VR: Participle
27. RP: Particle
28. PO: Possessive Ending
29. IN: Preposition
30. PP: Pronoun
31. SY: Symbol
32. VB: Verb
33. Refer to the Penn Treebank tagset for a complete list of tags

34.

Several other steps such as standardizing the case of the text, removing URLs, numbers
or special characters, normalize the data using stemming/lemmatization, etc. are also
involved in the text analysis pipeline.

Problem Statement
Consider that you need to build a spam detection system using the various NLP
techniques that we have discussed so far. You are given the following blocks to design
the same. Choose the various blocks that you will use to do the same.
• Gather relevant data - Build a repository of emails marked as spam/non-spam by
users
 • Data pre-processing
o Tokenization
o Punctuation removal
o Stop word removal
o Normalize the case of the text
o POS tagging
o Stemming
o Lemmatization
• Feature engineering
o Count the number of links in text - Is spam mail more prone to contain
links than regular email?
o Count the number of capitalized words - Do spammers use more
capitalized words?
o Look for unusual words, spelling mistakes, etc. - Does spam often contain
unusual words (mix of alpha numeric characters) or words with a lot of
spelling errors, etc.?
• Machine Learning - This block will learn what makes an email spam/not-spam
based on the preceding blocks and classify new emails as spam/non-spam.

Problem Statement
Consider the example for ambiguity in POS tagging explained in the previous video:
1. Google is headquartered in California
2. Google the headquarters of Infosys
What would be your approach to tag the part-of-speech of the word 'Google' correctly?
Here are a few approaches that you could adopt - list out the pros and cons of these and
any other approaches that you can think of.
• Approach 1: Always give the word the POS that has been associated with it the
most in historic data. For example, if Google is used more as a Verb, then in both
the sentences we can assign its POS tag as VB
• Approach 2: Create a rule set for tagging words. First create a dictionary of
words and tags that can be used with them, then disambiguate the tag for a word
based on hand crafted rules aligned to the grammar of the language used.
 • Approach 3: Look at the words that precede or succeed the word. Build a
probabilistic model to estimate the tag of the center word based on the
preceding and succeeding word.

What is Deep Learning?

Deep learning is a class of machine learning algorithms based on artificial neural
networks. Several kinds of deep learning architectures are in use today for solving a
wide range of problems in areas such as computer vision, speech recognition, machine
translation, image and video analysis, reinforcement learning and game AI. Deep
learning architectures rely on stacking up several layers of neural networks where each
layer learns to transform the input into abstract representations.

The key advantage that deep learning techniques offer in comparison to other machine
learning techniques is their ability to automatically learn higher level features from the
data. For example, if we were to use a classical machine learning algorithm from an
application like spam detection, we would engineer several features like - length of
email, contains URLs, contains unusual words, etc. ; however when using deep learning
the layer of neural networks are able to learn such features by themselves requiring
lesser manual feature engineering. The caveat however is that deep learning
architectures are data and resource hungry and take a lot of time and resource while
training.

Let us now look at some of the key advancements that deep learning has brought into
the field of NLP
Some of the key advancements that were brought out by using deep learning
techniques for NLP are listed below
Distributed representations and word embedding
Distributed representations overcome limitation of manual feature representations. In
these, we represent words (or tokens) as vectors (sequence of numbers) that can be
used directly in machine learning for various subsequent tasks like sentiment analysis
or machine translation. These vectors represent syntactic and semantic information of
the tokens. A popular application of word embedding is in expressing semantic
relationships among words e.g. 'man' - 'woman' = 'king' - 'queen' (word analogies).
Commonly used word embedding models are word2vec and GloVe. Word embeddings
have been used for various tasks in NLP such as POS tagging, named entity recognition
and sentiment analysis.
Use of Convolutional Neural Networks for text
Convolutional neural networks are neural networks that contain convolutional layers in
them. These layer perform the convolution operation on the input signal. These layers
apply a feature function on the input signal and help in extracting higher level
abstractions which can effectively be used in various NLP tasks such as sentiment
analysis, machine translation and question answering, speech recognition, etc.
Use of Recurrent Neural Networks for NLP
Recurrent neural networks (RNNs) help build dynamic models that best capture
variations in sequential and time based data which is often the nature of natural
language. These networks by virtue of their structure have a memory component
associated with them which helps maintain context over a long sequence of
tokens/words. RNNs have been used in several NLP applications and a few notable
RNN architectures like BERT, ElMo, etc. have performed better than previous
benchmarks and achieved state of the art results.
The following table compiles the state of the art deep learning models for NLP based on
the task (as of July 2019)

Task Model Performance score

97.56 % per token

POS Tagging Deep Memory Network (DMN)
accuracy

Deep fully-connected NN with 94.30 unlabelled

Dependency Parsing
features including POS attachment score

Constituency seq2seq learning with LSTM +

93.5% F1 score
Parsing attention

Named Entity
Dilated CNN with CRF 90.54% F1 score
Recognition

Semantic role Bidirectional LSTM with highway

83.40% F1 score
labeling connections

Sentiment Tree LSTm with refined word

90.3% accuracy
classification embedding

Machine translation seq2seq with CNN 41.29 BLEU score

Let us now look at the capabilities of various software libraries and platforms available
for building NLP systems
Frameworks and libraries for NLP

Library/Framework Capabilities

Provides a suite of open source python modules, data sets, and

Natural Language tutorials. Some of the key features that NLTK provides out of the
Toolkit (Python) box include - tokenization, POS tagging, NER, sentiment
analysis, corpora (data sets)

Open source library for advanced NLP tasks written in Python

and Cython. Some of the key features include - tokenization,
spaCy
NER, POS tagging, sentiment analysis, dependency parsing,
word vectors for deep learning

Open source vector space and topic modelling toolkit

Gensim implemented in Python. Gensim is popular for its topic
modelling, word vectors and corpora

Open source NLP package built by the Stanford NLp group.

Originally built in Java and now also exposes a Python package.
Stanford NLP is typically used to build an NLP pipeline
Stanford NLP
comprising tasks such as tokenization, POS tagging, NER, word
vectors for deep learning, and pre trained models for more than
50 languages.

A natural language pipeline that supports massive multilingual

applications. Offers the following features in multiple
languages:
Language detection (196 Languages)
• Tokenization (165 Languages)
• Word Embeddings (137 Languages)
Polyglot
• Sentiment Analysis (136 Languages)
• Morphological analysis (135 Languages)
• Transliteration (69 Languages)
• Named Entity Recognition (40 Languages)
• Part of Speech Tagging (16 Languages)
Library/Framework Capabilities

One of the most popular open source machine learning library

scikit-learn in Python. Offers a simple to use API to build and test
supervised and unsupervised machine learning models

Popular deep learning libraries that can be used to train deep

neural networks for NLP tasks. Several open source code
TensorFlow, Keras,
repositories for common NLP tasks such as sentiment analysis,
PyTorch
machine translation, named entity recognition, word vectors
etc. have been built using these libraries

Platforms for NLP

In addition to several open source libraries, you can also take advantage of platforms
built by several companies like Google, Amazon and Microsoft. Here is a brief
description of the platforms that you can use:
Platforms/Services provided by Amazon over Amazon Web Services
• Amazon Comprehend: NLP service that sues machine learning to find insights in
text. You can perform language detection, named entity recognition, sentiment
analysis, POS tagging, and topic modelling using this service
• Amazon Lex: Service for building conversational interfaces using voice or text.
Provides automatic speech recognition and natural language understanding
capabilities
• Amazon Polly: Service that converts text into life like speech in 25+ languages
• Amazon Transcribe: Service with automatic speech recognition capabilities that
converts speech to text. Can transcribe speech in real-time as well.
• Amazon Translate: A neural machine translation service that delivers fast, high
quality and affordable machine translation.
• Amazon Textract: A service that automatically extracts text and data from
scanned documents
Platforms/Services provided by Google over Google Cloud Platform
• Google Cloud Natural Language: Perform syntax analysis, entity analysis,
custom entity extraction, sentiment analysis, content classification, and build
machine learning models on cloud
 • Google Cloud Translation: Dynamically translate between multiple languages.
• Google Cloud Speech-to-Text API: Performs speech recognition across 120
different languages
• Google Cloud Text-to-Speech API: Synthesize natural sounding speech in 32
different voices across multiple languages and variants
• DialogFlow: Enables the end-to-end development of conversational interfaces
(chatbots) across devices and platforms
Platforms/Services provided by Microsoft over Microsoft Azure
• Microsoft Cognitive Services - Speech Services: Automatic speech-to-text
transcription, Natural text-to-speech, and real time speech translation
• Microsoft Cognitive Services - Text Analytics: Named Entitry Recognition (NER),
Key Phrase Extraction, Sentiment Analysis
• Microsoft Cognitive Services - Translator Text: Automatic language detection and
text translation
• Microsoft Cognitive Services - QnA Maker: QnA extraction from text, Create
knowledge base from set of QnAs, Semantic matching for knowledge bases
• Microsoft Azure Bot Service: End-to-end development of intelligent, enterprise-
grade chatbots
• Microsoft Cognitive Services - Language Understanding (LUIS): Build natural
language understanding into your apps, bots and IoT devices

REFERENCES
Below are a few recommended books that you can refer to:
1. Speech and Language Processing authored by Dan Jurafsky and James H Martin
2. Natural Language Processing with Python authored by Steven Bird, Ewan Klein
and Edward Loper.
3. Foundations of Statistical Natural Language Processing authored by Christopher
Manning and Hinrich Schutze
You could also refer to the online documentation of the following libraries/frameworks:
1. NLTK (Natural Language Tool Kit)
2. spaCy
3. Stanford NLP
Several acclaimed NLP researchers/professionals publish papers and articles on NLP
and other related fields, some of them you could look up for are:
1. Ian Goodfellow
2. Yoshua Bengio
3. Sebastin Ruder
4. Jurgen Schmidhuber
5. Geoffrey Hinton
6. Yann LeCun