THE NEED OF NLP

Chapter 1 GENERIC NLP SYSTEM

Introduction LEVELS OF NLP
to Natural STAGES IN BUILDING A NATURAL
Language LANGUAGE PROCESSING SYSTEM
Processing CHALLENGES AND AMBIGUITIES IN NLP
DESIGN
Challenges and ambiguities in NLP
Design
Importance of NLP
The Need of NLP
 Humans communicate through some form of language either by text or speech.
 To make interactions between computers and humans, computers need to understand natural
languages used by humans.
 Natural language processing is all about making computers learn, understand, analyse, manipulate and
interpret natural(human) languages.
 NLP stands for Natural Language Processing, which is a part of Computer Science, Human language,
and Artificial Intelligence.
 Processing of Natural Language is required when you want an intelligent system like robot to perform
as per your instructions, when you want to hear decision from a dialogue based clinical expert system,
etc.
 The ability of machines to interpret human language is now at the core of many applications that we
use every day - chatbots, Email classification and spam filters, search engines, grammar checkers, voice
assistants, and social language translators.
 The input and output of an NLP system can be Speech or Written Text
Components of NLP
 NLU – Natural Language Understanding
NLG – Natural Language Generation

Which is hard NLU or NLG?

NLP Terminology
Phonology − It is study of organizing sound Example:
systematically. Morphology: The study of the
formation and internal structure of words. "cat" vs. "bat" – The difference in sounds /k/
and /b/ changes the meaning of the words.
Morpheme − It is primitive unit of meaning in
a language. Application: Speech-to-text systems process
phonemes to recognize spoken words.
Syntax: The study of the formation and
internal structure of sentences.
Semantics: The study of the meaning of
sentences.
NLP Terminology
Phonology − It is study of organizing sound •Examples of morphemes:
systematically. Morphology: The study of the • un- (prefix meaning "not")
formation and internal structure of words. • happy (root word)
Morpheme − It is primitive/smallest unit of • -ness (suffix forming a noun)
meaning in a language.
Example:
Syntax: The study of the formation and
•Word: unhappiness
internal structure of sentences.
• Morphemes: un- + happy + -ness
Semantics: The study of the meaning of
sentences.
NLP Terminology
Phonology − It is study of organizing sound Example:
systematically. Morphology: The study of the
formation and internal structure of words. •Correct: "She is reading a book."

Morpheme − It is primitive/smallest unit of •Incorrect: "Reading she a book is."

meaning in a language. Application: Grammar-checking tools analyze
Syntax: The study of the formation and syntax to detect errors.
internal structure of sentences.
Semantics: The study of the meaning of
sentences.
NLP Terminology
Phonology − It is study of organizing sound Example:
systematically. Morphology: The study of the
formation and internal structure of words. Sentence: "I’m feeling blue."
◦ Literal meaning: The person’s skin is blue.
Morpheme − It is primitive/smallest unit of ◦ Semantic meaning: The person is feeling sad.
meaning in a language.
Syntax: The study of the formation and
internal structure of sentences.
Semantics: The study of the meaning of
sentences.
NLP Terminology
Pragmatics − It deals with using and Example:
understanding sentences in different situations •Sentence: "It’s cold in here."
and how the interpretation of the sentence is • Possible meaning: A request to close the
affected. window.
Discourse − It deals with how the
immediately preceding sentence can affect the
interpretation of the next sentence.
World Knowledge − It includes the general
knowledge about the world.
NLP Terminology
Pragmatics − It deals with using and Example:
understanding sentences in different situations Sentence 1: "John went to the store."
and how the interpretation of the sentence is
affected. Sentence 2: "He bought some milk."
◦ Interpretation: "He" refers to John, connecting
Discourse − It deals with how the the two sentences.
immediately preceding sentence can affect the
interpretation of the next sentence.
World Knowledge − It includes the general
knowledge about the world.
NLP Terminology
Pragmatics − It deals with using and Example:
understanding sentences in different situations Sentence: "She used an umbrella."
and how the interpretation of the sentence is ◦ Interpretation: It’s likely raining, as umbrellas
affected. are typically used in rain.
Discourse − It deals with how the
immediately preceding sentence can affect the
interpretation of the next sentence.
World Knowledge − It includes the general
knowledge about the world.
 Stages of NLP
Lexical
Analysis

Syntactic
Analysis

Semantic
Analysis

Discourse
Analysis

Pragmatic
Analysis
Lexical Analysis
 Breaking/Deciphering text into Lexicons / lexemes e.g. paragraph, sentence, phrases, word etc
Lexemes are split into morphemes. There are 2 types of morphemes:
• Free morphemes function independently as words (like “cow” and “house”).

• Bound morphemes make up larger words. The word “unimaginable” contains the
morphemes “un-” (a bound morpheme signifying a negative context), “imagine”
(the free morpheme root of the whole word), and “-able” (a bound morpheme
denoting the root morpheme’s ability to end).
“Wherever Ram went misfortune dogged him”.
“The chair emphasised the need for adult education”.

‘one sense per discourse assumption’

“Watch what you want, when you want”.

 Syntactic Analysis
Syntax describes how a language’s words and phrases arrange to form
sentences. Syntactic analysis checks word arrangements for proper grammar.

For example, “Ramesh wrote the paper” passes a syntactic analysis check
because it’s grammatically correct.
Conversely, a syntactic analysis categorizes a sentence like “Ramesh do
jumps” as syntactically incorrect.
Challenges in syntactic Parsing
Structural Ambiguity – Identify the ambiguities in below sentences
 “The old men and women were taken to safe locations”.
 “I saw the boy with a Telescope”
Semantic Analysis
Semantic analysis is concerned with the meaning representation, word sense disambiguation
and Entity Recognition (as we saw structural analysis limitations).

It mainly focuses on the literal meaning of words, phrases, and sentences.

The semantic analyzer disregards sentence such as “hot ice-cream”.

Discourse Analysis
Discourse integration analyzes prior words and sentences to understand the
meaning of ambiguous language. Cohesion and Coherence, Contextual
Understanding.
•Text:
"Alice went to the library. She borrowed a book.
•"Problem: The pronoun "She" needs to be resolved to "Alice".
•Discourse Analysis: Identifies that "She" refers to "Alice" based on context.
•“John go to school. It is open today. Should you bunk, father will be very angry”.
Pragmatic Analysis
Pragmatism describes the interpretation (intent) of language’s intended
meaning.
Pragmatic analysis attempts to derive the intended—not literal—meaning of
language.

“Mumbai speaks to all its people.”

“Boy go upstairs and see if my sandals are under the divan. Do not be late, I
just have
15 minutes to catch the train”.
A pragmatic analysis deduces that this sentence is a metaphor for how
people emotionally connect with places.
Generic NLP system
Challenges and Ambiguities in
NLP Design
Ambiguity
Contextual Understanding
Processing Multilingual Content
Ethical Concerns and Biases in NLP Models
Scalability and Computational Requirements
Real-Time Processing and Responsiveness
Data Quality and Availability
 Case Study – Building Health
Chatbot
Text Acquisition (Data Collection)
Text Preprocessing –
•Apply tokenization, stopword removal, stemming, and lemmatization.
•Handle domain-specific preprocessing (e.g., standardizing "bp" to "blood pressure").
•Handle special cases like abbreviations, dates, and numeric values.

Generate Embedding - (Word2Vec, GloVe).

Language Modeling - Building a unigram and bigram model from the dataset.
Named Entity Recognition (NER)
• Extract important entities such as diseases, medication names, and symptoms
Text Classification
• Categorize user queries into predefined classes. Create categories like "Appointment Booking," "Symptom Inquiry," and
"Medication Reminder.“

Evaluate
Spelling Correction: Edit
Distance
 The minimum edit distance between two strings Is the minimum number of editing operations
 Insertion

 Deletion

 Substitution
Example

Edit distance from ‘intention’ to ‘execution’

Example – Levenshtein Distance

Edit distance from ‘intention’ to ‘execution’

If each operation has a cost of 1 (Levenshtein)

Distance between these is 5
If substitution costs 2 (alternate version)
I Distance between these is 8
Dynamic Programming
Approach
Solution