Unit # 1

Concepts &
Challenges of NLP
 Introduction to NLP
• NLP is a broader field that encompasses the entire process of manipulating and
understanding natural language. This includes a wide range of tasks and methods
, techniques that allow computers to read,
decipher, understand, and make sense of
human language in a valuable way. Managing Human
• NLP’s function is to translate structured Computer Interaction
and unstructured text; It involves both
interpreting from Structured to
Unstructured or vice versa. Machine Perception
• NLP is an interdisciplinary field of
artificial intelligence and linguistics
that bridges this gap between Natural Language
computers and natural languages. Understanding (NLC)
• It covers everything from
speech recognition to
text analysis, Natural Language
Natural Generation (NLG)
language generation
and translation. Language
Processing
• Specifically NLP involves Natural Language
Classification
 Issues & Challenges in NLP
Natural Language Processing involves a variety of challenges due to the inherent
complexity of human language which it has to deal with.

Lack of Structured Data:

Natural languages are not structured like programming languages, which makes
processing them more difficult. This includes dealing with incomplete sentences,
errors in grammar, and variations in formatting and punctuation.
Pt. showed sx of flu. 38.5C, 120/80. Advised rest, fluids, & paracetamol
BD. Follow-up in 3d.

Hi there, my order #12345 was supposed to arrive last Tues but no

show :(
Data Sparsity and Quality:
Obtaining high-quality, annotated data for training NLP models, especially for less
common languages, is a persistent issue.
Experience is like a comb that life gives you when you are bald.
The ball slipped from his hands like butter from hot paratha.
The third umpires should be changed as often as nappies and for the same
reason.
Troubles are like babies, the more you nurse them, the more they grow.
We really dug deep today. Despite the ref's calls, we played our hearts out.
 Issues & Challenges in NLP (contd)
Domain-Specific Challenges:
Each domain, like healthcare, legal, or finance, has its own jargon, regulations, and
nuances that NLP systems must adapt to for effective performance.
I am short. I am long

‘COLD might refer to a common illness, but 'COLD' can also be an

acronym for chronic obstructive lung disease, which has a very different
meaning
Resource-Intensive Models:
Many state-of-the-art NLP models require significant computational resources,
making them inaccessible for some users or applications.

Machine Translation:
Translating text from one language to another while retaining the original meaning,
tone, and context is a significant challenge due to the complexity and variability of
languages.

I'm feeling blue

 Issues & Challenges in NLP (contd)
Sentiment Analysis:
Accurately gauging the sentiment or emotional tone of text is complex, especially
when considering cultural differences in expressing emotions.

Ambiguity:
Human language is often ambiguous, making it difficult for algorithms to determine the
correct meaning of words and sentences. This includes lexical ambiguity (where a
single word has multiple meanings), syntactic ambiguity (where the structure of a
sentence allows for multiple interpretations), and semantic ambiguity (where the
meaning of a sentence can be interpreted in different ways).

Context:
Understanding context is crucial in NLP. The meaning of words can change based on
context, and different cultures or social groups might use language in unique ways.
Algorithms must be able to understand and adapt to these nuances.
 Introduction to NLP (contd)
1. Human Computer Interaction
(HCI) is enabling of computers to understand, interpret, and generate human
language in a way that is both natural and effective for users.
In the context of HCI, NLP is used for:

• Voice recognition and response systems, which allow users to interact with
computers using spoken language.
• Chatbots and virtual assistants that can understand and respond to text or voice
queries.
• Automated translation services, which enable users to interact with systems
regardless of language barriers.
• Sentiment analysis to gauge user responses and tailor interactions accordingly.
• Text summarization to present information to users in a concise and user-friendly
manner.
 Introduction to NLP (contd)
2. Machine Perception
Machine perception in the context of Natural Language Processing (NLP) refers to the
ability of a machine or a computer system to interpret and understand human
language in a way that is meaningful.
• Speech Recognition is the ability to accurately convert spoken words into
written text. This is fundamental in understanding spoken language.
• Semantic Analysis Ideals with interpreting the meaning of sentences, phrases,
or entire texts. This includes understanding relationships between words and
phrases, and how these relationships contribute to meaning.
• Contextual Awareness is about understanding the context in which language is
used. This might include the situational context, cultural nuances, or specific
domain knowledge relevant to the text.
• Emotion and Sentiment Analysis is about recognizing and interpreting the
emotional tone or sentiment expressed in language, such as detecting whether
a statement is positive, negative, or neutral, and identifying specific emotions
like happiness, anger, or sarcasm.
 Introduction to NLP (contd)
3. Natural Language Generation

NLG refers to the process of generating natural language text or speech from a
machine representation system such as a database or a semantic representation like
a knowledge graph. The primary goal of NLG is to produce coherent, contextually
relevant, and human-like text or speech.
In more detail, NLG involves several steps:
Content Determination decides what information should be included in the
generated text.
Structuring Organises the selected information into a logical sequence.
Lexicalization Choose the specific words to express the content.
Aggregation Merge sentences or phrases for conciseness and fluency.
Referring Expression Generation: Generating appropriate nouns or pronouns to
ensure clarity and coherence.
Realization: Constructing the final text or speech output, ensuring it follows the
rules of grammar and syntax of the target language.
NLG is widely used in various applications, such as report generation, summarizing
data, creating conversational agents (like chatbots), and in assistive technologies
(like generating narrative from visual data for the visually impaired). It plays a crucial
role in making machines communicate with humans in a natural and intuitive way,
bridging the gap between complex data and human understanding.
 Introduction to NLP (contd)
4. Natural Language Understanding

Natural Language Understanding (NLU) is a subfield of Natural Language Processing

(NLP) that focuses on machine reading comprehension. It's the process by which a
computer program understands and interprets human language in a meaningful and
useful way. NLU goes beyond merely parsing text and involves understanding the
intentions and meanings behind the words.
Key aspects of Natural Language Understanding include:

Contextual Interpretation: NLU doesn't just analyze individual words in isolation. It

considers the context in which words are used. This means understanding the same
word can have different meanings in different situations.
Entity Recognition: Identifying and categorizing key information in text, such as
names of people, places, organizations, dates, etc.
Relationship Understanding: Discerning how entities relate to each other within the
text.
Semantic Analysis: Understanding the meaning or the semantics of sentences. This
includes understanding synonyms, antonyms, and the overall sentiment of the text.
Intent Recognition: In conversational systems, NLU is crucial for understanding the
user's intent – what action they want to perform or what information they are seeking.
Pragmatic Analysis: Understanding text within the context of the conversation or the
document as a whole, including understanding sarcasm, irony, or humor.
 Introduction to NLP (contd)
5. Natural Language Classifier
A Natural Language Classifier is a tool or system within the field of Natural
Language Processing (NLP) that categorizes or classifies text into predefined
categories or labels. It essentially interprets the intent behind a piece of text and
assigns it to a category that has been determined in advance.
The classifier works by examining the text and using various algorithms to predict the
most relevant category for that text. The basic way how it operates is as below.

Training involves training on a dataset where text examples are tagged with
appropriate labels. For instance, in a sentiment analysis classifier, examples of text
would be labeled as 'positive', 'negative', or 'neutral'. This training helps the system
learn how different text features are associated with each category.

Feature Extraction extracts features from the text, such as specific words, phrases,
syntax, or even the structure of the text. These features help the classifier in making
predictions.

Model Selection and Training is used for classification, such as Naive Bayes,
Support Vector Machines (SVM), or Neural Networks. The chosen model is trained
on the extracted features and their corresponding labels.
 Introduction to NLP (contd)
Classification i.e once trained, the classifier can take new, unseen text and predict
the most likely category for this text based on its learned understanding.

Evaluation and Optimization is evaluating using metrics like accuracy, precision,

recall, and F1 score. Based on these evaluations, adjustments might be made to
improve performance.
Natural Language Classifiers have a wide range of applications, such as sorting
emails into spam and non-spam categories, sentiment analysis on social media
posts, categorizing customer feedback, and more. They are crucial in automating
and enhancing the understanding of large volumes of text, making them invaluable
in areas like customer service, market analysis, and social media monitoring.
Thanks