Unit - 1

INTRODUCTION
Basic concepts of Natural Language Processing, origins and evolution of NLP,
language and knowledge, issues and challenges in NLP, Types of ambiguities, Word and nonword
errors, Phases of Natural Language Processing.
• Phonology is a subfield of linguistics that focuses on the study of the sounds, or phonemes, used in
human language. It explores the ways in which speech sounds function and are organized within a
particular language or languages. (SOUND)
• Pragmatics is a branch of linguistics that studies the use of language in context and how context
influences the interpretation of meaning. It goes beyond the study of sentence structure and
grammar to examine how language is used in real-life situations, considering the social, cultural,
and situational factors that shape communication. (CONTEXT)
• Morphology is the branch of linguistics that deals with the study of the structure and formation of
words. It explores the internal structure of words, their meaningful units (morphemes), and the
rules governing the combination of these morphemes. (COMPONENTS)
• Syntax is the branch of linguistics that studies the rules governing the structure of sentences in a
language. It deals with how words are combined to form grammatically correct and meaningful
sentences. (STRUCTURE English - SOV)
• Semantics is the branch of linguistics concerned with the meaning of words, phrases, sentences,
and the relationships between them. It explores how words and expressions convey meaning in a
language and how context influences interpretation. (MEANING)
 "Yesterday, Sarah quickly read an intriguing book about space at the library."
1. Phonology:
2. Pragmatics: She enjoys reading and borrowed a book from the library
3. Morphology:
1. "Yesterday“, "quickly" are adverbs.
2. "Sarah" is a proper noun.
3. "read" is a base verb in past tense.
4. "intriguing" is an adjective
5. "book“, "space“, "library ", and "is" are nouns.
6. "about“, "at" is a preposition.
7. "the" is a definite article.
4. Syntax:
Syntax involves the correct arrangement of words to form grammatically correct sentences..
5. Semantics:
1. Semantics deals with the meaning of words and sentences. In our example:
1. "Yesterday" refers to the day before today.
2. "Sarah" is a specific person.
3. "quickly" indicates the speed of Sarah's reading.
4. "intriguing" describes the book as fascinating.
5. "book" refers to a written or printed work.
6. "space" refers to the vast, seemingly infinite expanse beyond Earth.
7. "library" is a place where books are kept and can be borrowed.
Basic concepts of Natural Language Processing
• Natural Language Processing (NLP) is a field of artificial intelligence that focuses on the
interaction between computers and human language. It involves the development of algorithms and
models to enable machines to understand, interpret, and generate human language. Here are some
basic concepts of Natural Language Processing:
• Tokenization:
• Definition: Tokenization is the process of breaking down a text into smaller units, such as
words or phrases, referred to as tokens.
• Purpose: It is a fundamental step in NLP to analyze and process textual data.
• Part-of-Speech Tagging (POS):
• Definition: POS tagging involves assigning parts of speech (like noun, verb, adjective, etc.) to
each word in a sentence.
• Purpose: Helps in understanding the grammatical structure and meaning of a sentence.
• Named Entity Recognition (NER):
• Definition: NER identifies and classifies entities (such as persons, organizations, locations) in
a text.
• Purpose: Useful for extracting structured information from unstructured text.
• Stemming and Lemmatization:
• Stemming: Reducing words to their root or base form (e.g., running → run).
• Lemmatization: Reducing words to their base or dictionary form (e.g., better → good).
• Purpose: Normalizing words to handle variations and improve analysis.
• Sentiment Analysis:
• Definition: Analyzing text to determine the sentiment expressed, such as positive, negative, or neutral.
• Purpose: Used to understand opinions and emotions expressed in textual data.
• Syntax and Parsing:
• Syntax: The arrangement of words and phrases to create well-formed sentences.
• Parsing: Analyzing the grammatical structure of a sentence.
• Purpose: Helps in understanding the relationship between words and constructing the meaning of a
sentence.
• Bag of Words (BoW) and Term Frequency-Inverse Document Frequency (TF-IDF):
• BoW: Represents text as an unordered set of words and their frequencies.
• TF-IDF: Weighs the importance of words based on their frequency in a document relative to their
frequency across all documents.
• Purpose: Used for text representation and information retrieval.
• Word Embeddings:
• Definition: Representing words as vectors in a continuous vector space.
• Purpose: Captures semantic relationships between words, facilitating machine understanding.
• Language Models:
• Definition: Statistical models that assign probabilities to sequences of words.
• Purpose: Used in tasks like speech recognition, machine translation, and text
generation.
• Machine Translation:
• Definition: Automatically translating text from one language to another.
• Purpose: Enables cross-language communication and information access.
• NLP is a vast field with ongoing research and applications in various
domains, including chatbots, virtual assistants, sentiment analysis, and
information extraction. Advances in deep learning, especially with models
like transformers, have significantly contributed to the progress in natural
language understanding and generation.
Origins and Evolution of NLP
The origins and evolution of Natural Language Processing (NLP) can be traced through several key milestones:
1. Early Foundations (1950s-1960s):
1. The roots of NLP date back to the mid-20th century when computer scientists and linguists began exploring the
possibility of machines understanding human language.
2. Notable figures during this period include Alan Turing, who proposed the Turing Test in 1950 as a criterion for
machine intelligence, and Claude Shannon, who applied information theory to language.
2. Machine Translation Era (1950s-1960s):
1. The Georgetown-IBM experiment in 1954 marked one of the earliest attempts at machine translation, focusing on
translating Russian sentences into English.
2. The Automatic Language Processing Advisory Committee (ALPAC) review in the 1960s, while critical of early
machine translation efforts, spurred interest and research in computational linguistics.
3. Rule-Based Systems (1960s-1980s):
1. Early NLP systems relied on rule-based approaches and handcrafted linguistic rules to process and understand
language.
2. Projects like SHRDLU (1970) demonstrated the ability to manipulate objects in a block world using natural language
commands, showcasing the potential of rule-based systems.
4. Statistical Methods (1990s):
1. In the 1990s, statistical approaches gained prominence, with researchers incorporating probabilistic models and
machine learning techniques.
2. IBM's Candide system (1992) utilized statistical methods for speech recognition and language modeling, marking a
shift toward data-driven approaches.
5. Corpus Linguistics and Machine Learning (2000s):
1. The availability of large text corpora enabled the application of machine learning techniques, such as Hidden Markov
Models and Maximum Entropy Models, for various NLP tasks.
2. The advent of the Internet and the creation of annotated datasets like Penn Treebank facilitated the training of more
sophisticated models.
6. Rise of Neural Networks (2010s-Present):
1. The introduction of deep learning, particularly recurrent neural networks (RNNs) and later transformer models, revolutionized NLP.
2. Models like Word2Vec, GloVe, and the breakthrough of transformer-based models like BERT, GPT, and T5 significantly improved the
performance of NLP tasks, achieving state-of-the-art results.
7. Transfer Learning and Pre-trained Models:
1. Transfer learning became a dominant paradigm, where pre-trained models on large datasets were fine-tuned for specific NLP tasks.
2. This approach, exemplified by models like BERT (2018) and GPT-3 (2020), demonstrated remarkable success across a wide range of
natural language understanding tasks.
8. Continued Research and Applications:
1. Ongoing research focuses on addressing challenges such as understanding context, handling ambiguity, and improving the ethical
considerations in NLP applications.
2. NLP applications have expanded to include virtual assistants, sentiment analysis, chatbots, machine translation, and more.
The field continues to evolve rapidly, driven by advancements in deep learning, increased computational power, and a
growing awareness of the societal impact of NLP technologies. Ethical considerations, fairness, and interpretability are
becoming integral parts of NLP research and development.
Language and Knowledge
Language and knowledge are intricately connected, playing fundamental roles in human cognition,
communication, and the acquisition of understanding. Here's a brief exploration of their relationship:
1. Expressing Knowledge through Language:
1. Language serves as a medium for expressing, sharing, and transmitting knowledge. It
provides a structured system of symbols, such as words and grammar, allowing individuals to
articulate their thoughts, experiences, and insights.
2. Acquiring Knowledge through Language:
1. The process of learning and acquiring knowledge is closely tied to language. From early
childhood, individuals use language to comprehend and internalize information presented in
various forms, such as spoken words, written text, or visual representations.
3. Cognitive Development:
1. Language plays a crucial role in cognitive development. As children learn language, they also
develop cognitive abilities, memory, problem-solving skills, and abstract thinking—all
essential components of acquiring knowledge.
4. Communication and Collaboration:
1. Effective communication, facilitated by language, is essential for collaborative knowledge-
building. Through conversations, discussions, and debates, individuals share perspectives,
challenge ideas, and collectively contribute to the creation of knowledge.
5. Symbolic Representation:
Language enables the symbolic representation of abstract concepts. Through words and
linguistic structures, individuals can convey complex ideas, theories, and philosophical concepts,
allowing for the representation of knowledge beyond immediate sensory experiences.
6. Preservation of Knowledge:
Writing, a form of language, has played a pivotal role in preserving knowledge across
generations. Written language allows for the documentation and dissemination of information,
contributing to the accumulation of collective knowledge over time.
7. Specialized and Technical Language:
Different domains and disciplines develop specialized language or jargon to express nuanced
concepts and precise details. This specialized language is a tool for experts to communicate
efficiently within their fields of knowledge.
8. Interconnectedness of Languages:
The diversity of languages worldwide reflects the richness of cultural knowledge. Interactions
between languages through translation and cross-cultural communication contribute to the
exchange and enrichment of global knowledge.
9. Computational Representation:
In the digital age, computational linguistics plays a role in representing, processing, and
analyzing knowledge. Natural Language Processing (NLP) techniques enable machines to
understand and generate human-like language, facilitating knowledge extraction from vast
datasets.
Issues and Challenges in NLP
1. Ambiguity:
1. Natural language is inherently ambiguous, and words or phrases can have multiple meanings.
Resolving ambiguity is challenging, especially in contexts with word sense ambiguity,
syntactic ambiguity, or semantic ambiguity.
2. Lack of Context Understanding:
1. Understanding context is crucial for accurate language comprehension. NLP systems often
struggle to capture the broader context of a conversation, leading to misinterpretations and
errors.
3. Named Entity Recognition (NER):
1. Identifying and classifying named entities (such as names of people, organizations, locations)
accurately is a challenging task. Variability in naming conventions, context-dependent
entities, and new entities pose difficulties for NER systems.
4. Coreference Resolution:
1. Resolving references to entities in a text (coreference resolution) is challenging. Determining
when pronouns or other referring expressions relate to the same entity requires a deep
understanding of context.
5. Lack of Common Sense Reasoning:
1. NLP models often struggle with common sense reasoning. Understanding implicit meanings,
cultural nuances, and drawing inferences that are apparent to humans remains a significant
challenge.
1. Data Quality and Bias:
1. The quality and bias in training data can impact the performance of NLP models. Biases
present in training data can be inadvertently learned and perpetuated by models, leading to
biased outputs.
2. Handling Negation and Negativity:
1. NLP systems may struggle to accurately interpret negations and expressions of negativity.
Distinguishing between positive and negative sentiments in complex sentences is a challenge.
3. Language Variability:
1. Natural languages exhibit variability in terms of dialects, slang, and cultural expressions. NLP
systems need to be robust enough to handle diverse linguistic variations.
4. Data Scarcity for Low-Resource Languages:
1. Many NLP models are trained on large datasets primarily in major languages. Low-resource
languages may lack sufficient training data, hindering the development of accurate models for
these languages.
5. Explainability and Interpretability:
1. Ensuring that NLP models are transparent, interpretable, and can provide explanations for
their predictions is an ongoing challenge, especially in critical applications where trust and
accountability are crucial.
Types of ambiguities
• Ambiguities in Natural Language Processing (NLP) refer to situations where the meaning of a
word, phrase, or sentence is unclear or has multiple interpretations. These ambiguities pose
challenges for NLP systems, as determining the correct interpretation is essential for accurate
language understanding. Here are some types of ambiguities commonly encountered in NLP:
1. Lexical Ambiguity:
1. Definition: Lexical ambiguity arises when a word has multiple meanings.
2. Example: The word "bank" can refer to a financial institution or the side of a river.
2. Syntactic Ambiguity:
1. Definition: Syntactic ambiguity occurs when a sentence can be parsed in multiple ways, leading to different
interpretations.
2. Example: "I saw the man with the telescope." Is the man holding the telescope, or did the speaker use the
telescope to see the man?
3. Semantic Ambiguity:
1. Definition: Semantic ambiguity involves multiple interpretations of the meaning of a phrase or sentence.
2. Example: "She can't bear children." Does this mean she is unable to have children, or does it imply she cannot
tolerate having them?
4. Anaphoric Ambiguity:
1. Definition: Anaphoric ambiguity arises when a pronoun or expression refers to something mentioned earlier in
the text, and it is unclear what it refers to.
2. Example: "John told Bob he was leaving." Who is leaving, John or Bob?
1. Temporal Ambiguity:
1. Definition: Temporal ambiguity involves uncertainty about the timing or sequence of events in a
sentence.
2. Example: "After she sang, she played the piano." Did she sing before or after playing the piano?
2. Referential Ambiguity:
1. Definition: Referential ambiguity occurs when it is unclear which entity or object a word or phrase
refers to.
2. Example: "The old man and the child were sitting on the bench. He gave her a candy." Who gave the
candy, the old man or the child?
3. Pragmatic Ambiguity:
1. Definition: Pragmatic ambiguity is related to the context and the speaker's intentions, making the
meaning dependent on the situation.
2. Example: "Could you pass the salt?" The meaning depends on whether the speaker wants the salt or is
asking someone else.
4. Quantifier Ambiguity:
1. Definition: Quantifier ambiguity arises when quantifiers like "some," "all," or "many" are imprecise or
have multiple interpretations.
2. Example: "Some cats are black." Does this mean at least one cat is black, or does it imply that not all
cats are black?
Addressing these ambiguities is crucial for NLP systems to achieve accurate and context-aware
language understanding. Advanced techniques, such as contextual embeddings, deep learning, and
attention mechanisms, are employed to mitigate the impact of these ambiguities in modern NLP
applications.
Word and nonword errors

• Word and nonword errors refer to mistakes or inaccuracies that can occur
during language processing tasks.
• Word Errors:
1. Definition: Word errors in NLP involve mistakes related to the recognition,
interpretation, or understanding of words within a given context.
2. Examples:
• Misinterpretation: If a system misinterprets the meaning of a word in a sentence, it may lead
to errors in understanding the overall context.
• Misspelling: Errors in recognizing or correcting misspelled words can affect the accuracy of
tasks such as text classification or sentiment analysis.
• Part-of-Speech Errors: Misidentifying the part of speech of a word (e.g., confusing a noun
with a verb) can impact syntactic analysis.
The cat is sleeping on the mat.
The cat is sleeping on the math.
• Nonword Errors:
• Definition: Nonword errors, also known as non-lexical errors, involve
mistakes related to sequences of characters or phonemes that do not
correspond to valid words in the language.
• Examples:
• Typos and Misspellings: Errors introduced due to typographical mistakes or
misspelled words that do not exist in the language.
• Phonetic Mistakes: Mispronunciations or misinterpretations of spoken words,
particularly in speech recognition systems.
• Neologisms: Recognition errors when dealing with newly coined or invented words
that are not present in the system's vocabulary.
I have a cat.
I have a dat.
Exercise
Original Recorded

The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the lazy log.

She enjoys playing the piano. She enjoys playing the piana.

The conference room is booked for the meeting. The conference room is booked for the meetting.

Artificial intelligence is advancing rapidly. Artificial intelligence is advancing rabbitly.

The scientist conducted a comprehensive analysis. The scientist conducted a comprehensive analogy.

The software update improves system performance. The software update improves system performence.
Mitigation Strategies:
• Spell Checking: Employing spell-checking algorithms can help identify and correct word
errors, especially in applications like text processing and document editing.
• Phonetic Algorithms: For speech recognition systems, using phonetic algorithms can
assist in handling nonword errors by mapping phonetic representations to valid words.
• Contextual Embeddings: Utilizing contextual embeddings, such as those generated by
pre-trained language models like BERT or GPT, can enhance the understanding of words
in context, reducing word errors.
• Domain-Specific Dictionaries: Customizing dictionaries or vocabularies for specific
domains can improve the accuracy of word recognition, addressing both word and
nonword errors.
• Addressing word and nonword errors is crucial for enhancing the overall performance and
reliability of NLP systems, especially in applications that require precise language
understanding and interpretation.
Phases of Natural Language Processing
• Natural Language Processing (NLP) involves several phases or stages to process and understand
human language. Here are the key phases of Natural Language Processing:
1. Lexical Analysis (Tokenization):
1. Definition: The process of breaking down a text into words or tokens.
2. Objective: To create a list of meaningful units (words) for further analysis.
2. Morphological Analysis:
1. Definition: Analyzing the structure and forms of words to understand their meaning.
2. Objective: Identification of word stems, prefixes, suffixes, and grammatical components.
3. Syntactic Analysis (Parsing):
1. Definition: Parsing involves analyzing the grammatical structure of sentences to understand the relationships
between words.
2. Objective: Building a syntactic tree to represent the hierarchical structure of a sentence.
4. Semantic Analysis:
1. Definition: Analyzing the meaning of words and sentences in context.
2. Objective: Understanding the intended meaning of the text, considering word sense disambiguation and
context.
1. Discourse Integration:
1. Definition: Integrating sentences or phrases to understand the discourse or larger context.
2. Objective: Recognizing relationships between sentences and paragraphs for coherent interpretation.
2. Pragmatic Analysis:
1. Definition: Analyzing the intended meaning based on the context of language use.
2. Objective: Considering the speaker's or writer's intentions and the impact of context on interpretation.
3. Named Entity Recognition (NER):
1. Definition: Identifying and classifying entities such as names of people, organizations, locations, and more.
2. Objective: Extracting relevant information from the text for further analysis.
4. Coreference Resolution:
1. Definition: Resolving references to entities mentioned in the text.
2. Objective: Determining which words or phrases refer to the same entities for coherent understanding.
5. Sentiment Analysis:
1. Definition: Analyzing the sentiment expressed in a text (positive, negative, neutral).
2. Objective: Understanding the emotions or opinions conveyed in the language.
6. Machine Translation:
1. Definition: Translating text from one language to another using automated algorithms.
2. Objective: Facilitating communication and understanding across different languages.
7. Text-to-Speech and Speech-to-Text Conversion:
1. Definition: Converting written text into spoken words or vice versa.
2. Objective: Enabling communication between humans and machines through speech.
These phases collectively contribute to the comprehensive processing of natural language, enabling machines to
understand, interpret, and generate human-like text. Advanced techniques, including machine learning and deep
learning, play a crucial role in enhancing the effectiveness of each phase in NLP applications.
 Unit - 2
KEY COMPONENTS
Basics of morphological analysis, syntactic analysis, semantic analysis, and
pragmatic analysis. Data Pre-Processing. Text tokenization. Part of Speech Tagging (POST). POS
Taggers. Case study of parsers of NLP systems: ELIZA, LUNAR.
Basics of morphological analysis
• Morphological analysis is a fundamental concept in linguistics and natural language processing.

• It involves the study of the structure and formation of words.

• It focuses on understanding how words are constructed from smaller units called morphemes,
which are the smallest meaningful units of language.

• Morpheme: A morpheme is the smallest unit of meaning in a language. It can be a word or a part
of a word that carries meaning. Morphemes can be classified into two types:

• Free morpheme: A morpheme that can stand alone as a word, such as "book", "run", or "happy".

• Bound morpheme: A morpheme that cannot stand alone and must be attached to a free morpheme
to convey meaning, such as prefixes (e.g., "un-", "pre-"), suffixes (e.g., "-ing", "-ly"), or roots (e.g.,
"anti-", "bio-").
• Word Formation: Morphological analysis examines how words are formed through various processes, including:

• Affixation: Adding prefixes or suffixes to a base word, such as "unhappiness" (prefix "un-" + base word "happy" +
suffix "-ness").

• Compounding: Combining two or more words to form a new word, such as "bookshelf" (book + shelf) or
"software" (soft + ware).

• Derivation: Creating a new word by adding affixes to change the grammatical category or meaning of the base
word, such as "happiness" (happy + -ness) or "worker" (work + -er).

• Inflection: Adding inflectional suffixes to indicate grammatical features like tense, number, or gender, such as
"walks" (walk + -s) or "cats" (cat + -s).
Morphological Analysis
• Tokenization: In NLP, tokenization involves breaking text into individual words or tokens.

• Morphological rules help in identifying word boundaries.

• For instance, in the sentence "I love running", tokenization would separate "running" from the rest
of the sentence as a distinct token.

• Stemming: Stemming aims to reduce words to their base or root form by removing affixes. For
example, "running" -> "run".

• Lemmatization: This process involves determining the dictionary form or lemma of a word by
considering its morphological properties and part of speech. For instance, the lemma of "running"
is "run".
Morphological analysis
• Morpheme Representation:
• Morphemes are the smallest units of meaning in a language. They can be roots, prefixes, suffixes, or
inflections.
• For example, in the word "unhappiness":
• "un-" is a prefix meaning "not"
• "happi" is the root or stem
• "-ness" is a suffix indicating a state or quality

• Feature Extraction:
• Feature extraction involves identifying and representing morphological features of words.
• Features can include:
• Presence of prefixes, suffixes, or affixes: e.g., "un-" in "unhappiness"
• Stem or root of the word: e.g., "happi" in "unhappiness"
• Length of the word: e.g., the number of characters in "unhappiness"
• Frequency of specific morphemes: e.g., how often the prefix "un-" appears in a text corpus
• Morphological Rules:
• Morphological rules describe patterns of word formation in a language.
• For example:

• Words ending in "-ing" are often gerunds or present participles: e.g., "running", "walking"

• Words starting with "un-" are usually negations: e.g., "unhappy", "unlike"

• Tokenization and Stemming:

• Tokenization involves breaking text into tokens or words. In morphological analysis, it's essential to handle compound words and hyphenated terms
correctly.

• Stemming reduces words to their base or root form. It aims to remove inflections and variations to simplify analysis.

• For example:

• Tokenizing the sentence "I am running" results in ["I", "am", "running"]

• Stemming the word "running" yields its base form "run"

• Lemmatization:
• Lemmatization is similar to stemming but considers the word's part of speech and context to
determine its base or dictionary form (lemma).
• For example:
• Lemmatizing the word "running" as a verb results in "run"
• Lemmatizing the word "better" as an adjective gives "good"
Example
import nltk
nltk.download('wordnet')
from nltk.corpus import wordnet

word = "running"

morphemes = nltk.word_tokenize(word)

for morpheme in morphemes:

synsets = wordnet.synsets(morpheme)
if synsets:
print(f"Morpheme: {morpheme}")
for synset in synsets:
print(f"Synset: {synset.name()}, Definition: {synset.definition()}")
print()
else:
print(f"No morphological analysis found for morpheme: {morpheme}\n")
 Example
import nltk
from nltk.corpus import wordnet

sentence = "The quick brown fox jumps over the lazy dog"
words = nltk.word_tokenize(sentence)

for word in words:

synsets = wordnet.synsets(word)
if synsets:
print(f"Word: {word}")
for synset in synsets:
print(f"Synset: {synset.name()}, Definition:
{synset.definition()}")
print()
else:
print(f"No morphological analysis found for word: {word}\n")
Exercise
import requests
from bs4 import BeautifulSoup
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import wordnet

url = 'https://wealthygorilla.com/best-short-moral-stories/'

response = requests.get(url)
html_content = response.text

soup = BeautifulSoup(html_content, 'html.parser')

# Extract text from the parsed HTML

text = soup.get_text()
tokens = word_tokenize(text)

for word in tokens:

synsets = wordnet.synsets(word)
if synsets:
print(f"Word: {word}")
for synset in synsets:
print(f"Synset: {synset.name()}, Definition: {synset.definition()}")
print()
else:
print(f"No morphological analysis found for word: {word}\n")
Syntactic analysis
• Syntactic analysis, also known as parsing, is a crucial step in natural language processing
(NLP) that involves analyzing the grammatical structure of sentences to understand their
syntax or structure.

• This process is essential for tasks like part-of-speech tagging, named entity recognition,
and sentiment analysis.

• Syntactic analysis helps computers understand the relationships between words in a

sentence, which is necessary for accurately interpreting the meaning of text.
• Tokenization: Before performing syntactic analysis, the text is typically tokenized, which involves splitting
it into individual words or tokens. Each token represents a distinct unit of meaning, such as words,
punctuation marks, or numbers.
• Part-of-Speech (POS) Tagging: POS tagging is the process of assigning a grammatical category or tag to
each word in a sentence, such as noun, verb, adjective, etc. This step helps identify the syntactic role of each
word in the sentence, which is crucial for understanding its structure.
• Phrase Structure Parsing: In phrase structure parsing, the syntactic structure of a sentence is represented
as a hierarchical tree structure called a parse tree or syntax tree. This tree illustrates the relationships
between words and phrases in the sentence, showing how they combine to form larger linguistic units. The
most common formalism for parse trees is context-free grammars (CFGs), which describe the syntax of a
language using production rules.
• Dependency Parsing: Dependency parsing focuses on identifying the relationships between words in a
sentence in terms of dependencies. A dependency represents a grammatical relationship between a head
word and its dependents. Dependency parsing produces a directed graph called a dependency tree, where
each word is a node, and the dependencies between words are represented as directed edges.
• Semantic Role Labeling (SRL): Semantic role labeling is a task that involves identifying the semantic roles
of words or phrases in a sentence, such as agent, patient, theme, etc. This step helps extract the underlying
meaning of the sentence by identifying who is doing what to whom.
• Syntactic Ambiguity Resolution: Syntactic analysis also involves resolving syntactic ambiguities that arise
when a sentence can be parsed in multiple ways. Ambiguity resolution techniques help determine the most
likely interpretation of a sentence based on context, grammar rules, and semantic constraints.
 Example
import nltk

nltk.download('brown')

from nltk.corpus import brown

sentences = brown.sents()

pos_tagged_sentences = [nltk.pos_tag(sentence) for sentence in sentences]

for sentence_pos_tags in pos_tagged_sentences[:3]:

print(sentence_pos_tags)
POS TAGS
1.CC: Coordinating 14.NNP: Proper noun, 29.VBG: Verb, gerund or
conjunction singular present participle
2.CD: Cardinal number 15.NNPS: Proper noun, plural 30.VBN: Verb, past participle
3.DT: Determiner 16.PDT: Predeterminer 31.VBP: Verb, non-3rd person
4.EX: Existential there 17.POS: Possessive ending singular present
5.FW: Foreign word 18.PRP: Personal pronoun 32.VBZ: Verb, 3rd person
singular present
6.IN: Preposition or 19.PRP$: Possessive pronoun 33.WDT: Wh-determiner
subordinating conjunction 20.RB: Adverb
7.JJ: Adjective 34.WP: Wh-pronoun
21.RBR: Adverb, comparative 35.WP$: Possessive wh-
8.JJR: Adjective, comparative 22.RBS: Adverb, superlative pronoun
9.JJS: Adjective, superlative 23.RP: Particle 36.WRB: Wh-adverb
10.LS: List item marker 24.SYM: Symbol
11.MD: Modal 25.TO: to
12.NN: Noun, singular or 26.UH: Interjection
mass
13.NNS: Noun, plural 27.VB: Verb, base form
28.VBD: Verb, past tense
Example
1. CC (Coordinating conjunction): 9. JJS (Adjective, superlative):
Jack and Jill went up the hill. He is the tallest person in the room.
2. CD (Cardinal number): 10.LS (List item marker):
There are 10 apples in the basket. 1. First item 2. Second item
3. DT (Determiner): 11.MD (Modal):
The cat is sleeping. He should study for the exam.
4. EX (Existential there): 12.NN (Noun, singular or mass):
There is a problem with the computer. The cat is on the mat.
5. FW (Foreign word): 13.NNS (Noun, plural):
C'est la vie. The cats are on the mats.
6. IN (Preposition or subordinating conjunction): 14.NNP (Proper noun, singular):
She went to the store. Paris is the capital of France.
7. JJ (Adjective): 15.NNPS (Proper noun, plural):
The sky is blue. The Smiths are coming over for dinner.
8. JJR (Adjective, comparative): 16. PDT (Predeterminer):
This book is longer than that one. Example: "All the apples are ripe."
Example
17. POS (Possessive ending): The temperature is 20°C.
Mary's book is on the table. 25.TO (to):
18.PRP (Personal pronoun): He went to the store.
She is going to the store. 26.UH (Interjection):
19.PRP$ (Possessive pronoun): Wow! That was amazing.
That is his car. 27.VB (Verb, base form):
20.RB (Adverb): She can swim.
She ran quickly. 28.VBD (Verb, past tense):
21.RBR (Adverb, comparative): He walked to the park.
He runs faster than her. 29.VBG (Verb, gerund or present participle):
22.RBS (Adverb, superlative): They are swimming in the pool.
He speaks the loudest. 30.VBN (Verb, past participle):
23.RP (Particle): The cake was eaten.
The dog ran off.
24.SYM (Symbol):
Example
31. VBP (Verb, non-3rd person singular present):
I walk to work every day.
32.VBZ (Verb, 3rd person singular present):
She walks to work every day.
33.WDT (Wh-determiner):
Which book do you want?
34.WP (Wh-pronoun):
Who is at the door?
35.WP$ (Possessive wh-pronoun):
Whose book is this?
36.WRB (Wh-adverb):
Where are you going?
Semantic Analysis
• Semantic analysis, also known as semantic parsing or semantic
understanding
• It focuses on extracting the meaning or semantics from natural
language text.
• It aims to understand the meaning of words, phrases, sentences, or
entire documents in order to enable machines to comprehend human
language more like humans do.
1.Word Sense Disambiguation (WSD): It deals with identifying the correct sense of a word with
multiple meanings based on the context in which it appears. For example, distinguishing between
"bank" as a financial institution and "bank" as a river bank.
2.Named Entity Recognition (NER): It involves identifying and classifying named entities (such as
persons, organizations, locations, dates, etc.) mentioned in text. NER is crucial for understanding the
entities mentioned in documents and their relationships.
3.Semantic Role Labeling (SRL): It aims to identify the predicate-argument structure of a sentence
by assigning semantic roles such as agent, patient, instrument, etc., to each constituent. This helps in
understanding the relationships between different parts of a sentence.
4.Semantic Parsing: It involves mapping natural language utterances into formal representations of
meaning, such as logical forms or semantic graphs. Semantic parsers analyze the syntax and
semantics of sentences to generate structured representations that can be used for further processing
or inference.
5.Sentiment Analysis: While not strictly semantic analysis, sentiment analysis involves determining
the sentiment or opinion expressed in a piece of text. This could be positive, negative, or neutral
sentiment, and it provides insights into the subjective meaning of the text.
6.Semantic Similarity: It involves measuring the similarity between words, phrases, sentences, or
documents based on their semantic content. Techniques such as word embeddings or semantic
vectors are often used to represent the meaning of text and compute similarity scores.
7.Textual Entailment Recognition: It deals with determining whether one piece of text (the
hypothesis) can be inferred or entailed from another piece of text (the premise). This task is
fundamental for understanding logical relationships and implications between textual statements.
Word Sense Disambiguation (WSD):
Example: "The bank was crowded."
WSD helps determine the correct sense based on the context.
If the context is about people standing in line, then "bank" likely refers to a financial institution.
Named Entity Recognition (NER):
Example: "Apple Inc. is headquartered in Cupertino, California."
In this sentence, "Apple Inc." is an organization, and "Cupertino, California" is a location. NER identifies
and classifies these named entities mentioned in the text.
Semantic Role Labeling (SRL):
Example: "The cat chased the mouse."
In this sentence, "cat" is the agent performing the action (chasing), and "mouse" is the patient receiving
the action. SRL assigns semantic roles to each constituent to understand the relationships between them.
Semantic Parsing:
Example: Text: "Find Italian restaurants in New York City."
Logical Form: Find(Restaurant(Type=Italian), Location=New York City)
Semantic parsing converts the natural language query into a structured representation that a machine can
understand and process. In this example, the query is parsed into a logical form specifying the type of
restaurant (Italian) and the location (New York City) to search for.
Sentiment Analysis:
Example: "The movie was fantastic! I loved every minute of it."
Sentiment analysis determines the sentiment expressed in the text.
In this case, the sentiment is positive, indicating that the speaker enjoyed the movie.
Semantic Similarity:
Example: "The cat is on the mat." vs. "The feline is on the rug."
Semantic similarity measures the likeness between two pieces of text. In this example, even though
different words are used ("cat" vs. "feline", "mat" vs. "rug"), the meaning remains similar, indicating a high
semantic similarity.
Textual Entailment Recognition:
Example:
Premise: "The sun rises in the east."
Hypothesis: "The sun sets in the west."
Textual entailment recognition determines whether the hypothesis can be inferred from the premise.
In this case, the premise entails the hypothesis because if the sun rises in the east, it must also set in the
west.
I'm craving Italian food. Is there a good Italian restaurant near Central Park in New York City?

Word Sense Disambiguation (WSD):

In the sentence, "Italian" could refer to the cuisine type or nationality.
WSD determines that "Italian" here refers to the cuisine type based on the context of "craving Italian food."
Named Entity Recognition (NER):
NER identifies named entities mentioned in the text. In this case, "Central Park" and "New York City" are
recognized as locations.
Semantic Role Labeling (SRL):
SRL assigns semantic roles to each constituent. For example:
"I" is the experiencer or agent expressing desire.
"Italian food" is the theme or object of desire.
"Central Park" is the location where the user wants to find a restaurant.
Semantic Parsing:
The natural language query is parsed into a structured representation that the system can understand. It
could be represented as:
Query: Find(Restaurant(Type=Italian), Location=Central Park, City=New York City)
Sentiment Analysis:
Sentiment analysis determines the sentiment expressed in the query. Here, the sentiment is positive (the
user is expressing desire), indicating a positive sentiment towards Italian food.
Semantic Similarity:
The system may compare the user query with the restaurant database to find similar restaurants. It
considers semantic similarity to match user preferences. For example, if a restaurant serves Italian cuisine
and is located near Central Park, it would be considered similar to what the user is looking for.
Textual Entailment Recognition:
The system may need to infer additional information from the user query. For example:
Premise: "User is looking for an Italian restaurant near Central Park."
Hypothesis: "There exists a restaurant serving Italian cuisine near Central Park."
The premise logically entails the hypothesis, indicating that the user query implies the existence of
such a restaurant.
Word Sense Disambiguation (WSD):
from nltk.wsd import lesk
from nltk.tokenize import word_tokenize

sentence = "I went to the bank to deposit money."

word_to_disambiguate = "bank"

# Disambiguate the word "bank" in the given sentence

meaning = lesk(word_tokenize(sentence),
word_to_disambiguate)
print(meaning.definition())
•
Named Entity Recognition (NER)
from nltk import ne_chunk, word_tokenize, pos_tag
from nltk.chunk import tree2conlltags

sentence = "Apple Inc. is headquartered in Cupertino, California."

tokens = word_tokenize(sentence)
pos_tags = pos_tag(tokens)

ner_tags = ne_chunk(pos_tags)
for chunk in tree2conlltags(ner_tags):
if chunk[2] != 'O': # O represents non-entity words
print(chunk)
Sentiment Analysis

from nltk.sentiment import SentimentIntensityAnalyzer

nltk.download('vader_lexicon')
sentence = "I love this movie! It's fantastic."

# Initialize sentiment analyzer

sid = SentimentIntensityAnalyzer()

# Analyze sentiment
sentiment_scores = sid.polarity_scores(sentence)

print(sentiment_scores)
Semantic Role Labeling (SRL):
NLTK doesn't have built-in support for SRL, but you can use external libraries or models trained on SRL
datasets to perform this task.

Semantic Parsing:
NLTK doesn't have built-in support for semantic parsing, but you can build parsers using NLTK's grammar
and parsing tools, such as the Recursive Descent Parser or Chart Parser.

Semantic Similarity:
NLTK provides methods for computing semantic similarity between words using word
embeddings or other techniques. For example, you can use WordNet or pre-trained word
embeddings like Word2Vec or GloVe to compute similarity scores between words.

Textual Entailment Recognition:

NLTK provides resources for working with textual entailment tasks, such as datasets and
algorithms for recognizing textual entailment.
!pip install -U spacy
!python -m spacy download en_core_web_sm
import spacy

nlp = spacy.load("en_core_web_sm")

text = "Apple Inc. is considering buying a startup in the UK for $1 billion. The CEO, Tim Cook, is interested in expanding the company's presence
overseas."

doc = nlp(text)
named_entities = [(ent.text, ent.label_) for ent in doc.ents]
print("Named entities:", named_entities)

sentiment_scores = {'pos': 0, 'neg': 0, 'neutral': 0}

for sentence in doc.sents:
polarity = 0
for token in sentence:
polarity += token.sentiment
if polarity > 0:
sentiment_scores['pos'] += 1
elif polarity < 0:
sentiment_scores['neg'] += 1
else:
sentiment_scores['neutral'] += 1

print("Sentiment scores:", sentiment_scores)

Sentiment Analysis
• It focuses on understanding the intended meaning of utterances,
taking into account factors such as speaker intentions,
presuppositions, implicatures, and discourse coherence. Pragmatic
analysis plays a crucial role in NLP tasks such as sentiment analysis,
speech recognition, dialogue systems, and machine translation.
1. Speech Acts: Pragmatic analysis considers the illocutionary force of utterances, which refers to the intended
action or speech act performed by the speaker. Speech acts can include assertions, questions, commands,
promises, requests, and apologies. Understanding the intended speech act is essential for interpreting the
meaning of utterances correctly.
2. Presuppositions: Pragmatic analysis examines the presuppositions underlying utterances, which are implicit
assumptions or beliefs that speakers take for granted in their communication. Presuppositions can influence the
interpretation of utterances and contribute to the overall meaning conveyed.
3. Implicatures: Pragmatic analysis involves identifying implicatures, which are implied meanings that arise
from the context of the utterance rather than the literal meaning of the words. Grice's Cooperative Principle
and conversational maxims provide a framework for understanding implicatures, such as the maxim of
quantity, quality, relevance, and manner.
4. Discourse Coherence: Pragmatic analysis considers discourse coherence, which refers to the logical and
cohesive flow of information in a conversation or text. It involves understanding how individual utterances
relate to each other and contribute to the overall coherence and coherence of the discourse.
5. Contextual Factors: Pragmatic analysis takes into account contextual factors such as the speaker's identity,
social status, cultural background, and situational context. These factors influence language use and contribute
to the interpretation of meaning in communication.
1.Pragmatic Ambiguity: Pragmatic analysis addresses pragmatic
ambiguity, which occurs when an utterance can be interpreted in
multiple ways depending on the context and the speaker's intentions.
Resolving pragmatic ambiguity requires considering contextual cues
and inferring the intended meaning from the broader context.
2.Computational Pragmatics: In computational linguistics, pragmatic
analysis involves developing algorithms and models to automatically
analyze and understand pragmatic aspects of language in NLP
applications. This includes tasks such as speech act recognition,
presupposition detection, implicature generation, and discourse
parsing.
Example

import nltk
nltk.download('punkt')
nltk.download('words')
nltk.download('stopwords')
nltk.download('vader_lexicon')
nltk.download('maxent_ne_chunker')
nltk.download('averaged_perceptron_tagger')
from nltk.sentiment import SentimentIntensityAnalyzer
from nltk.text import Text

text = "I want to buy a book. Do you have any recommendations?"

tokens = nltk.word_tokenize(text)
sentences = nltk.sent_tokenize(text)
pos_tags = nltk.pos_tag(tokens)
•
Example
entities = nltk.ne_chunk(pos_tags)

nltk_text = Text(tokens)
book_concordance = nltk_text.concordance('book')
collocations = nltk_text.collocations()
sid = SentimentIntensityAnalyzer()
sentiment_scores = sid.polarity_scores(text)

print("Tokens:", tokens)
print("Sentences:", sentences)
print("POS Tags:", pos_tags)
print("Named Entities:", entities)
print("Concordance for 'book':", book_concordance)
print("Collocations:", collocations)
print("Sentiment scores:", sentiment_scores)
Example
import nltk
from nltk.sentiment import SentimentIntensityAnalyzer

conversation = [
("A: Could you pass me the salt?", "B: Sure, here you go."),
("B: I'm sorry, I can't make it to the meeting tomorrow.", "A: That's okay, we'll catch you up on
what you missed."),
("A: Can you help me with this assignment?", "B: I'm really busy right now, maybe later."),
]

sid = SentimentIntensityAnalyzer()

for idx, (utterance_a, utterance_b) in enumerate(conversation, start=1):

print(f"Utterance {idx} (Speaker A):", utterance_a)
print(f"Utterance {idx} (Speaker B):", utterance_b)
Contd.
sentiment_a = sid.polarity_scores(utterance_a)
sentiment_b = sid.polarity_scores(utterance_b)

print(f"Sentiment (Speaker A):", sentiment_a)

print(f"Sentiment (Speaker B):", sentiment_b)

speech_act_a = "Request" if "?" in utterance_a else "Assertion"

speech_act_b = "Response" if "Sure" in utterance_b or "okay" in utterance_b else "Refusal"

print(f"Speech Act (Speaker A):", speech_act_a)

print(f"Speech Act (Speaker B):", speech_act_b)

implicature_a = "impolite" if sentiment_a['compound'] < -0.5 else "polite"

implicature_b = "busy" if sentiment_b['compound'] < -0.5 else "available"

print(f"Implicature (Speaker A):", implicature_a)

print(f"Implicature (Speaker B):", implicature_b)
print("-" * 50)
Data Pre-Processing
• Data pre-processing is a critical step in the data analysis and machine
learning pipeline that involves transforming raw data into a format
suitable for analysis or model training. It encompasses a variety of
tasks aimed at cleaning, formatting, and preparing the data to ensure
that it is accurate, consistent, and relevant for the intended analysis
or task. Here's an overview of the key steps involved in data pre-
processing:
1. Data Cleaning:
1. Handling Missing Values: Identify and handle missing or null values in the dataset, either by imputing them with a suitable value or by
removing them entirely.
2. Removing Duplicates: Detect and remove duplicate records from the dataset to avoid redundancy and ensure data integrity.
3. Outlier Detection and Treatment: Identify outliers or anomalous data points that deviate significantly from the rest of the dataset and decide
whether to remove them or adjust them appropriately.
2. Data Transformation:
1. Feature Scaling: Normalize or standardize numerical features to ensure that they have a similar scale and distribution, which can improve
the performance of certain machine learning algorithms.
2. Encoding Categorical Variables: Convert categorical variables into a numerical format suitable for analysis or model training, using
techniques such as one-hot encoding or label encoding.
3. Feature Engineering: Create new features or transform existing features to capture more relevant information and improve model
performance. This may involve techniques such as binning, log transformations, or polynomial features.
3. Data Reduction:
1. Dimensionality Reduction: Reduce the number of features in the dataset while preserving as much relevant information as possible, using
techniques such as principal component analysis (PCA) or feature selection methods.
2. Sampling: If the dataset is too large or imbalanced, consider sampling techniques such as random sampling or stratified sampling to create a
smaller, more manageable dataset without losing important information.
4. Data Integration:
1. Merge or Join Datasets: Combine multiple datasets into a single dataset to enrich the available information and facilitate analysis or model
training.
2. Handling Inconsistent Data Formats: Ensure that data from different sources are formatted consistently and merge them appropriately to
avoid discrepancies.
5. Data Normalization and Standardization:
1. Scale the data to a consistent range to ensure that the values have a similar scale and distribution, which can improve the performance of
certain algorithms, especially those sensitive to feature scales.
6. Data Splitting:
1. Divide the dataset into training, validation, and test sets to evaluate the performance of the model and prevent overfitting. The training set is
used to train the model, the validation set is used to tune hyperparameters and evaluate model performance during training, and the test set is
used to assess the model's performance on unseen data.
Example
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import PorterStemmer, WordNetLemmatizer

nltk.download('punkt')
nltk.download('stopwords')
nltk.download('averaged_perceptron_tagger')
nltk.download('maxent_ne_chunker')
nltk.download('words')

text = "The quick brown fox jumps over the lazy dog. John Doe works at Google, Inc. He is a
software engineer."

tokens = word_tokenize(text)
Contd.
stopwords_list = set(stopwords.words('english'))
filtered_tokens = [word for word in tokens if word.lower() not in stopwords_list]
porter = PorterStemmer()
stemmed_tokens = [porter.stem(word) for word in filtered_tokens]
lemmatizer = WordNetLemmatizer()
lemmatized_tokens = [lemmatizer.lemmatize(word) for word in filtered_tokens]
pos_tags = nltk.pos_tag(filtered_tokens)
entities = nltk.ne_chunk(pos_tags)
print("Original Text:", text)
print("Tokens:", tokens)
print("Filtered Tokens (Stopword Removal):", filtered_tokens)
print("Stemmed Tokens:", stemmed_tokens)
print("Lemmatized Tokens:", lemmatized_tokens)
print("POS Tags:", pos_tags)
print("Named Entities:", entities)
OUTPUT
Original Text: The quick brown fox jumps over the lazy dog. John Doe works at Google,
Inc. He is a software engineer.

Tokens: ['The', 'quick', 'brown', 'fox', 'jumps', 'over', 'the', 'lazy', 'dog', '.', 'John', 'Doe',
'works', 'at', 'Google', ',', 'Inc', '.', 'He', 'is', 'a', 'software', 'engineer', '.’]

Filtered Tokens (Stopword Removal): ['quick', 'brown', 'fox', 'jumps', 'lazy', 'dog', '.',
'John', 'Doe', 'works', 'Google', ',', 'Inc', '.', 'software', 'engineer', '.’]

Stemmed Tokens: ['quick', 'brown', 'fox', 'jump', 'lazi', 'dog', '.', 'john', 'doe', 'work', 'googl',
',', 'inc', '.', 'softwar', 'engin', '.’]

Lemmatized Tokens: ['quick', 'brown', 'fox', 'jump', 'lazy', 'dog', '.', 'John', 'Doe', 'work',
'Google', ',', 'Inc', '.', 'software', 'engineer', '.’]

POS Tags: [('quick', 'JJ'), ('brown', 'NN'), ('fox', 'NN'), ('jumps', 'NNS'), ('lazy', 'JJ'), ('dog',
'NN'), ('.', '.'), ('John', 'NNP'), ('Doe', 'NNP'), ('works', 'VBZ'), ('Google', 'NNP'), (',', ','),
('Inc', 'NNP'), ('.', '.'), ('software', 'NN'), ('engineer', 'NN'), ('.', '.')]
Important of Data Pre-processing
1. Noise Reduction: Text data often contains noise in the form of special characters, punctuation, and
irrelevant words. Preprocessing helps to remove or minimize this noise, which can improve the quality of
the data and the performance of downstream NLP tasks.
2. Normalization: Text data may contain variations in spelling, capitalization, and word forms.
Preprocessing techniques such as lowercasing, stemming, and lemmatization help to normalize the text,
making it consistent and easier to analyze.
3. Dimensionality Reduction: Text data can be high-dimensional, especially when represented as a bag-of-
words or TF-IDF matrix. Preprocessing techniques like stopword removal and feature selection help to
reduce the dimensionality of the data, which can improve the efficiency of algorithms and reduce
computational costs.
4. Improving Model Performance: Clean and preprocessed data can lead to better performance of NLP
models. By removing irrelevant information and standardizing the text, preprocessing can help models
focus on the most important features and patterns in the data, leading to more accurate predictions and
classifications.
5. Facilitating Interpretation: Preprocessing can make the text data more interpretable and understandable.
For example, part-of-speech tagging and named entity recognition provide valuable insights into the
linguistic structure and content of the text, which can aid in analysis and interpretation.
6. Enabling Generalization: Preprocessing ensures that the data is in a suitable format for modeling and
analysis. By standardizing the text and removing inconsistencies, preprocessing helps models generalize
well to new, unseen data, which is essential for robust performance in real-world applications.
ELIZA
• Background:
• ELIZA, named after Eliza Doolittle from George Bernard Shaw's play "Pygmalion", was developed by Joseph
Weizenbaum in the mid-1960s.
• It was one of the earliest attempts at creating a conversational agent or chatbot using natural language
processing techniques.
• ELIZA aimed to simulate a Rogerian psychotherapist by engaging users in text-based conversations.
• Parser Functionality:
• ELIZA's parser relied on simple pattern matching and transformation rules.
• It used a set of pre-defined patterns or regular expressions to analyze user input.
• When a pattern matched the input, ELIZA applied transformation rules to generate a response.
• The transformation rules typically involved replacing pronouns, rephrasing statements, or asking open-ended
questions to keep the conversation going.
• For example, if a user input contained phrases like "I feel X" or "My X hurts", ELIZA might respond with
statements like "Tell me more about your X" or "How does X make you feel?".
• Impact:
• ELIZA had a significant impact on both the public and the field of artificial intelligence.
• It captured the public's imagination and sparked widespread interest in conversational agents and AI.
• ELIZA demonstrated the potential for using natural language processing techniques to create engaging and
interactive human-computer interfaces.
• It also raised ethical questions about the implications of AI and the nature of human-computer interactions.
LUNAR
• Background:
• LUNAR (Linguistic Universal Numerical Analyzer and Reformatter) was developed in the 1970s by Roger Schank and his
colleagues at Yale University.
• It was designed to parse and understand natural language queries about lunar rock samples collected during the Apollo
missions.
• LUNAR aimed to provide researchers with a natural language interface to query and analyze the data collected from the
lunar samples.
• Parser Functionality:
• LUNAR employed a more sophisticated parser compared to ELIZA, based on the conceptual dependency theory.
• Conceptual dependency theory is a semantic analysis approach that represents the meaning of sentences in terms of actions,
actors, objects, and goals.
• LUNAR attempted to parse and understand the underlying meaning of user queries by analyzing the relationships between
concepts expressed in the text.
• It used a set of semantic primitives and rules to represent the meaning of sentences in a formal, structured manner.
• For example, a user query like "What are the chemical compositions of the lunar samples?" would be parsed into a
conceptual structure representing a request for information about the chemical compositions of lunar samples.
• Impact:
• LUNAR demonstrated the feasibility of using semantic analysis techniques to parse and understand natural language
queries.
• It showcased the potential for more sophisticated approaches to natural language processing beyond simple pattern
matching.
• LUNAR's development contributed to the advancement of natural language understanding and paved the way for future
research in semantic parsing and knowledge representation.
 Unit – 3
TOOLS AND TECHNIQUES:
Word-to-Vec conversion. Term Frequency-Inverse Document Frequency.
FrameNet. English WordNet and Indian WordNet. Components of WordNet. Semantic analysis
using WordNet. Understanding Natural Language Tool Kit (NLTK) tool for using WordNet. NLP
and Indian languages.
Word-to-Vec conversion
Word2Vec is like a smart tool that looks at all the sentences in this book and tries to
understand what each word means based on the other words around it.
1.Learning from Context: Word2Vec learns from the words that are near each other in
sentences. It looks at a target word and tries to predict the words nearby.
2.Two Approaches: There are two main ways Word2Vec does this:
1. Continuous Bag of Words (CBOW): Imagine a missing word in a sentence, and Word2Vec tries
to guess what that word is based on the other words around it.
2. Skip-gram: This time, Word2Vec starts with one word and tries to predict the words around it.
3.Making Word Representations: Word2Vec turns each word into a special kind of
number. These numbers show the meaning of each word based on how they're used in
sentences. For example, words with similar meanings will have similar numbers.
4.Practical Use: Once Word2Vec learns from the book, it can help with lots of tasks! Like:
1. Understanding how words are related (like "king" is to "queen" as "man" is to "woman").
2. Figuring out which words are similar (like "cat" and "dog" are similar because they're both
animals).
3. Helping computers understand human language better in things like search engines, chatbots, and
more.
Continuous Bag of Words (CBOW)
In this architecture, the model predicts the target word based on its
context words within a fixed window size. It takes the context words as
input and predicts the target word in the middle.

Skip-gram
In this architecture, the model predicts the context words given a target
word. It takes the target word as input and tries to predict the
surrounding context words.
Pre-trained Word Vectors:
• Pre-trained word vectors are word embeddings that are learned from a large
corpus of text data before they are used for a specific task.
• In this approach, Word2Vec models are trained on massive datasets like
Wikipedia or news articles to learn the relationships between words in a
language.
• These pre-trained word vectors capture semantic relationships and context
from the text they were trained on.
• They can be directly used in downstream natural language processing
(NLP) tasks without the need for additional training.
• Common pre-trained word vector models include Word2Vec, GloVe (Global
Vectors for Word Representation), and FastText.
Post-training Word Vectors:
• Post-training involves fine-tuning pre-trained word vectors on a specific
task or domain to adapt them to the target dataset or task.
• After loading pre-trained word vectors, you can continue training them on
your specific dataset to make them more relevant to your task.
• This fine-tuning process allows the word vectors to capture domain-specific
nuances and improve performance on the target task.
• For example, if you have a dataset related to medical text, you might fine-
tune pre-trained word vectors on medical literature to better represent
medical terminologies and concepts.
• Post-training can be done by updating the parameters of the pre-trained
Word2Vec model using backpropagation during training on the target
dataset.
Term Frequency-Inverse Document
Frequency
• It is a numerical statistic used in Natural Language Processing and information retrieval
to evaluate the importance of a word in a document relative to a collection of documents,
often referred to as a corpus.
• Term Frequency (TF): Term Frequency measures how frequently a term (word) appears in
a document. It is calculated by dividing the number of occurrences of a term in a
document by the total number of terms in that document. The idea is that words that
appear frequently in a document are more important than words that appear less
frequently.
• TF = (Number of times term appears in a document) / (Total number of terms in the
document)
• Inverse Document Frequency (IDF): Inverse Document Frequency measures the rarity of
a term across all documents in the corpus. It is calculated by dividing the total number of
documents by the number of documents containing the term, and then taking the
logarithm of that ratio. The IDF score penalizes terms that appear in many documents and
rewards terms that appear in few documents.
• IDF = log_e(Total number of documents / Number of documents containing the term)
• TF-IDF = TF * IDF
Equation and example
Importance
• Keyword Importance: TF-IDF helps in identifying keywords or terms that are most
relevant to a particular document or query. By giving higher weights to terms that appear
frequently in a document but rarely across the corpus, TF-IDF emphasizes terms that are
characteristic of the content of the document.
• Document Similarity: TF-IDF can be used to measure the similarity between documents
in a corpus. By representing documents as vectors based on their TF-IDF scores for
different terms, similarity measures such as cosine similarity can be applied to determine
how closely related two documents are in terms of their content.
• Information Retrieval: In information retrieval systems like search engines, TF-IDF is
used to rank documents based on their relevance to a user query. Documents containing
terms that are highly weighted by TF-IDF are considered more relevant and are typically
ranked higher in search results.
• Text Summarization: TF-IDF can be utilized in text summarization tasks to identify the
most important sentences or passages within a document. Sentences containing terms with
high TF-IDF scores are more likely to capture the main ideas or themes of the document
and thus can be included in the summary.
• Document Classification: TF-IDF features are commonly used in document
classification tasks, where documents are assigned to predefined categories or topics. By
extracting TF-IDF features from documents and training classification models on these
features, the models can learn to distinguish between different document categories based
on the importance of terms.
Example
• Document 1: "The quick brown fox jumps over the lazy dog."
• Document 2: "A brown fox is seen jumping over a dog."
• Document 3: "The cat and the dog are friends.“

• What can be concluded about the word “fox” based on its TF-IDF
value.
• In Document 1, "fox" appears once out of 9 words, so TF for "fox" in
Document 1 = 1/9.
• In Document 2, "fox" appears once out of 8 words, so TF for "fox" in
Document 2 = 1/9.
• In Document 3, "fox" appears once out of 7 words, so TF for "fox" in
Document 3 = 0/7.
• IDF for "fox" = log(3 / 2) ≈ 0.176.
• TF-IDF for "fox" in Document 1 = (1/9) * 0.176 ≈ 0.0196.
• TF-IDF for "fox" in Document 2 = (1/9) * 0.176 ≈ 0.0196.
• TF-IDF for "fox" in Document 3 = (0) * 0.176 ≈ 0.
FrameNet
• FrameNet is a lexical database and computational resource for English developed
at the International Computer Science Institute (ICSI) in Berkeley, California.
• It provides a structured representation of the way people understand the meaning
of words and phrases in various contexts by organizing them into semantic frames.
• Each frame consists of a set of frame elements (roles), along with lexical units
(words or phrases) that evoke the frame and describe the roles.
Example
• Frame: Buying
• Frame Elements:
1. Buyer: The person or entity purchasing something.
2. Seller: The person or entity selling something.
3. Goods: The items being purchased.
4. Price: The amount of money exchanged for the goods.
5. Transaction: The act of buying and selling.
• Lexical Units (Words or Phrases):
1. Buy
2. Purchase
3. Acquire
4. Shop
5. Sell
6. Trade
7. Bargain
8. Deal
Example Sentence: "The customer (Buyer) went to the store (Seller) and bought
(Transaction) a new laptop (Goods) for $1000 (Price)."
English Wordnet
• WordNet is a lexical database for the English language that groups words into sets of
synonyms called synsets, provides short definitions, and records relationships between
these synonym sets.
• It's structured like a thesaurus but with more elaborate information.
Key Features
1.Synsets: WordNet organizes words into synsets, which are sets of words that are
synonymous or semantically related. Each synset represents a distinct concept or
meaning.
2.Word Relationships: WordNet captures various semantic relationships between
words, such as hyponymy (is-a relationships), hypernymy (part-of relationships),
meronymy (whole-part relationships), and antonymy (opposite meanings).
3.Linguistic Hierarchy: WordNet arranges words in a hierarchical structure, with
more general concepts at the top and more specific concepts at the bottom. This
hierarchy allows for easy navigation between related words.
4.Definitions: Each synset in WordNet is accompanied by a short definition that
helps clarify its meaning and usage.
5.Part of Speech: WordNet distinguishes between different parts of speech (nouns,
verbs, adjectives, adverbs) and provides separate synsets for each.
6.Applications: WordNet is widely used in various natural language processing
tasks, such as text analysis, information retrieval, machine translation, and word
sense disambiguation.
English and Hindi Wordnet

• https://wordnet.princeton.edu/
• https://www.cfilt.iitb.ac.in/wordnet/webhwn/
•ह दिं ी शब्द सिंकल्पनाकोश
Components of WordNet
 Components of WordNet
1. Synsets (Synonym Sets): Synsets are groups of words that are synonymous or semantically related. Each
synset represents a distinct concept or meaning. For example, the synset {car, automobile, motorcar,
machine} represents the concept of a vehicle.
2. Words and Lemmas: WordNet includes a vast collection of words from the English language. Each word
is associated with one or more synsets and may have multiple senses or meanings.
3. Part of Speech (POS): Words in WordNet are categorized based on their part of speech, such as nouns,
verbs, adjectives, and adverbs. This helps in organizing the lexicon and identifying word relationships
based on their grammatical roles.
4. Semantic Relationships: WordNet captures various semantic relationships between words, including
hyponymy (is-a relationship), hypernymy (has-a relationship), meronymy (part-of relationship), holonymy
(whole-of relationship), antonymy (opposite meaning), entailment (logical implication), and more.
5. Glosses and Definitions: Each synset in WordNet is accompanied by a gloss or definition that describes
its meaning. These glosses provide additional context and clarification for the synsets.
6. Hierarchical Structure: Synsets in WordNet are organized in a hierarchical structure, with broader
concepts at higher levels and more specific concepts at lower levels. This hierarchical organization allows
for easy navigation and exploration of related concepts.
7. Polysemy and Homonymy: WordNet distinguishes between polysemous words (words with multiple
related meanings) and homonymous words (words with unrelated meanings). Each sense of a polysemous
word is represented by a separate synset, while homonymous words are treated as distinct entries.
8. Word Relationships: In addition to the semantic relationships mentioned earlier, WordNet also includes
other word relationships such as similarity (similarity between words), entailment (logical inference
between verbs), and derivational morphology (word formation processes like affixation and derivation).
Practice
• Document 1: "Machine learning algorithms are used to extract
meaningful insights from unstructured text."
• Document 2: "Text mining applications process large volumes of
textual data to uncover patterns and trends."
• Document 3: "Data scientists use advanced statistical techniques to
derive meaningful patterns and trends from datasets.“

• Calculate the TF-IDF values for the term "text" in each document:
Answer
• Document 1: TF-IDF = TF * IDF = 1 * 0.176 ≈ 0.176 (Low TF-IDF value)
• Document 2: TF-IDF = TF * IDF = 2 * 0.176 ≈ 0.352 (Medium TF-IDF
value)
• Document 3: TF-IDF = TF * IDF = 0 * 0.176 ≈ 0 (Zero TF-IDF value)
• In this example:
• Document 1 represents a low TF-IDF value, as the term "text" appears only
once and is relatively common.
• Document 2 represents a medium TF-IDF value, as the term "text" appears
twice and is moderately informative.
• Document 3 represents a zero TF-IDF value, as the term "text" does not
appear, indicating its absence or negligible importance.
Q&A
1.How does FrameNet contribute to semantic analysis in natural
language processing? Provide examples.
2.Compare and contrast English WordNet and Indian WordNet,
highlighting their key differences.
3.What are the main components of WordNet? Explain each component
briefly.
4.How does FrameNet categorize lexical units into frames? Provide
examples of frames and their lexical units.
5.Describe the process of word sense disambiguation using FrameNet
and WordNet.
Semantic analysis using WordNet
• Semantic analysis using WordNet involves leveraging WordNet, a
lexical database of the English language, to understand the meaning
and relationships between words.
• It employs a structured hierarchy of concepts called "synsets"
(synonym sets), which group together words that have similar
meanings.
• Semantic analysis using WordNet typically includes multiple steps:
1.Word Sense Disambiguation: Identifying the correct sense of a word in a
given context.
WordNet provides multiple senses for many words, and disambiguation helps in
selecting the appropriate sense based on the surrounding words and context.
2.Synonymy and Hyponymy: Exploring synonyms (words with similar
meanings) and hyponyms (words that are more specific than a given word)
to understand the semantic relationships between words.
WordNet organizes words into hierarchies based on these relationships, allowing for
more nuanced analysis.
3.Antonymy: Identifying antonyms (words with opposite meanings) to
further understand the contrasts and relationships between words.
4.Semantic Similarity: Quantifying the similarity between words or phrases
based on their semantic meanings. This can be useful for tasks like
information retrieval, text summarization, and machine translation.
5.Ontology Development: Building ontologies or knowledge graphs by
mapping concepts in WordNet to real-world entities and relationships. This
structured representation facilitates various natural language processing
tasks, such as question answering and semantic search.
• Word Sense Disambiguation:
• I saw a bat.
• I deposited money in the bank.
• Synonymy and Hyponymy:
• WordNet identifies that "cat" and "feline" are synonyms, indicating that they both refer to the same
concept.
• Understanding that "apple" is a hyponym of "fruit," indicating that an apple is a specific type of fruit.
• Antonymy:
• Example 1: "hot" as an antonym of "cold," representing opposite temperature states.
• Example 2: Recognizing "happy" and "sad" as antonyms, representing contrasting emotional states.
• Semantic Similarity:
• Example 1: WordNet quantifies the semantic similarity between "car" and "vehicle" as high, indicating
that they belong to the same category.
• Example 2: Assessing the semantic similarity between "run" and "sprint" to understand their closeness
in meaning within the context of movement.
• Ontology Development:
• Example 1: Mapping concepts such as "animal," "plant," and "artifact" in WordNet to corresponding
entities in a broader knowledge graph for comprehensive ontology development.
• Example 2: Extending WordNet's hierarchy to include domain-specific terms and relationships, such
as medical conditions and treatments, to build a specialized ontology for healthcare applications.
NLP and Indian languages
Natural Language Processing (NLP) for Indian languages poses unique challenges due to the
linguistic diversity and complexity of Indian languages.
• Script Diversity: Indian languages are written in multiple scripts such as Devanagari (used for
Hindi, Marathi, Nepali, etc.), Tamil script (for Tamil), Bengali script (for Bengali), and many
others. Handling text in different scripts requires specialized preprocessing and encoding
techniques.
• Morphological Complexity: Indian languages often exhibit rich morphological features,
including complex word forms, inflections, and compound words. Morphological analysis and
processing are crucial for tasks like tokenization, stemming, and lemmatization.
• Limited Resources: Compared to major languages like English, resources such as annotated
corpora, lexicons, and language models are scarce for Indian languages. Building
comprehensive resources for Indian languages is a significant challenge due to the diversity of
languages and dialects.
• Code-Switching: Many Indian language speakers frequently code-switch between multiple
languages or mix languages within a single sentence or conversation. Handling code-switching
poses challenges for tasks like language identification, part-of-speech tagging, and named
entity recognition.
• Speech Processing: Speech recognition and synthesis for Indian languages require specialized
models and datasets due to variations in pronunciation, accents, and dialects. Developing
accurate speech recognition systems for Indian languages is an ongoing research area.
• Named Entity Recognition (NER): NER systems for Indian languages need to recognize and
classify named entities such as person names, locations, organizations, and others specific to Indian
culture and context. Building annotated datasets and models for NER in Indian languages is
challenging but essential for applications like information extraction and entity linking.
• Language Variation: Indian languages exhibit significant variation in vocabulary, grammar, and
syntax across regions and dialects. NLP tools and models need to account for these variations to
ensure accurate and robust performance across different linguistic contexts.
• Low-Resource Settings: Many Indian languages are considered low-resource languages in the
context of NLP, lacking sufficient annotated data and research attention. Addressing the needs of
low-resource languages requires innovative approaches such as transfer learning, domain
adaptation, and crowd-sourcing techniques.
Important Resources
• https://anoopkunchukuttan.gitlab.io/publications/presentations/wildre_
keynote_2020.pdf

• https://www.analyticsvidhya.com/blog/2020/01/3-important-nlp-
libraries-indian-languages-python/

• https://link.springer.com/article/10.1007/s42452-020-2983-x
 Unit – 4
APPLICATIONS OF NLP:
Word Sense Disambiguation, Text Summarization, Optical Character
Recognition, Sentiment Analysis and Opinion Mining, Chatbots and Voice Assistants, Automated
Question Answering, Machine Translation.
Word Sense Disambiguation
Word Sense Disambiguation (WSD) is a fundamental task in natural language
processing (NLP) that aims to determine the intended meaning of a word
within a given context.
Many words in natural language have multiple meanings or senses, and WSD
seeks to identify the correct sense of a word based on the surrounding words
or phrases in a sentence.
This is crucial for various NLP applications, such as machine translation,
information retrieval, and sentiment analysis, where accurately understanding
the meaning of words is essential for producing meaningful and accurate
results.
WSD can be approached using various techniques, including knowledge-
based methods, supervised and unsupervised learning algorithms, and hybrid
approaches that combine multiple strategies for better accuracy.
Sentence: "I need to book a flight for my vacation."
In this sentence, the word "book" could have multiple meanings. It could
refer to:
1.A physical object consisting of pages bound together.
2.The act of reserving or arranging travel plans, such as booking a flight.
Word Sense Disambiguation aims to determine the correct sense of "book"
based on the context of the sentence. In this case, given the presence of
"flight" and "vacation," it's more likely that "book" refers to the action of
reserving travel plans rather than a physical object. Therefore, through WSD,
we can disambiguate the word "book" to its appropriate sense in this context.
 Text Summarization
• Text summarization is a Natural Language Processing (NLP) technique used to create
a concise and coherent summary of a longer text while retaining its key information. It
aims to reduce the length of the text while preserving its most important ideas and
concepts. Text summarization can be categorized into two main types: extractive
summarization and abstractive summarization.
Extractive Summarization
• In extractive summarization, the summary is generated by selecting a subset of
sentences from the original text.
• The selected sentences are typically the most informative or representative
sentences from the original text.
• Extractive summarization methods usually involve the following steps:
• Text Preprocessing: Tokenization, sentence splitting, stop word removal, and
stemming.
• Sentence Representation: Converting sentences into numerical vectors.
• Sentence Scoring: Assigning importance scores to each sentence using various
methods such as TF-IDF, TextRank, or graph-based algorithms.
• Sentence Selection: Selecting the top-ranked sentences based on their
importance scores to form the summary.
Abstractive Summarization
• In abstractive summarization, the summary is generated by interpreting and
paraphrasing the original text.
• Abstractive summarization methods involve generating new sentences that may
not exist in the original text but capture its main ideas.
• Abstractive summarization methods usually involve the following steps:
• Text Preprocessing: Similar to extractive summarization.
• Text Representation: Converting the text into a format suitable for neural
networks, such as word embeddings.
• Sequence-to-Sequence Modeling: Training a neural network to generate
summaries by learning the mapping between input text and output summaries.
• Decoding: Generating summaries by decoding the output of the neural
network into human-readable text.
• Challenges in Text Summarization:
• Preserving the important information while reducing the length of the text.
• Ensuring coherence and readability of the summary.
• Handling different types of input texts such as news articles, research papers, or social media
posts.
• Dealing with ambiguity and redundancy in the original text.
• Evaluating the quality of generated summaries objectively.
• Applications of Text Summarization:
• News Summarization: Generating concise summaries of news articles.
• Document Summarization: Creating summaries of long documents such as research papers,
reports, or legal documents.
• Email Summarization: Summarizing long email threads to highlight important points.
• Social Media Summarization: Generating summaries of social media posts or comments.
• Chatbot Responses: Generating concise responses in chatbot conversations.
 Optical Character Recognition
• Optical Character Recognition (OCR) is a technology used to convert different types of
documents, such as scanned paper documents, PDF files, or images captured by a digital
camera, into editable and searchable data. OCR systems analyze the structure of a document
and identify the individual characters in the text, recognizing them as alphanumeric symbols.
1. Preprocessing:
• Image Acquisition: The document to be recognized is captured using a scanner, digital
camera, or other image-capturing devices.
• Image Enhancement: The captured image is enhanced to improve its quality and clarity.
This may involve processes like noise removal, contrast adjustment, and edge sharpening.
2. Text Detection:
• The OCR system identifies the areas within the image that contain text.
• Techniques such as edge detection, contour tracing, and connected component analysis are
used to locate text regions.
3. Text Segmentation:
• Once the text regions are identified, the OCR system segments the text into individual
characters or words.
• Techniques such as line segmentation, word segmentation, and character segmentation are
used to separate the text elements.
4. Feature Extraction:
• The system extracts features from the segmented text elements to represent them
numerically.
• Features may include shape, size, orientation, and texture of characters.
• Feature extraction is essential for training machine learning models used in OCR.
5. Character Recognition:
• The extracted features are matched against a database of known characters.
• Machine learning algorithms such as neural networks, Support Vector Machines (SVM), or
Hidden Markov Models (HMM) are commonly used for character recognition.
• The system assigns the most likely character to each extracted feature based on the training
data.
• 6. Post-processing:
• Error Correction: Post-processing techniques are used to correct any recognition
errors made during character recognition. This may involve using language
models or context-based algorithms to improve accuracy.
• Text Reconstruction: Recognized characters are combined to form words,
sentences, and paragraphs.
• Document Formatting: The OCR system reconstructs the original document
layout, including fonts, styles, and formatting.
• 7. Output:
• The final output of the OCR process is an editable and searchable text document.
• The recognized text can be exported to various file formats such as TXT, DOC,
PDF, or HTML.
Applications of OCR
• Document Digitization: Converting printed documents into editable and
searchable digital formats.
• Data Entry Automation: Automating data entry tasks by extracting text from
scanned documents.
• Text Translation: Translating printed text into different languages.
• Handwriting Recognition: Recognizing handwritten text and converting it into
digital format.
• License Plate Recognition: Extracting text from images of license plates for
vehicle identification.
• Automatic Number Plate Recognition (ANPR): Recognizing vehicle registration
numbers for surveillance and security purposes.
• Document Analysis: Analyzing the content of documents for information
extraction, classification, and indexing.
Opinion Mining
• Product Reviews: Analyzing customer reviews to understand opinions and sentiments about
products and services.
• Social Media Analysis: Analyzing sentiments expressed on social media platforms to gauge
public opinion on various topics.
• Brand Monitoring: Monitoring online mentions and opinions about brands and products.
• Market Research: Analyzing customer feedback and surveys to identify trends and patterns
in opinions.
• Political Analysis: Analyzing public opinions and sentiments about political candidates,
parties, and policies.
• Customer Feedback Analysis: Analyzing feedback from customer support interactions,
surveys, and emails to identify areas for improvement.
Chatbots and Voice Assistants
• Chatbots and voice assistants are sophisticated AI applications designed to simulate
human conversation.
• They leverage NLP, machine learning, and artificial intelligence to understand and
respond to user queries in a conversational manner.
• These technologies are increasingly integrated into various domains such as customer
service, healthcare, education, and personal productivity, providing efficient and
interactive user experiences.
Chatbots
1. Definition and Types
• Rule-Based Chatbots: Operate based on predefined rules and simple conditional statements.
They can handle straightforward queries but struggle with complex conversations.
• AI-Based Chatbots: Utilize machine learning and NLP to understand context and provide
more accurate and relevant responses. They can handle a wider range of queries and learn
from interactions to improve over time.
2. Key Components
• Natural Language Understanding (NLU): Interprets user input, identifying intent and
extracting relevant entities.
• Dialogue Management: Maintains the state of the conversation and determines the
appropriate response.
• Natural Language Generation (NLG): Converts the system's response into human-readable
text.
3. Benefits
• 24/7 Availability: Provide round-the-clock assistance.
• Scalability: Handle multiple interactions simultaneously.
• Cost Efficiency: Reduce operational costs by automating routine tasks.
Voice Assistant
1. Definition and Examples
• Voice assistants are AI systems that understand and respond to voice commands.
Popular examples include Amazon's Alexa, Apple's Siri, Google Assistant, and
Microsoft's Cortana.
2. Key Components
• Automatic Speech Recognition (ASR): Converts spoken language into text.
• Natural Language Understanding (NLU): Interprets the text to understand user intent.
• Text-to-Speech (TTS): Converts text responses back into spoken language.
3. Benefits
• Hands-Free Operation: Allow users to perform tasks without using their hands,
improving convenience and accessibility.
• Personalization: Tailor responses based on user preferences and past interactions.
• Integration: Seamlessly integrate with other services and devices to enhance
functionality.
Automated Question Answering
• Automated Question Answering (QA) systems are designed to provide precise
answers to user queries posed in natural language.
• Leveraging advancements in NLP, machine learning, and information retrieval,
• QA systems are capable of understanding, processing, and responding to questions
with high accuracy and relevance.
• They are used across various domains such as customer support, education,
healthcare, and search engines.
Types of Question Answering Systems
1. Rule-Based QA Systems
• Operate using a predefined set of rules and patterns.
• Limited to specific domains where the rules are explicitly defined.
2. Retrieval-Based QA Systems
• Search a large corpus of documents to find and extract the answer.
• Use keyword matching and ranking algorithms to identify relevant passages.
3. Generative QA Systems
• Generate answers from scratch using machine learning models.
• Capable of producing more nuanced and contextually appropriate responses.
4. Hybrid QA Systems
• Combine retrieval-based and generative approaches to improve accuracy.
• Retrieve relevant documents or passages first and then generate answers based on
the retrieved content.
 Key Components of QA Systems
1. Question Processing
• Tokenization: Breaking down the question into individual words or tokens.
• Part-of-Speech Tagging: Identifying the grammatical parts of speech of each token.
• Named Entity Recognition (NER): Detecting entities such as names, dates, and locations.
• Dependency Parsing: Analyzing the syntactic structure of the question to understand
relationships between words.
2. Document Retrieval
• Indexing: Organizing a large corpus of documents to facilitate efficient searching.
• Search Algorithms: Techniques like TF-IDF, BM25, or neural retrieval models to find
relevant documents.
• Ranking: Ordering the retrieved documents based on relevance to the query.
3. Answer Extraction
• Passage Selection: Identifying specific passages or sentences within the
documents that are most likely to contain the answer.
• Answer Scoring: Assigning scores to candidate answers based on their
likelihood of being correct.
• Answer Formatting: Structuring the selected answer in a coherent and
understandable manner.
4. Answer Generation
• For generative models, this involves creating responses that are contextually
and grammatically correct.
• Techniques like sequence-to-sequence learning, transformer models, and
fine-tuning on large datasets are commonly used.
Machine Translation
• It involves using computational methods to translate text or speech from one
language to another.
• With the rapid growth of global communication, MT systems have become
increasingly important in breaking down language barriers and facilitating cross-
cultural interactions.
• MT systems leverage NLP, machine learning, and deep learning techniques to
perform translations accurately and efficiently.
 Types of Machine Learning
1. Rule-Based Machine Translation (RBMT)
• Mechanism: Uses a set of linguistic rules and dictionaries for translation.
• Process: Involves syntactic and semantic analysis of the source text, applying grammatical
rules of the target language to generate the translation.
• Advantages: High interpretability and control over the translation process.
• Disadvantages: Requires extensive linguistic knowledge and manual rule creation, leading to
scalability issues.
2. Statistical Machine Translation (SMT)
• Mechanism: Uses statistical models based on bilingual text corpora to generate translations.
• Process: Involves aligning parallel texts (sentences in both source and target languages) to
build probabilistic models that predict the most likely translation.
• Advantages: Capable of handling a wide range of language pairs and domains with sufficient
training data.
• Disadvantages: Requires large amounts of bilingual data and often produces less fluent
translations compared to human-generated text.
3. Example-Based Machine Translation (EBMT)
• Mechanism: Relies on a database of previously translated examples to perform
translations.
• Process: Matches new input sentences with examples in the database and uses analogical
reasoning to produce translations.
• Advantages: Effective for languages and domains with extensive bilingual corpora.
• Disadvantages: Limited by the quality and coverage of the example database.
4. Neural Machine Translation (NMT)
• Mechanism: Uses neural networks, particularly deep learning models, to perform end-
to-end translation.
• Process: Typically employs encoder-decoder architectures with attention mechanisms to
learn representations of the source and target texts.
• Advantages: Produces more fluent and accurate translations, can handle long-range
dependencies and contextual information.
• Disadvantages: Requires significant computational resources and large datasets for
training.
Components of MT
• 1. Encoder-Decoder Architecture
• Encoder: Processes the input sentence and converts it into a fixed-length context
vector.
• Decoder: Generates the translated sentence from the context vector.
• Bidirectional Encoders: Capture context from both directions (left-to-right and
right-to-left) for better understanding of the source sentence.
• 2. Attention Mechanisms
• Purpose: Allows the model to focus on specific parts of the source sentence while
generating each word in the target sentence.
• Types: Additive attention (Bahdanau) and multiplicative attention (Luong).
• Benefit: Improves translation accuracy and fluency by considering relevant source
words for each target word.
• 3. Transformer Models
• Mechanism: Use self-attention mechanisms to process all words in a sentence
simultaneously, rather than sequentially.
• Architecture: Comprises multiple layers of encoders and decoders, each with
self-attention and feed-forward neural networks.
• Advantages: Handles long-range dependencies efficiently and scales well with
large datasets.
• 4. Pretrained Language Models
• Examples: BERT, GPT, T5, and mBERT.
• Mechanism: Pretrained on large corpora and fine-tuned for specific translation
tasks.
• Advantages: Leverage transfer learning to improve translation quality, especially
for low-resource languages.