UNIT-IV

1. Define a Language Model. Explain about different types of Language Models.

Language Model
A Language Model estimates the probability distribution of sequences of words.

Unigram Model

A model that simply relies on how often a word occurs without looking at previous words
is called unigram.

Equation:

E.g.

Bigram Model

If a model considers only the previous word to predict the current word, then it is
called bigram model.

Equation
Example

Trigram Model

If a model considers two previous words to predict the current word, then it is
called trigram model.
Equation:

E.g.

N-Gram Model

An N-gram means a sequence of N words.

If a model considers previous N-1 words to predict the current word then it is called N-
Gram Model.

An n-gram model for the above example would calculate the following probability:

P('The prime minister of our country') = P('The', 'prime', 'minister', 'of', ‘our’,’country’) =
P('The')P('prime'|'The')P('minister'|'The prime')P('of'|'The prime minister')P(‘our’| ‘The prime
minister of’)P(‘country’|’The prime minister of our’)
Since it's impractical to calculate these conditional probabilities, using Markov assumption, we
approximate this to a bigram model:
P('The prime minister of our country') ~
P('The')P('prime'|'The')P('minister'|'prime')P('of'|minister')P(‘our’|’of’)P(‘country’|’our’).

Applications of N-Gram Models

 N-gram models are widely used in statistical natural language processing.
 In speech recognition, phonemes and sequences of phonemes are modeled using a n-gram
distribution.
 For parsing, words are modeled such that each n-gram is composed of n words.

Limitations of N-Gram Models

Data sparsity:
Some n-grams may not occur frequently or at all in the training data, resulting in low or
zero probabilities. This can lead to inaccurate or incomplete language models that fail to
capture the diversity and variability of natural language.
 2. A) Explain about Language Model evaluation
Language Models can be evaluated using the following metrics
1. Entropy
2. Cross entropy
3. Perplexity
Entropy :
Entropy is defined as a measure of randomness or disorder of a system.
Cross Entropy:
Cross Entropy is a measure of the difference between two probability distributions for a
given set of events.
Perplexity
Perplexity is a useful metric to evaluate models in Natural Language Processing (NLP).
Perplexity is calculated as exponent of the loss obtained from the model.

Formally, the perplexity is the function of the probability that the probabilistic language model
assigns to the test data. For a test set W = w1, w2, …, wN, the perplexity is the probability of the
test set, normalized by the number of words:

Using the chain rule of probability, the equation can be expanded as follows;

This equation can be modified to accommodate the language model that we use. For example, if
we use a bigram language model, then the equation can be modified as follows;
2.B) Explain about parameter estimation in Language Models
Maximum Likelihood Estimation
The maximum likelihood estimation is a method that determines values for parameters of
the model. It is the statistical method of estimating the parameters of the probability distribution
by maximizing the likelihood function. The point in which the parameter value that maximizes
the likelihood function is called the maximum likelihood estimate.

We can use Maximum Likelihood Estimation to estimate the Bigram and Trigram
probabilities.

For Bigram probability,

Example:

The bigram probability is calculated by dividing the number of times the string “prime minister”
appears in the given corpus by the total number of times the word “prime” appears in the same
corpus.

For Trigram probability,

Example:

The trigram probability is calculated by dividing the number of times the string “prime minister
of” appears in the given corpus by the total number of times the string “prime minister” appears
in the same corpus.

Backoff and Smoothing

Backoff and smoothing are techniques in NLP to adjust probabilities to tackle data
sparseness and parameter estimation while building NLP models.
Backoff
The process of moving back to the lower-level language model is known as Backoff.
Smoothing:
Smoothing is about taking some probability mass from the events seen in training and
assigns it to unseen events.
Types of Smoothing
Smoothing is of different types in Natural Language Processing.
1. Laplace Smoothing (Add-1 smoothing)
2. Additive Smoothing (Add-K smoothing)
3. Good Turing Smoothing
 4. Linear Interpolation
5. Kneser-Ney Smoothing
6. Katz smoothing
7. Church and Gale Smoothing

Laplace smoothing (Add-1 smoothing)

We have used Maximum Likelihood Estimation (MLE) for training the parameters of an
N-gram model. The problem with MLE is that it assigns zero probability to unknown (unseen)
words. This is because, MLE uses a training corpus. If the word in the test set is not available in
the training set, then the count of that particular word is zero and it leads to zero probability.
To eliminate these zero probabilities, we can do smoothing.
Smoothing is about taking some probability mass from the events seen in training and
assigns it to unseen events. Add-1 smoothing (also called as Laplace smoothing) is a simple
smoothing technique that Add 1 to the count of all n-grams in the training set before normalizing
into probabilities.
Example:
Recall that the unigram and bi-gram probabilities for a word w are calculated as follows;
P(w) = C(w)/N
P(wn|wn-1) = C(wn-1 wn)/C(wn-1)
Where, P(w) is the unigram probability, P(wn-1 wn) is the bigram probability, C(w) is the count of
occurrence of w in the training set, C(wn-1 wn) is the count of bigram (wn-1 wn) in the training set,
N is the total number of word tokens in the training set.
Add-1 smoothing for unigrams
PLaplace(w) = (C(w)+1)/N+|V|
Here, N is the total number of tokens in the training set and |V| is the size of the vocabulary
represents the unique set of words in the training set.
As we have added 1 to the numerator, we have to normalize that by adding the count of unique
words with the denominator in order to normalize.
Add-1 smoothing for bigrams
PLaplace(wn|wn-1) = (C(wn-1 wn)+1)/C(wn-1)+|V|

Additive Smoothing (Add-K Smoothing)

This is very similar to “Add One” or Laplace smoothing. Instead of adding 1 as like in
Laplace smoothing, a k value is added
Good-Turing smoothing
Good Turing Smoothing technique uses the frequencies of the count of occurrence of N-
Grams for calculating the maximum likelihood estimate.
As per the Good-turing Smoothing, the probability will depend upon the following:
 In case, the bigram has never occurred in the corpus (which is the reality), the
probability will depend upon the number of bigrams which occurred exactly one
time and the total number of bigrams.
 In case, the bigram has occurred in the corpus, the probability will depend upon
number of bigrams which occurred more than one time of the current bigram, total
number of bigram which occurred same time as the current bigram (to/bigram) and
total number of bigrams.
Kneser-Ney Smoothing
In Good Turing smoothing, it is observed that the count of n-grams is discounted by a
constant/absolute value such as 0.75. The same intuition is applied for Kneser-Ney
Smoothing where absolute discounting is applied to the count of n-grams in addition to
adding the product of interpolation weight and probability of word to appear as novel
continuation.
Katz Smoothing
Good-turing technique is combined with interpolation. This method outperforms Good-
Turing by redistributing different probabilities to different unseen units.
Church and Gale Smoothing
Good-turing technique is combined with bucketing.
 Each n-gram is assigned to one of several buckets based on its frequency predicted
from lower-order models.
 Good-turing estimate is calculated for each bucket.
2.C) Explain about Language Model Adoption
Language: Model:
A language model is a statistical tool that analyzes the pattern of human language for the
prediction of words.
Types of Language Models
Language Models are classified into two types. They are
1. Statistical Language Models
2. Neural Language Models
Applications of Language Models
 Natural Language Processing
 Sentiment Analysis
 Chatbot Development
 Language Translation
 Content Generation and automation
 Custom Large Language Model development
 Question Answering systems
 Code generation and debugging
 Improving search engines
Pretrained Models
 Google BERT
BERT stands for Bidirectional Encoder Representations from Transformers
 Code BERT
 Hugging Face Transformers
 Open NMT
 Facebook RoBERTa
 ELMo
 GPT-3
 XLNet
 ULM Fit
Large Language Model(LLM):
A large language model is a computerized language model, embodied by an artificial neural
network using an enormous amount of parameters, that are trained on many GPUs in relatively
short time due to
Explain about Hidden Markov Model
Hidden Markov Model (HMM)
 Hidden Markov Model (HMM) is a simple sequence labeling model.
 It is a statistical Markov model in which the system being modeled is assumed to be a
Markov process with unobserved (i.e. hidden) states.
 By relating the observed events (Example - words in a sentence) with the hidden states
(Example - part of speech tags), it helps us in finding the most probable hidden state
sequence (Example – most relevant POS tag sequence for the given input sentence).
 HMM can be defined formally as a 5-tuple (Q, A, O, B, π) where each component can be
defined as follows.

Component Detailed Description

components

Q q1, q2, q3, …, qN Set of N hidden states

A a11, a12, …, ann  Set of transition probabilities

 A is the state transition probability matrix
 Each aij in A represents a transition probability value of moving
from state i to state j.
 Sum of transition probability values from a single state to all
other states should be 1. That is,

O o1, o2, …, oT A sequence of T observations

B bi(ot)  A sequence of observation likelihoods (emission probabilities)

 Each bi(ot) represents the emission probability. That is, the
probability of an observation ot which is generated from a state i.
π π1, π2, …, πN  Set of initial probabilities.
 π1 is the probability that the Markov chain will start in state i.
 if π1 = 0, it implies that the state i cannot be an initial state.
 The sum of all initial probabilities should be 1. That is,
 Understanding Hidden Markov Model - Example:
These components are explained with the following HMM. In this example, the states are related
to the weather conditions (Hot, Wet, Cold) and observations are related to the fabrics that we
wear (Cotton, Nylon, Wool).

As per the given HMM,

 Q = set of states = {Hot, Wet, Cold}

 A = transition probability matrix

o Transition probability matrix
Current state
Previous state Hot Wet Cold
Hot 0.6 0.3 0.1
Wet 0.4 0.4 0.2
Cold 0.1 0.4 0.5
o How to read this matrix? In this matrix, for example, aij is a transition probability from state i to
state j [which is represented as conditional probability P(j|i)];
aij = a11 = P(Hot|Hot) = 0.6
aij = a23 = P(Cold|Wet) = 0.2
 aij = a31 = P(Hot|Cold) = 0.1
o Sum of transition probability from a single state to all the other states = 1. In other words, we
would say that the total weights of arcs (or edges) going out of a state should be equal to 1. In
our example;
P(Hot|Hot)+P(Wet|Hot)+P(Cold|Hot) = 0.6+0.3+0.1 = 1

 O = sequence of observations = {Cotton, Nylon, Wool}

 B = Emission probability matrix
o Emission probability matrix
Cotton Nylon Wool
Hot 0.8 0.5 0.05
Wet 0.15 0.4 0.2
Cold 0.05 0.1 0.75
o The above said matrix consists of emission probability values represented as bi(ot). bi(ot) is the
probability of an observation o t generated from a state bi. For example, P(Nylon | Hot) = 0.5,
P(Wool | Cold) = 0.75 etc.
 π = [π1, π2, …, πN] = set of prior probabilities = [0.6, 0.3, 0.1]. Here, the values
refer to the prior probabilities P(Hot) = 0.6, P(Wet) = 0.3, and P(Cold) = 0.1
Explain about Word senses and Ambiguity
Challenges in Ambiguity
Ambiguity presents in almost all the steps of natural language processing. (Steps of NLP –
lexical analysis, syntactic analysis, semantic analysis, discourse analysis, and pragmatic
analysis).
Lexical ambiguity:
“The chicken is ready to eat”
Does the word “chicken” denote a live chicken or the cooked chicken meat? If it points to a live
chicken, the meaning will be “The chicken is ready to eat its food”. Otherwise, it means “The
cooked chicken is ready to be served for someone to eat”.
Also, the word “ready” is ambiguous and have two possible senses here. One is “cooked for eat”
and the other is “gear up to eat”.
What is the challenge?
The challenge in lexical ambiguity is to understand the correct sense of a word with respect to
the context in which it is used.
Solutions:
Part-of-speech tagging, and word sense disambiguation.
Syntactic ambiguity:
“The boy saw a girl in the bus”
This sentence would be understood in two ways based on the way we connect the prepositional
phrase “in the bus”. Possible understanding would be “The boy who was in the bus saw the girl”,
“The boy saw the girl where the girl is in the bus”.
What is the challenge?
The challenge in syntactic ambiguity is to connect the prepositional phrase (“in the bus”) with
subject or object.
Solutions:
Parsing and parse trees
Semantic ambiguity:
“Raghu and Geetha are married”
Possible interpretations would be (a) “Raghu and Geetha both are married to each other”, (b)
“Raghu is married and Geetha is also married, but not to each other”.
What is the challenge?
Though the words present in the sentence is correctly understood, still there may be more than
one ways to understand the sentence as a whole.
Apart from these, the following are the knowledge required to resolve ambiguity in general in
natural language processing;
 Phonetics, Morphology, Syntax, Semantics, Pragmatics, and Discourse.

Semantic Form and Logical Form

The semantics, or meaning, of an expression in natural language can be abstractly represented as
a logical form. Once an expression has been fully parsed and its syntactic ambiguities resolved,
its meaning should be uniquely represented in logical form.
 UNIT-V

Explain about automatic Text Summarization

Automatic Text Summarization
Automatic text summarization (automatic summarization/text summarization) is the
reduction of a given text to a smaller number of sentences without leaving out the main
ideas of the original text
Types of Text Summarization
There are two types of text summarization methods, namely extractive and abstractive.
Extractive Summarization
Extractive summarization is essentially picking out sentences from the text that can best
represent its summary.
Abstractive Summarization
Abstractive Text Summarization is the task of generating a short and concise summary
that captures the salient ideas of the source text. The generated summaries potentially
contain new phrases and sentences that may not appear in the source text.

Five techniques for text summarization in Python

1. Gensim
Gensim is an open-source topic and vector space modeling toolkit within the Python
programming language.
2. Sumy
Sumy is another library in Python that uses various algorithms to perform text summarization.
 LexRank
LexRank is a graphical-based summarizer.
 Luhn
Developed by an IBM researcher of the same name, Luhn is one of the oldest
summarization algorithms and ranks sentences based on a frequency criterion for
words.
 LSA
Latent semantic analysis is an automated method of summarization that utilizes
term frequency with singular value decomposition. It has become one of the most
used summarizers in recent years.
3. NLTK
The ‘Natural Language Toolkit’ is an NLP-based toolkit in Python that helps with text
summarization.
4. T5
To make use of Google’s T5 summarizer, there are a few prerequisites.
First, you will need to install PyTorch and Hugging Face’s Transformers.
5. GPT-3
GPT-3 is a successor to the GPT-2 API and is much more capable and functional.
Techniques for text Summarization
5W-1H technique of summarization
The 5W1H is a questioning and problem-solving method that aims to view ideas and
issues from different perspectives. It helps you to understand a problem better and find
the root cause of it. 5W is an acronym for What, Where, When, Why, and Who, while the
letter H stands for How.
3-2-1 technique of summarization
The 3-2-1 exit slip strategy is a method of summarizing one's learning with a basic format
in which: Students write three things they learned in today's lesson. Next, students write
two things they liked or two interesting facts about the lesson. Finally, students write one
question they still have about the lesson.
Tools for Text Summarization
 Summarize Bot
 Resoomer
 SMMRY
 TextSummarization
 Text Compactor
 Genei
 Jasper
 and ChatGPT Plus
Data sets for Text Summarization
 CNN / Dailymail
 Reddit TIFU
 Webis-TLDR-17 / Reddit TL;DR
 XSum
Define Information Retrieval System. Explain about different types of Information
retrieval.
An information retrieval (IR) system is a set of algorithms that facilitate the relevance of
displayed documents to searched queries.

An information retrieval comprises of the following four key elements:

1. D − Document Representation.
2. Q − Query Representation.
3. F − A framework to match and establish a relationship between D and Q.
4. R (q, di) − A ranking function that determines the similarity between the query and the
document to display relevant information.

There are three types of Information Retrieval (IR) models:

1. Classical IR Model — It is designed upon basic mathematical concepts and is the most
widely-used of IR models. Classic Information Retrieval models can be implemented with
ease. Its examples include Vector-space, Boolean and Probabilistic IR models. In this system,
the retrieval of information depends on documents containing the defined set of queries. There
is no ranking or grading of any kind. The different classical IR models take Document
Representation, Query representation, and Retrieval/Matching function into account in their
modelling. This is one of the most used Information retrieval models.

2. Non-Classical IR Model — They differ from classic models in that they are built upon
propositional logic. Examples of non-classical IR models include Information Logic, Situation
Theory, and Interaction models.

3. Alternative IR Model — These take principles of classical IR model and enhance upon to
create more functional models like the Cluster model, Alternative Set-Theoretic Models Fuzzy
Set model, Latent Semantic Indexing (LSI) model, Alternative Algebraic Models Generalized
Vector Space Model, etc.
Boolean Model — This model required information to be translated into a Boolean
expression and Boolean queries. The latter is used to determine the information needed to be
able to provide the right match when the Boolean expression is found to be true. It uses
Boolean operations AND, OR, NOT to create a combination of multiple terms based on what
the user asks. This is one of the information retrieval models that is widely used.

2. Vector Space Model — This model takes documents and queries denoted as vectors and
retrieves documents depending on how similar they are. This can result in two types of
vectors which are then used to rank search results either

 Binary in Boolean VSM.

 Weighted in Non-binary VSM.
. Probability Distribution Model — In this model, the documents are considered as
distributions of terms and queries are matched based on the similarity of these representations.
This is made possible using entropy or by computing the probable utility of the document.
They are if two types:

 Similarity-based Probability Distribution Model

 Expected-utility-based Probability Distribution Model
4. Probabilistic Models — The probabilistic model is rather simple and takes the probability
ranking to display results. To put it simply, documents are ranked based on the probability of
their relevance to a searched query. This is one of the most basic information retrieval
techniques used.
Define Machine translation. Explain about problems in Machine Translation
Machine Translation
Machine Translation (MT) is the task of automatically converting one natural language
into another, preserving the meaning of the input text, and producing fluent text in the output
language.
Types of Machine Translation
1. Statistical Machine Translation or SMT
 It works by alluding to statistical models that depend on the investigation of huge
volumes of bilingual content.
 It expects to decide the correspondence between a word from the source language
and a word from the objective language.
 A genuine illustration of this is Google Translate.
2. Rule-based Machine Translation or RBMT

 RBMT basically translates the basics of grammatical rules.

 It directs a grammatical examination of the source language and the objective

language to create the translated sentence.

 But, RBMT requires broad editing, and its substantial reliance on dictionaries

implies that proficiency is accomplished after a significant period.

3. Hybrid Machine Translation or HMT

 HMT, as the term demonstrates, is a mix of RBMT and SMT.

 It uses a translation memory, making it unquestionably more successful regarding

quality

 . There are several approaches to HMT like multi-engine, statistical rule

generation, multi-pass, and confidence-based.

4. Neural Machine Translation or NMT

 NMT is a type of machine translation that relies upon neural network models

(based on the human brain) to build statistical models with the end goal of

translation.
  The essential advantage of NMT is that it gives a solitary system that can be

prepared to unravel the source and target text. Subsequently, it doesn't rely upon

specific systems that are regular to other machine translation systems, particularly

SMT.

Advantages of Machine Translation

 Fast: Provides almost instant translations.

 Scale: Capable of handling vast amounts of content very quickly.
 Flexible: One system is capable of translating content into multiple languages.
 Usability: Easy for language professionals and everyday users alike.
 Integration: Language pros can use machine translation to speed up their workflow.
 Automation: Automating the first stage of translation makes the whole process
faster and more affordable to the end client.
Problems in Machine Translation

 Quality: Even the best AI translation tools are far away from matching the quality of
professional translators.
 Consistency: Quality varies greatly depending on the complexity of input language
and the linguistic distance between the source and target languages.
 Word-for-word output: Despite improvements, algorithms still produce outputs
largely consisting of word-for-word translations.
 Grammar: Although this is one of the biggest areas of improvement in recent years,
grammar remains a challenge for machine translation, especially between languages
with significantly different grammar systems.
 Context: Again, AI technologies have dramatically improved contextual
understanding but the end results are far from matching human capabilities.
 Nuance: Algorithms struggle to determine and replicate the nuances of human
language.
Explain about Cross Lingual Information Retrieval (CLIR)
Cross-lingual Information Retrieval is the task of retrieving relevant information when the
document collection is written in a different language from the user query.

Translation Approaches

CLIR requires the ability to represent and match information in the same representation

space even if the query and the document collection are in different languages.

In CLIR, this translation process can be in several ways.

 Document translation is to map the document representation into the query

representation space.
 Query translation is to map the query representation into the document

representation space.

 Pivot language or Interlingua is to map both document and query representations to

a third space.
Explain about Multilingual Information Retrieval
Multilingual Information Retrieval (MLIR) refers to the ability to process a query for
information in any language, search a collection of objects, including text, images,
sound files, etc., and return the most relevant objects, translated if necessary into the
user's language.
Explain about Latent Semantic analysis?
Latent Semantic Analysis is a natural language processing method that uses the statistical
approach to identify the association among the words in a document.
Singular Value Decomposition is the statistical method that is used to find the
latent(hidden) semantic structure of words spread across the document.
Let
C = collection of documents.
d = number of documents.
n = number of unique words in the whole collection.
M=dXn
The SVD decomposes the M matrix i.e word to document matrix into three matrices as follows
M = U∑VT
where
U = distribution of words across the different contexts
∑ = diagonal matrix of the association among the contexts
VT = distribution of contexts across the different documents
Explain about Text Rank algorithm
TextRank – is a graph-based ranking model for text processing which can be used in
order to find the most relevant sentences in text and also to find keywords.

Page Rank:
 PageRank (PR) is an algorithm used by Google Search to rank websites in their search
engine results.
 PageRank was named after Larry Page, one of the founders of Google.
 PageRank is a way of measuring the importance of website pages.
 The formula for calculating the page rank is

PR(p_i, t) → Page rank at t^th iterations for i^th webpage.

d → Damping Factor (Way to do teleportation)
L→ length of outgoing links
N→ length of webPages

Page Rank Algorithm

1. Initialise a vector “V” where all element is equal to 1 and size is equal to the number
of nodes. and also define no of iteration “Iter”
2. Normalize Vector “V”.
3. Take the damping factor value like “0.85”
4. Compute the PageRank of each node by the above formula.
5. repeat step 4 for a given number of iterations “Iter”..

Text Rank Algorithm:

1. Take a document with n sentence.
2. compute embedding of each sentence using Tf-IDF, Bert, etc.
3. calculate the similarity between each sentence, which will be nothing but a matrix M
which you see above in implementation.
4. Normalise the cosine Matrix M, so that each row sum equals one.
5. Then use page rank algorithms to compute the rank of each sentence.
Explain about Heap’s Law and Zipf’s Law
Statistical Properties of terms in Information Retrieval
1. Heap’s Law: Estimating the number of terms
2. Zipf’s Law: modeling the distribution of terms
Heap’s Law:
The law can be described like as the number of words in a document increases, the rate of
the count of distinct words available in the document slows down.
The documented definition of Heaps’ law (also called Herdan's law) says that the
number of unique words in a text of n words is approximated by
V(n) = K n^β
where K is a positive constant and
β is between 0 and 1.
K is often upto 100 and
β is often between 0.4 and 0.6.

Zipf’s Law:
Zipf's law is a relation between rank order and frequency of occurrence: it states that
when observations (e.g., words) are ranked by their frequency, the frequency of a particular
observation is inversely proportional to its rank, Frequency ∝ 1 Rank .
Application of zipf’s law:
Zipf's Law provides a distributional foundation for models of the language learner's
exposure to segments, words and constructs, and permits evaluation of learning models
Graph:

Relation between Heap’s Law and Zipf’s Law

The relation of Zipf's law with the Heaps' law is observed like as the length of the document
(words in the document) keeps on increasing then after certain point we can see that no much
unique words are added to the vocabulary.