Probabilisti

c Theory in
Natural
Language Presented By:
Jenil Pavagadhi (21BIT177)

Processing Hepin Ramani (21BIT174)

Maulik Shah (21BIT264)
Agenda
1 Introduction and Fundamentals
We'll begin with an introduction to probabilistic models in NLP and explore the importance of probability in
language processing. This will be followed by a review of fundamental probability concepts essential for
understanding NLP applications.

2 Core Probabilistic Models

Next, we'll dive into Bayesian inference and its applications in NLP. We'll then explore language modeling
techniques, including N-grams and Markov models, which form the backbone of many NLP systems.

3 Advanced Topics and Applications

Finally, we'll examine more advanced probabilistic models such as Hidden Markov Models (HMMs) and
probabilistic parsing. We'll conclude by discussing real-world applications and challenges in implementing
probabilistic NLP systems.
Introduction to Probabilistic Models in NLP

1 Handling Uncertainty 2 Data-Driven Approach 3 Wide Range of

Applications
Probabilistic models in NLP are These models assign
designed to handle the probabilities to words, Probabilistic models are used
inherent uncertainty and sentences, or structures based in various NLP tasks, including
ambiguity in natural language. on large-scale analysis of speech recognition, machine
They provide a framework for language data. This data- translation, text generation,
making decisions based on driven approach allows NLP and sentiment analysis. They
incomplete or noisy systems to learn patterns and form the foundation for many
information, which is common relationships in language state-of-the-art language
in real-world language data. without explicit programming technologies we use daily.
of linguistic rules.
Motivation: Why Probability is Crucial in NLP
Ambiguity Resolution Handling Noise and Variation Learning from Data

Natural language is inherently Real-world language data often Probabilistic models excel at learning
ambiguous at multiple levels - lexical, contains noise, errors, and variations. patterns from large datasets. This
syntactic, and semantic. Probabilistic Probabilistic models are robust to data-driven approach allows NLP
models provide a principled way to such imperfections, making them systems to capture subtle linguistic
resolve these ambiguities by ideal for tasks like speech recognition nuances and adapt to different
assigning likelihoods to different or processing user-generated content. domains or languages without
interpretations. For instance, in the They can account for spelling extensive manual rule-writing. It's
sentence "I saw the man with the mistakes, dialectal variations, and particularly valuable in multilingual
telescope," a probabilistic model can even transcription errors in a and cross-domain NLP applications.
help determine whether "with the systematic way.
telescope" modifies "saw" or "man"
based on contextual probabilities.
Fundamental Concepts in Probability
Theory (I)

Random Variables Probability Distribution Joint Probability

In NLP, random variables often A probability distribution describes Joint probability, denoted as P(A, B),
represent linguistic units like words, the likelihood of different outcomes represents the likelihood of two
phrases, or sentences. They can for a random variable. In NLP, this events occurring together. In NLP,
take on different values from a could be the distribution of words in this concept is vital for
defined set, allowing us to model a language or the probability of understanding co-occurrences of
the uncertainty in language. For different parse trees for a sentence. words or linguistic phenomena. For
example, a random variable might Understanding these distributions is instance, the joint probability of
represent the next word in a crucial for tasks like language "New York" appearing in text helps
sentence, with its possible values modeling and syntactic analysis. in named entity recognition and
being all words in the vocabulary. collocation analysis.
Fundamental Concepts in Probability
Theory (II)
Concept Definition NLP Application

Conditional Probability P(A|B): Probability of A given B has Next word prediction, part-of-
occurred speech tagging

Chain Rule P(A,B,C) = P(A|B,C) * P(B|C) * P(C) Computing probability of word

sequences

Bayes' Theorem P(A|B) = (P(B|A) * P(A)) / P(B) Text classification, sentiment

analysis

These fundamental concepts form the backbone of probabilistic NLP. Conditional probability is crucial in context-
dependent tasks, while the chain rule allows us to compute probabilities of sequences, essential in language modeling.
Bayes' theorem, a cornerstone of probabilistic reasoning, enables us to update our beliefs based on new evidence,
making it invaluable in classification tasks and probabilistic inference in NLP.
Bayesian Inference in NLP
Prior Probability
Start with initial beliefs about language phenomena, based on domain knowledge or
previous data.

Likelihood
Observe new data and calculate how likely it is under different hypotheses.

Posterior Probability
Update beliefs by combining prior knowledge with new evidence using Bayes' theorem.

Decision
Make informed decisions based on updated probabilities, improving NLP task
performance.

Bayesian inference is a powerful framework in NLP, allowing systems to learn and adapt from data.
It's particularly useful in tasks like spam detection, where the system can update its understanding
of spam characteristics over time. In machine translation, Bayesian methods help in selecting the
most probable translation by considering both language model probabilities and translation model
likelihoods.
Language Modeling Using N-grams
N-gram Definition Markov Assumption
An n-gram is a contiguous sequence of n items from N-gram models rely on the Markov assumption,
a given text. In NLP, these items are typically words which states that the probability of a word depends
or characters. N-gram models predict the probability only on a fixed number of preceding words. This
of a word given its n-1 preceding words. simplification makes language modeling
computationally tractable.

Probability Calculation Applications

The probability of a sequence is calculated using the N-gram models are used in various NLP tasks,
chain rule of probability, multiplying the conditional including predictive text input, speech recognition,
probabilities of each word given its preceding words. machine translation, and spelling correction. They
These probabilities are estimated from large text provide a simple yet effective way to capture local
corpora. context in language.
Challenges in N-gram Modeling

Data Sparsity Smoothing Techniques Computational Efficiency

One of the main challenges in n-gram To address data sparsity, various As n increases, storing and computing
modeling is data sparsity. As n increases, smoothing techniques are employed. probabilities for all n-grams becomes
the number of possible n-grams grows These methods redistribute probability computationally expensive. Efficient
exponentially, making it impossible to mass to unseen events, ensuring non- data structures and pruning techniques
observe all combinations in training zero probabilities for all possible n- are necessary to manage large-scale n-
data. This leads to unreliable probability grams. Common approaches include gram models in practical NLP
estimates for rare or unseen n-grams. Laplace smoothing, Good-Turing applications.
estimation, and interpolation methods.
Markov Models in NLP
Applications
Markov Chain
Markov models are used in speech recognition,
A Markov chain is a sequence of states where named entity recognition, and machine
the probability of each state depends only on translation. They provide a probabilistic
the previous state. In NLP, states often framework for modeling sequences and
represent words or linguistic units. transitions in language.

1 2 3 4

Hidden Markov Model (HMM) Limitations and Extensions

HMMs extend Markov chains by introducing While powerful, Markov models are limited by
hidden states. These models are crucial in tasks their fixed-order dependency assumption. More
like part-of-speech tagging, where observed advanced models like Maximum Entropy Markov
words are generated by hidden grammatical Models (MEMMs) and Conditional Random Fields
states. (CRFs) address some of these limitations.
Hidden Markov Models (HMMs)
Hidden Markov Models (HMMs) are powerful probabilistic models used extensively in Natural Language Processing.
These models consist of a sequence of hidden states and observed outputs, where the hidden states represent
underlying linguistic structures, such as parts of speech, while the observed outputs are the actual words we see in
a sentence.
HMMs are particularly effective in tasks like Part-of-Speech (POS) tagging and Named Entity Recognition (NER). In
POS tagging, for instance, the model assigns grammatical categories to words based on their context and probability
distributions learned from training data.

1 2 3

Input Sequence Hidden State Prediction Output Tags

The sentence "He will race The HMM predicts the most likely The model outputs the predicted
tomorrow" is input to the HMM. sequence of hidden states (parts tags: Pronoun, Auxiliary Verb,
of speech) for each word. Verb, Adverb.
Probabilistic Context-Free Grammars
(PCFGs)
Probabilistic Context-Free Grammars (PCFGs) extend traditional Context-Free Grammars (CFGs) by assigning
probabilities to production rules, resolving ambiguities in sentence structure.

PCFGs are particularly useful in parsing complex sentences and determining the most likely syntactic structure.

Grammar Rules Ambiguity Resolution

PCFGs define probabilistic rules for sentence PCFGs resolve structural ambiguities by selecting the
structure, like S → NP VP (0.8) | VP (0.2). parse with the highest probability.

Training Applications
PCFGs learn probabilities from large annotated Used in syntactic parsing, language modeling, and in
corpora. more complex NLP systems.
Applications in NLP (I) - Speech Recognition
Automatic Speech Recognition (ASR) converts spoken language to text using probabilistic models. These models analyze
speech patterns, phonetic sequences, and language models to predict the most likely word sequence.
ASR systems often combine HMMs with neural networks to improve accuracy. These hybrid models account for
pronunciation, variations, accents, and noise.

Audio Input
The system receives an audio signal of spoken words.

Feature Extraction
The signal is processed to extract relevant acoustic features.

Acoustic Modeling
HMMs model the relationship between acoustic features and phonemes.

Language Modeling

Probabilistic language models predict likely word sequences.

Text Output
The system produces the most probable text transcription.
Applications in NLP (II) - Machine
Translation
Machine Translation (MT) automates text translation between languages using probabilistic models. These models
calculate word alignments and sentence probabilities across different languages, enabling the system to determine
the most likely translation for a given input sentence. Statistical Machine Translation (SMT) systems rely heavily on
probabilistic models, while more advanced Neural Machine Translation (NMT) approaches incorporate deep learning
techniques but still leverage probabilistic foundations for sequence modeling and attention mechanisms.

Source Language Target Language Translation Probability

"I love cats" "J'aime les chats" 0.75

"I love cats" "J'adore les chats" 0.20

"I love cats" "Je suis un chat" 0.05

Current Trends and Future Directions
The field of probabilistic models in NLP is rapidly evolving, with current trends focusing on the integration of deep learning
techniques with traditional probabilistic approaches. Models like BERT (Bidirectional Encoder Representations from Transformers)
utilize probabilistic attention mechanisms to capture contextual information in text, significantly improving performance on
various NLP tasks.

Looking to the future, researchers are exploring ways to combine the strengths of deep learning models with the interpretability
and data efficiency of probabilistic techniques. This hybrid approach aims to address challenges such as data sparsity and model
complexity while improving accuracy and generalization across diverse language tasks.

Neural-Probabilistic Data-Efficient Multilingual Models Adaptive Systems

Fusion Learning
Creating unified probabilistic Building models that can
Combining neural networks Developing techniques to frameworks that can handle continuously update their
with probabilistic graphical train robust models with multiple languages and probabilistic estimates based
models for improved limited data, especially for transfer knowledge between on new data and user
performance and low-resource languages. them. feedback.
interpretability.