Index

1. Department Vision and Mission

2. Course Objectives & Course Outcome
3. Scheme & Syllabus
4. Text Book & Reference Book
5. Unit - 5 Information

1
Amit Pimpalkar
 Department Vision and Mission

Our Vision
• To Continually improve the education environment, in order to develop
graduates with strong academic and technical background needed to
achieve distinction in the discipline. The excellence is expected in various
domains like workforce, higher studies or lifelong learning. To strengthen
links between industry through partnership and collaborative
development works.

Our Mission
• To develop strong foundation of theory and practices of computer science
amongst the students to enable them to develop into knowledge,
responsible professionals, lifelong learners and implement the latest
computing technologies for the betterment of the society.
2
Amit Pimpalkar
 Course Objectives & Course Outcome
Course Objective Course Outcomes
I. To familiarize the concepts and After the completion of the course, student will be able
techniques of Natural language to
Processing for analyzing words 1. Apply the Principles and Process of Human
based on Morphology and Languages using computers.
CORPUS. 2. Demonstrate the state-of-the-art algorithms and
II. To relate mathematical techniques for text-based processing of natural
foundations, Probability theory languages with respect to morphology.
with Linguistic essentials such 3. Perform POS tagging for a given natural language
as syntactic and semantic 4. Create Linguistics CORPUS based on Text Corpus
analysis of text. method
III. To apply the Statistical learning 5. Realize semantics and pragmatics of natural
methods and cutting-edge languages for text processing
research models to solve NLP 6. Develop a Statistical Methods for Real World NLP
problems Applications.
3
Amit Pimpalkar
 Scheme & Syllabus
Load Credit Total Mark Continuous Assessment ESE Mark
3 hrs (Theory) + 0 hr (Tutorial) 3 100 40 60

UNIT I: Introduction to NLP, Morphology: Introduction to NLP, Stages of NLP,

Ambiguity, Information Theory Essentials , Linguistic Essentials : Parts of Speech and
Morphology, Morphological analysis and generation using Finite State Automata and
Finite State transducer.
UNIT II: Markov Model and POS Tagging: Markov Model: Hidden Markov model,
Fundamentals, Probability of properties, Parameter estimation, Variants, Multiple input
observation. The Information Sources in Tagging: Markov model taggers, Viterbi
algorithm, Applying HMMs to POS tagging, Applications of Tagging.
UNIT 3: Syntax and Semantics: Shallow Parsing and Chunking, Shallow Parsing with
Conditional Random Fields (CRF), Lexical Semantics, WordNet, Thematic Roles,
Semantic Role Labelling with CRFs.
4
Amit Pimpalkar
 Scheme & Syllabus
Load Credit Total Mark Continuous Assessment ESE Mark
3 hrs (Theory) + 0 hr (Tutorial) 3 100 40 60
UNIT IV: Language Modelling: Corpus based work, Statistical Inference: n -gram Models
over Sparse Data, Methodological Preliminaries, Supervised Disambiguation: Bayesian
classification, An information- theoretic approach, Dictionary-Based Disambiguation:
Disambiguation based on sense, Thesaurus-based disambiguation, Disambiguation based on
translations in a second-language corpus.
UNIT V: Probabilistic Parsing and Disambiguation: Probabilistic Context Free Grammars
and Probabilistic parsing The Probability of a String, Problems with the Inside-Outside
Algorithm, Parsing for disambiguation, Treebanks, Parsing models vs. language models, Phrase
structure grammars and dependency, Lexicalized models using derivational histories,
Dependency-based models.
UNIT VI: NLP Applications : Statistical Alignment and Machine Translation, Text alignment,
Word alignment, Information extraction, Text mining, Information Retrieval, NL interfaces,
Sentimental Analysis, Question Answering Systems, Social network analysis.
5
Amit Pimpalkar
 Text & Reference Book

Text Books:

1. Christopher D. Manning and Hinrich Schutze, “Foundations of Natural Language

Processing”, 6th Edition, The MIT Press Cambridge, Massachusetts London, England,
2003
2. Daniel Jurafsky and James H. Martin “Speech and Language Processing”, 3rd edition,
Prentice Hall, 2009.

Reference Books:

1. James Allen “Natural Language Understanding”, Pearson Publication 8th Edition.

2012.

6
Amit Pimpalkar
 Text & Reference Book

7
Amit Pimpalkar
 UNIT 5: Probabilistic Parsing and Disambiguation

Probabilistic Context Free Grammars and Probabilistic parsing

The Probability of a String,
Problems with the Inside-Outside Algorithm,
Parsing for disambiguation,
Treebanks,
Parsing models vs. language models,
Phrase structure grammars and dependency,
Lexicalized models using derivational histories,
Dependency-based models

8
Amit Pimpalkar
 Remember Unit 3: Parsing (Top-down Parsing)
Grammar

S →NP VP
NP → N N
VP →V NP
NP → N

Lexicon

Fed : N
interest : N, V
rates : N
raises : N, V
 UNIT 5: Probabilistic CFGs
Handling Ambiguities
• The ambiguities handling algorithms are equipped to represent ambiguities
efficiently but not to resolve them.
• Methods available for resolving ambiguities include:
• Semantics (choose parse that makes sense).
• Statistics: (choose parse that is most likely).
• Probabilistic context-free grammars (PCFGs) offer a solution.

10
Amit Pimpalkar
 UNIT 5: Probabilistic CFGs
• A context-free grammar is a tuple <N, T, S, R>
• N : the set of non-terminals
• Phrasal categories: S, NP, VP, ADJ, etc.
• Parts-of-speech (pre-terminals): NN, JJ, DT, VB
• T : the set of terminals (the words)
• S : the start symbol
• Often written as ROOT or TOP
• Not usually the sentence non-terminal S
• R : the set of rules
• Of the form X  Y1 Y2 … Yk, with X, Yi  N
• Examples: S  NP VP, VP  VP CC VP
• Also called rewrites, productions, or local trees
• A PCFG adds:
• A top-down production probability per rule P(Y1 Y2 … Yk | X)
11
Amit Pimpalkar
 UNIT 5: Probabilistic CFGs
• The probabilistic model
• Assigning probabilities to parse trees
• Getting the probabilities for the model
• Parsing with probabilities
• Slight modification to dynamic programming approach
• Task is to find the max probability tree for an input
• Getting the Probabilities
• From an annotated database (a treebank)
• Learned from a corpus
• Assume PCFG is in Chomsky Normal Form
• (production is either A → B C or A → a)
12
Amit Pimpalkar
 UNIT 5: Probabilistic CFGs

Chomsky Normal Form (CNF)

All rules have form

A  BC and Aa

Non-Terminal Non-Terminal terminal

13
Amit Pimpalkar
 UNIT 5: Some Features of PCFGs

• A PCFG gives some idea of the possibility of different parses.

• However, the probabilities are based on structural factors and not lexical ones.
• PCFG are good for grammar induction.
• PCFGs are robust.
• PCFGs give a probabilistic language model for English.
• The predictive power of a PCFG (measured by entropy) tends to be greater
than for an HMM.
• PCFGs are not good models alone but they can be combined with a tri-gram
model.
• PCFGs have certain biases which may not be appropriate.

14
Amit Pimpalkar
 UNIT 5: Probabilistic CFGs
Examples:

S  AS S  AS
S a S  AAS
A  SA A  SA
Ab A  aa

Chomsky Normal Form Not Chomsky Normal Form ???

15
Amit Pimpalkar
 UNIT 5: A Simple PCFG (in CNF)
S  NP VP 1.0 NP  NP PP 0.4
VP  V NP 0.7 NP  astronomers 0.1
VP  VP PP 0.3 NP  ears 0.18
PP  P NP 1.0 NP  saw 0.04
P  with 1.0 NP  stars 0.18
V  saw 1.0 NP  telescope 0.1

astronomers saw stars with ears

Draw all parse tree for this sentence, (Top Down)

16
Amit Pimpalkar
 UNIT 5: A Simple PCFG (in CNF)
S  NP VP 1.0 NP  NP PP 0.4
VP  V NP 0.7 NP  astronomers 0.1
VP  VP PP 0.3 NP  ears 0.18
PP  P NP 1.0 NP  saw 0.04
P  with 1.0 NP  stars 0.18
V  saw 1.0 NP  telescope 0.1

astronomers saw stars with ears

17
Amit Pimpalkar
 UNIT 5: A Simple PCFG (in CNF)
Tree and String Probabilities S  NP VP 1.0 NP  NP PP 0.4
VP  V NP 0.7 NP  astronomers 0.1
VP  VP PP 0.3 NP  ears 0.18
PP  P NP 1.0 NP  saw 0.04
P  with 1.0 NP  stars 0.18
V  saw 1.0 NP  telescope 0.1

• w12 = astronomers saw stars with ears

• P(t1) = 1.0 * 0.1 * 0.7 * 1.0 * 0.4 * 0.18
* 1.0 * 1.0 * 0.18
= 0.0009072
• P(t2) = 1.0 * 0.1 * 0.3 * 0.7 * 1.0 * 0.18
* 1.0 * 1.0 * 0.18
= 0.0006804
• P(w12) = P(t1) + P(t2)
= 0.0009072 + 0.0006804
= 0.0015876 18
Amit Pimpalkar
 UNIT 5: A Simple PCFG (in CNF)

Grammar Prob Lexicon

S → NP VP 0.8 Det → the | a | that | this
S → Aux NP VP 0.1 + 1.0
0.6 0.2 0.1 0.1
S → VP 0.1 Noun → book | flight | meal | money
NP → Pronoun 0.2 0.1 0.5 0.2 0.2
+ 1.0
NP → Proper-Noun 0.2 Verb → book | include | prefer
NP → Det Nominal 0.6 0.5 0.2 0.3
Nominal → Noun 0.3 + 1.0 Pronoun → I | he | she | me
Nominal → Nominal Noun 0.2 0.5 0.1 0.1 0.3
Nominal → Nominal PP 0.5 Proper-Noun → Houston | NWA
VP → Verb 0.2 + 1.0 0.8 0.2
VP → Verb NP 0.5 Aux → does
VP → VP PP 0.3 1.0
PP → Prep NP 1.0 + 1.0 Prep → from | to | on | near | through
0.25 0.25 0.1 0.2 0.2

Amit Pimpalkar book the flight through Houston 19

 UNIT 5: A Simple PCFG (in CNF)
• Assume productions for each node are chosen independently.
• Probability of derivation is the product of the probabilities of its productions.
S D1
P(D1) = 0.1 x 0.5 x 0.5 x 0.6 x 0.6 x 0.5 x 0.1
0.3 x 1.0 x 0.2 x 0.2 x 0.5 x 0.8 VP
0.5
= 0.0000216
Verb NP 0.6
0.5
book Det Nominal
0.6 0.5
book the flight through Houston the Nominal PP 1.0
0.3
Noun Prep NP
0.2
0.5 0.2
flight through Proper-Noun
0.8
Houston
20
Amit Pimpalkar
 UNIT 5: A Simple PCFG (in CNF)
• Resolve ambiguity by picking most probable parse tree.
book the flight through Houston
S
P(D2) = 0.1 x 0.3 x 0.5 x 0.6 x 0.5 x 0.6 x D2
0.1
0.3 x 1.0 x 0.5 x 0.2 x 0.2 x 0.8
= 0.00001296 VP
0.3
VP 0.5
• Probability of a sentence is the sum of the
probabilities of all of its derivations. Verb NP 0.6
0.5
P(“book the flight through Houston”) book Det Nominal
PP
1.0
0.6
= P(D1) + P(D2) 0.3
the Noun Prep NP
= 0.0000216 + 0.00001296 0.2 0.2
0.5
flight through Proper-Noun
= 0.00003456 0.8
Houston
21
Amit Pimpalkar
 UNIT 5: A Simple PCFG (in CNF)

Amit Pimpalkar can you book TWA flights 22

 UNIT 5: A Simple PCFG (in CNF)
can you book TWA flights

P(S) = 3.2 x 10-6

23
Amit Pimpalkar
 UNIT 5: A Simple PCFG (in CNF)

24
Amit Pimpalkar
 UNIT 5: Example of Inside Probabilities (CYK Algorithm)
S  NP VP 1.0 NP  NP PP 0.4 John Cocke, Daniel Younger, Tadao
VP  V NP 0.7 NP  astronomers 0.1
Kasami, and Jacob Schwartz (1961)
VP  VP PP 0.3 NP  ears 0.18
PP  P NP 1.0 NP  saw 0.04
P  with 1.0 NP  stars 0.18 astronomers saw stars with ears
V  saw 1.0 NP  telescope 0.1

1 2 3 4 5
1 NP = 0.1
2 NP = 0.04
V = 1.0
3 NP = 0. 18
4 P = 1.0
5 NP = 0. 18
astronomers saw stars with ears

25
Amit Pimpalkar
 UNIT 5: Example of Inside Probabilities (CYK Algorithm)
S  NP VP 1.0 NP  NP PP 0.4
VP  V NP 0.7 NP  astronomers 0.1 astronomers saw stars with ears
VP  VP PP 0.3 NP  ears 0.18
PP  P NP 1.0 NP  saw 0.04
P  with 1.0 NP  stars 0.18
V  saw 1.0 NP  telescope 0.1

1 2 3 4 5
1 NP = 0.1 ----
2 NP = 0.04 VP = 0.126
V = 1.0 (1 x 0.7 x 0. 18)
3 NP = 0. 18
4 P = 1.0
5 NP = 0. 18
astronomers saw stars with ears

26
Amit Pimpalkar
 UNIT 5: Example of Inside Probabilities (CYK Algorithm)
S  NP VP 1.0 NP  NP PP 0.4
VP  V NP 0.7 NP  astronomers 0.1 astronomers saw stars with ears
VP  VP PP 0.3 NP  ears 0.18
PP  P NP 1.0 NP  saw 0.04
P  with 1.0 NP  stars 0.18
V  saw 1.0 NP  telescope 0.1

1 2 3 4 5
1 NP = 0.1 ---- S = 0.0126
(0.1 x 1.0 x 0.126 )
2 NP = 0.04 VP = 0.126
V = 1.0 (1 x 0.7 x 0. 18)
3 NP = 0. 18
4 P = 1.0
5 NP = 0. 18
astronomers saw stars with ears
27
Amit Pimpalkar
 UNIT 5: Example of Inside Probabilities (CYK Algorithm)
S  NP VP 1.0 NP  NP PP 0.4
VP  V NP 0.7 NP  astronomers 0.1
astronomers saw stars with ears
VP  VP PP 0.3 NP  ears 0.18
PP  P NP 1.0 NP  saw 0.04
P  with 1.0 NP  stars 0.18
V  saw 1.0 NP  telescope 0.1

1 2 3 4 5
1 NP = 0.1 S = 0.0126 ----
(0.1 x 1.0 x 0.126 )
2 NP = 0.04 VP = 0.126 ----
V = 1.0 (1 x 0.7 x 0. 18)
3 NP = 0. 18 ----
4 P = 1.0
5 NP = 0. 18
astronomers saw stars with ears
28
Amit Pimpalkar
 UNIT 5: Example of Inside Probabilities (CYK Algorithm)
S  NP VP 1.0 NP  NP PP 0.4
VP  V NP 0.7 NP  astronomers 0.1
astronomers saw stars with ears
VP  VP PP 0.3 NP  ears 0.18
PP  P NP 1.0 NP  saw 0.04
P  with 1.0 NP  stars 0.18
V  saw 1.0 NP  telescope 0.1

1 2 3 4 5
1 NP = 0.1 S = 0.0126 ----
(0.1 x 1.0 x 0.126 )
2 NP = 0.04 VP = 0.126 ----
V = 1.0 (1 x 0.7 x 0. 18)
3 NP = 0. 18 ----
4 P = 1.0 PP = 0. 18
(1.0 x 1.0 x 0.18)
5 NP = 0. 18
astronomers saw stars with ears

29
Amit Pimpalkar
 UNIT 5: Example of Inside Probabilities (CYK Algorithm)
S  NP VP 1.0 NP  NP PP 0.4
VP  V NP 0.7 NP  astronomers 0.1
astronomers saw stars with ears
VP  VP PP 0.3 NP  ears 0.18
PP  P NP 1.0 NP  saw 0.04
P  with 1.0 NP  stars 0.18
V  saw 1.0 NP  telescope 0.1

1 2 3 4 5
1 NP = 0.1 S = 0.0126 ---- S = 0.0015876
(0.1 x 1.0 x 0.126 ) (ref next slide for calculation)
2 NP = 0.04 VP = 0.126 ---- VP = 0.015876
V = 1.0 (1 x 0.7 x 0. 18) (ref next slide for calculation)

3 NP = 0. 18 ---- NP = 0.01296

(0.18 x 0.4 x 0.18)
4 P = 1.0 PP = 0. 18
(1.0 x 0.18)
5 NP = 0. 18
astronomers saw stars with ears
30
Amit Pimpalkar
 UNIT 5: Example of Inside Probabilities (CYK Algorithm)
 
S
VP 
NP VP
V NP
1.0
0.7
NP
NP 
NP PP 0.4
astronomers 0.1
astronomers1 saw2 stars3
VP  VP PP 0.3 NP  ears 0.18 with4 ears5
PP  P NP 1.0 NP  saw 0.04
P  with 1.0 NP  stars 0.18 Overall probability of the sentence:
V  saw 1.0 NP  telescope 0.1 P(w15)=P(t1)+P(t2)=0.0015876
Refer to Slide
Number 18 to
cross check

31
Amit Pimpalkar
 UNIT 5: Example of Inside Probabilities (CYK Algorithm)

S  NP VP 0.8 V  includes 0.05

NP  DT N 0.3 DT  the | a 0.4
VP  V NP 0.2 N  price 0.01
N  facemask 0.02

the price includes a facemask

32
Amit Pimpalkar
 UNIT 5: Example of Inside Probabilities (CYK Algorithm)
S  NP VP 0.8 V  includes 0.05
NP  DT N 0.3 DT  the | a 0.4
VP  V NP 0.2 N  meals 0.01
N  flight 0.02

the flight includes the meals

33
Amit Pimpalkar
 UNIT 5: Example of Inside Probabilities (CYK Algorithm)
S  NP VP 1.0 Vi  sleeps 1.0
VP  Vi 0.3 Vt  saw 1.0
VP  Vt NP 0.5 NN  man | woman 0.1
VP  VP PP 0.2 NN  telescope 0.3
NP  DT NN 0.8 NN  dog 0.5
NP  NP PP 0.2 DT  the 1.0
PP  IN NP 1.0 IN  with 0.6
IN  in 0.4

the dog saw the man with the telescope

34
Amit Pimpalkar
 UNIT 5: Example of Inside Probabilities (CYK Algorithm)

Grammar Prob Lexicon

S → NP VP 0.8 Det → the | a | that | this
S → Aux NP VP 0.1 + 1.0
0.6 0.2 0.1 0.1
S → VP 0.1 Noun → book | flight | meal | money
NP → Pronoun 0.2 0.1 0.5 0.2 0.2
+ 1.0
NP → Proper-Noun 0.2 Verb → book | include | prefer
NP → Det Nominal 0.6 0.5 0.2 0.3
Nominal → Noun 0.3 + 1.0 Pronoun → I | he | she | me
Nominal → Nominal Noun 0.2 0.5 0.1 0.1 0.3
Nominal → Nominal PP 0.5 Proper-Noun → Houston | NWA
VP → Verb 0.2 + 1.0 0.8 0.2
VP → Verb NP 0.5 Aux → does
VP → VP PP 0.3 1.0
PP → Prep NP 1.0 + 1.0 Prep → from | to | on | near | through
0.25 0.25 0.1 0.2 0.2

Amit Pimpalkar book the flight through Houston 35

 UNIT 5: Example of Inside Probabilities (CYK Algorithm)

Amit Pimpalkar can you book TWA flights 36

 UNIT 5: Problems with the inside-outside algorithm
• Remember
• CYK algorithm is for recognition (valid/invalid string based on grammar),
while Inside algorithm delves deeper, analyzing all possible sub-structures
within a string based on the grammar.

• Extremely Slow: For each sentence, each iteration of training is O(m3n3).

• Local Maxima are much more of a problem than in HMMs
• Satisfactory learning requires many more nonterminal than are theoretically
needed to describe the language.
• There is no guarantee that the learned nonterminal will be linguistically
motivated.
37
Amit Pimpalkar
 UNIT 5: Treebanks

• English Penn Treebank: Standard corpus for testing syntactic parsing consists
of 1.2 M words of text from the Wall Street Journal (WSJ).
• Typical to train on about 40,000 parsed sentences and test on an additional
standard disjoint test set of 2,416 sentences.
• Chinese Penn Treebank: 100K words from the Xinhua news service.
• Other corpora existing in many languages, see the Wikipedia article
“Treebank”
• Treebanks used in improving machine translation systems

38
Amit Pimpalkar
 Other Resources to study NLP
• https://onlinecourses.nptel.ac.in/noc21_cs19
• By Prof. Sourav Mukhopadhyay | IIT Kharagpur
• https://onlinecourses.nptel.ac.in/noc20_cs87
• By Prof. Ramaseshan R | Chennai Mathematical Institute (CMI)
• https://docs.microsoft.com/en-us/learn/paths/explore-natural-language-
processing/
• https://www.upgrad.com/machine-learning-nlp-pgc-iiitb
• https://www.amazon.science/latest-news/machine-learning-course-free-
online-from-amazon-machine-learning-university
• https://online.stanford.edu/courses/xcs224n-natural-language-processing-
deep-learning
• https://courses.cs.washington.edu/courses/cse517/
Amit Pimpalkar
39
 Other Resources to study NLP
• https://cse.iitk.ac.in/users/cs671/2013/resources.html
• https://home.cs.colorado.edu/~martin/slp.html
• https://web.stanford.edu/~jurafsky/NLPCourseraSlides.html
• https://nlp.stanford.edu/teaching/
• https://tildesites.bowdoin.edu/~allen/nlp/
• https://nlp-iiith.vlabs.ac.in/
• www.purenlp.com
• https://www.nlpworks.com/
• www.compendiumdev.co.uk/nlp

40
Amit Pimpalkar
 Other Resources to study NLP
• Natural Language Processing, IIT Kharagpur
• Prof. Pawan Goyal
• https://nptel.ac.in/courses/106105158 (Enrolment Opens: 2023-11-09 to 2024-01-29)

• Natural Language Processing, IIT Bombay

• Prof. Pushpak Bhattacharyya
• https://nptel.ac.in/courses/106101007 (Completed and Videos are available for self-learning)

• Applied Natural Language Processing, Chennai Mathematical Institute

• Prof. Ramaseshan R
• https://nptel.ac.in/courses/106106211 (Completed and Videos are available for self-learning)

• Principles and Parameters in Natural Language, IIT Madras

• Prof. Rajesh Kumar
• https://nptel.ac.in/courses/109106083(Completed and Videos are available for self-learning)
41
Amit Pimpalkar
 End of Unit-5

Amit Pimpalkar
Thank You ! 42
 43
Amit Pimpalkar