0% found this document useful (0 votes)

160 views454 pages

Raymond S. T. Lee - Natural Language Processing. A Textbook With Python Implementation-Springer (2024)

Uploaded by

diegohrey239

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

160 views454 pages

Raymond S. T. Lee - Natural Language Processing. A Textbook With Python Implementation-Springer (2024)

Uploaded by

diegohrey239

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 454

Raymond S. T.

Lee

Natural
Language
Processing
A Textbook with Python
Implementation
Natural Language Processing
Raymond S. T. Lee

Natural Language Processing

A Textbook with Python Implementation
Raymond S. T. Lee
United International College
Beijing Normal University-Hong Kong Baptist University
Zhuhai, China

ISBN 978-981-99-1998-7 ISBN 978-981-99-1999-4 (eBook)

https://doi.org/10.1007/978-981-99-1999-4

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore
Pte Ltd. 2024
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmission or information storage and retrieval, electronic adaptation, computer software, or by similar
or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
This book is dedicated to all readers and
students taking my undergraduate and
postgraduate courses in Natural Language
Processing, your enthusiasm in seeking
knowledge incited me to write this book.
Preface

Motivation of This Book

Natural Language Processing (NLP) and its related applications become part of
daily life with exponential growth of Artificial Intelligence (AI) in past decades.
NLP applications including Information Retrieval (IR) systems, Text Summarization
System, and Question-and-Answering (Chatbot) System became one of the preva-
lent topics in both industry and academia that had evolved routines and benefited
immensely to a wide array of day-to-day services.
The objective of this book is to provide NLP concepts and knowledge to readers
with a 14-h 7 step-by-step workshops to practice various core Python-based NLP
tools: NLTK, spaCy, TensorFlow Keras, Transformer, and BERT Technology to
construct NLP applications.

Organization and Structure of This Book

This book consists of two parts:

Part I Concepts and Technology (Chaps. 1–9)
Discuss concepts and technology related to NLP including: Introduction, N-gram
Language Model, Part-of-Speech Tagging, Syntax and Parsing, Meaning
Representation, Semantic Analysis, Pragmatic Analysis, Transfer Learning and
Transformer Technology, Major NLP Applications.
Part II Natural Language Processing Workshops with Python Implementation
(Chaps. 10–16)
7 Python workshops to provide step-by-step Python implementation tools includ-
ing: NLTK, spaCy, TensorFlow Keras, Transformer, and BERT Technology.
This book is organized and structured as follows:
Part I: Concepts and Technology
• Chapter 1: Introduction to Natural Language Processing

vii
viii Preface

This introductory chapter begins with human language and intelligence con-
stituting six levels of linguistics followed by a brief history of NLP with major
components and applications. It serves as the cornerstone to the NLP concepts
and technology discussed in the following chapters. This chapter also serves as
the conceptual basis for Workshop#1: Basics of Natural Language Toolkit
(NLTK) in Chap. 10.
• Chapter 2: N-gram Language Model
Language model is the foundation of NLP. This chapter introduces N-gram
language model and Markov Chains using classical literature The Adventures of
Sherlock Holmes by Sir Conan Doyle (1859–1930) to illustrate how N-gram
model works that form NLP basics in text analysis followed by Shannon’s model
and text generation with evaluation schemes. This chapter also serves as the con-
ceptual basis for Workshop#2 on N-gram modelling with NLTK in Chap. 11.
• Chapter 3: Part-of-Speech Tagging
Part-of-Speech (POS) Tagging is the foundation of text processing in
NLP. This chapter describes how it relates to NLP and Natural Language
Understanding (NLU). There are types and algorithms for POS Tagging includ-
ing Rule-based POS Tagging, Stochastic POS Tagging, and Hybrid POS Tagging
with Brill Tagger and evaluation schemes. This chapter also serves as the concep-
tual basis for Workshop#3: Part-of-Speech using Natural Language Toolkit in
Chap. 12.
• Chapter 4—Syntax and Parsing
As another major component of Natural Language Understanding (NLU),
this chapter explores syntax analysis and introduces different types of constitu-
ents in English language followed by the main concept of context-free grammar
(CFG) and CFG parsing. It also studies different major parsing techniques,
including lexical and probabilistic parsing with live examples for illustration.
• Chapter 5: Meaning Representation
Before the study of Semantic Analysis, this chapter explores meaning repre-
sentation, a vital component in NLP. It studies four major meaning representa-
tion techniques which include: first-order predicate calculus (FOPC), semantic
net, conceptual dependency diagram (CDD), and frame-based representation.
After that it explores canonical form and introduces Fillmore’s theory of univer-
sal cases followed by predicate logic and inference work using FOPC with live
examples.
• Chapter 6: Semantic Analysis
This chapter studies Semantic Analysis, one of the core concepts for learning
NLP. First, it studies the two basic schemes of semantic analysis: lexical and
compositional semantic analysis. After that it explores word senses and six com-
monly used lexical semantics followed by word sense disambiguation (WSD)
and various WSD schemes. Further, it also studies WordNet and online thesauri
for word similarity and various distributed similarity measurement including
Point-wise Mutual Information (PMI) and Positive Point-wise Mutual informa-
tion (PPMI) models with live examples for illustration. Chapters 4 and 5 also
Preface ix

serve as the conceptual basis for Workshop#4: Semantic Analysis and Word
Vectors using spaCy in Chap. 13.
• Chapter 7: Pragmatic Analysis
After the discussion of semantic meaning and analysis, this chapter explores
pragmatic analysis in linguistics and discourse phenomena. It also studies coher-
ence and coreference as the key components of pragmatics and discourse critical
to NLP, followed by discourse segmentation with different algorithms on Co-
reference Resolution including Hobbs Algorithm, Centering Algorithm, Log-
Linear Model, the latest machine learning methods, and evaluation schemes.
This chapter also serves as the conceptual basis for Workshop#5: Sentiment
Analysis and Text Classification in Chap. 14.
• Chapter 8: Transfer Learning and Transformer Technology
Transfer learning is a commonly used deep learning model to minimize com-
putational resources. This chapter explores: (1) Transfer Learning (TL) against
traditional Machine Learning (ML); (2) Recurrent Neural Networks (RNN), a
significant component of transfer learning with core technologies such as Long
Short-Term Memory (LSTM) Network and Bidirectional Recurrent Neural
Networks (BRNNs) in NLP applications, and (3) Transformer technology archi-
tecture, Bidirectional Encoder Representation from Transformers (BERT)
Model, and related technologies including Transformer-XL and ALBERT tech-
nologies. This chapter also serves as the conceptual basis for Workshop#6:
Transformers with spaCy and Tensorflow in Chap. 15.
• Chapter 9: Major Natural Language Processing Applications
This is a summary of Part I with three core NLP applications: Information
Retrieval (IR) systems, Text Summarization (TS) systems, and Question-and-
Answering (Q&A) chatbot systems, how they work and related R&D in building
NLP applications. This chapter also serves as the conceptual basis for
Workshop#7: Building Chatbot with TensorFlow and Transformer Technology
in Chap. 16.

Part II: Natural Language Processing Workshops with Python

Implementation in 14 h
• Chapter 10: Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)
With the basic NLP concept being learnt in Chap. 1, this introductory work-
shop gives a NLTK overview and system installation procedures are the founda-
tions of Python NLP development tool used for text processing which include
simple text analysis, text analysis with lexical dispersion plot, text tokenization,
and basic statistical tools in NLP.
• Chapter 11: Workshop#2 N-grams Modelling with Natural Language Toolkit
(Hour 3–4)
This is a coherent workshop of Chap. 2 using NTLK technology for N-gram
generation and statistics. This workshop consists of two parts. Part I introduces
N-gram language model using NLTK in Python and N-grams class to generate
N-gram statistics on any sentence, text objects, whole document, literature to
x Preface

provide a foundation technique for text analysis, parsing and semantic analysis
in subsequent workshops. Part II introduces spaCy, the second important NLP
Python implementation tools not only for teaching and learning (like NLTK) but
also widely used for NLP applications including text summarization, informa-
tion extraction, and Q&A chatbot. It is a critical mass to integrate with
Transformer Technology in subsequent workshops.
• Chapter 12: Workshop#3 Part-of-Speech Tagging with Natural Language Toolkit
(Hour 5–6)
In Chap. 3, we studied basic concepts and theories related to Part-of-Speech
(POS) and various POS tagging techniques. This workshop explores how to
implement POS tagging by using NLTK starting from a simple recap on tokeni-
zation techniques and two fundamental processes in word-level progressing:
stemming and stop-word removal, which will introduce two types of stemming
techniques: Porter Stemmer and Snowball Stemmer that can be integrated with
WordCloud commonly used in data visualization followed by the main theme of
this workshop with the introduction of PENN Treebank Tagset and to create your
own POS tagger.
• Chapter 13: Workshop#4 Semantic Analysis and Word Vectors using spaCy
(Hour 7–8)
In Chaps. 5 and 6, we studied the basic concepts and theories related to mean-
ing representation and semantic analysis. This workshop explores how to use
spaCy technology to perform semantic analysis starting from a revisit on word
vectors concept, implement and pre-train them followed by the study of similar-
ity method and other advanced semantic analysis.
• Chapter 14: Workshop#5 Sentiment Analysis and Text Classification (Hour 9–10)
This is a coherent workshop of Chap. 7, this workshop explores how to posi-
tion NLP implementation techniques into two important NLP applications: text
classification and sentiment analysis. TensorFlow and Kera are two vital compo-
nents to implement Long Short-Term Memory networks (LSTM networks), a
commonly used Recurrent Neural Networks (RNN) on machine learning espe-
cially in NLP applications.
• Chapter 15: Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)
In Chap. 8, the basic concept about Transfer Learning, its motivation and
related background knowledge such as Recurrent Neural Networks (RNN) with
Transformer Technology and BERT model are introduced. This workshop
explores how to put these concepts and theories into practice. More importantly,
is to implement Transformers, BERT Technology with the integration of spaCy’s
Transformer Pipeline Technology and TensorFlow. First, it gives an overview
and summation on Transformer and BERT Technology. Second, it explores
Transformer implementation with TensorFlow by revisiting Text Classification
using BERT model as example. Third, it introduces spaCy’s Transformer Pipeline
Technology and how to implement Sentiment Analysis and Text Classification
system using Transformer Technology.
Preface xi

• Chapter 16: Workshop#7 Building Chatbot with TensorFlow and Transformer

Technology (Hour 13–14)
In previous six NLP workshops, we studied NLP implementation tools and
techniques ranging from tokenization, N-gram generation to semantic and senti-
ment analysis with various key NLP Python enabling technologies: NLTK,
spaCy, TensorFlow and contemporary Transformer Technology. This final work-
shop explores how to integrate them for the design and implementation of a live
domain-based chatbot system on a movie domain. First, it explores the basis of
chatbot system and introduce a knowledge domain—the Cornell Large Movie
Conversation Dataset. Second, it conducts a step-by-step implementation of
movie chatbot system which involves dialog preprocessing, model construction,
attention learning implementation, system integration, and performance evalua-
tion followed by live tests. Finally, it introduces a mini project for this workshop
and present related chatbot datasets with resources in summary.

Readers of This Book

This book is both an NLP textbook and NLP Python implementation book tai-
lored for:
• Undergraduates and postgraduates of various disciplines including AI, Computer
Science, IT, Data Science, etc.
• Lecturers and tutors teaching NLP or related AI courses.
• NLP, AI scientists and developers who would like to learn NLP basic concepts,
practice and implement via Python workshops.
• Readers who would like to learn NLP concepts, practice Python-based NLP
workshops using various NLP implementation tools such as NLTK, spaCy,
TensorFlow Keras, BERT, and Transformer technology.

How to Use This book?

This book can be served as a textbook for undergraduates and postgraduate courses
on Natural Language Processing, and a reference book for general readers who
would like to learn key technologies and implement NLP applications with contem-
porary implementation tools such as NLTK, spaCy, TensorFlow, BERT, and
Transformer technology.
Part I (Chaps. 1–9) covers the main course materials of basic concepts and key
technologies which include N-gram Language Model, Part-of-Speech Tagging,
Syntax and Parsing, Meaning Representation, Semantic Analysis, Pragmatic
xii Preface

Analysis, Transfer Learning and Transformer Technology, and Major NLP

Applications. Part II (Chaps. 10–16) provides materials for a 14-h, step-by-step
Python-based NLP implementation in 7 workshops.
For readers and AI scientists, this book can be served as both reference in learn-
ing NLP and Python implementation toolbook on NLP applications by using the
latest Python-based NLP development tools, platforms, and libraries.
For seven NLP Workshops in Part II (Chaps. 10–16), readers can download all
JupyterNB files and data files from my NLP GitHub directory: https://github.com/
raymondshtlee/nlp/. For any query, please feel free to contact me via email: ray-
[email protected].

Zhuhai, China Raymond S. T. Lee

Acknowledgements

I would like to express my gratitude:

To my wife Iris for her patience, encouragement, and understanding, especially
during my time spent on research and writing in the past 30 years.
To Ms. Celine Cheng, executive editor of Springer NATURE and her profes-
sional editorial and book production team for their support, valuable comments,
and advice.
To Prof. Tang Tao, President of UIC, for the provision of excellent environment
for research, teaching, and writing this book.
To Prof. Weijia Jia, Vice President (Research and Development) of UIC for their
supports for R&D of NLP and related AI projects.
To Prof. Jianxin Pan, Dean of Faculty of Science and Technology of UIC, and
Prof. Weifeng Su, Head of Department of Computer Science of UIC for their con-
tinuous supports for AI and NLP courses.
To research assistant Mr. Zihao Huang for the help of NLP workshops prepara-
tion. To research student Ms. Clarissa Shi and student helpers Ms. Siqi Liu, Mr.
Mingjie Wang, and Ms. Jie Lie to help with literature review on major NLP applica-
tions and Transformer technology, and Mr. Zhuohui Chen to help for bugs fixing
and version update for the workshop programs.
To UIC for the prominent support in part by the Guangdong Provincial Key
Laboratory IRADS (2022B1212010006, R0400001-22), Key Laboratory for
Artificial Intelligence and Multi-Model Data Processing of Department of Education
of Guangdong Province and Guangdong Province F1 project grant on Curriculum
Development and Teaching Enhancement on course development UICR0400050-21
CTL for the provision of an excellent environment and computer facilities for the
preparation of this book.
Dr. Raymond Lee
December 2022
Beijing Normal University-Hong Kong Baptist University United
International College
Zhuhai
China

xiii
About the Book

This textbook presents an up-to-date and comprehensive overview of Natural

Language Processing (NLP) from basic concepts to core algorithms and key appli-
cations. It contains 7 step-by-step workshops (total 14 h) to practice essential Python
tools like NLTK, spaCy, TensorFlow Kera, Transformer, and BERT.
The objective of this book is to provide readers with fundamental knowledge,
core technologies, and enable to build their own applications (e.g. Chatbot systems)
using Python-based NLP tools. It is both a textbook and toolbook intended for
undergraduate students from various disciplines who want to learn, lecturers and
tutors who want to teach courses or tutorials for undergraduate/graduate students on
the subject and related AI topics, and readers with various backgrounds who want
to learn and build practicable applications after completing 14 h Python-based
workshops.

xv
Contents

Part I Concepts and Technology

1 Natural Language Processing �� 3
1.1 Introduction�� 3
1.2 Human Language and Intelligence �� 4
1.3 Linguistic Levels of Human Language�� 6
1.4 Human Language Ambiguity�� 7
1.5 A Brief History of NLP�� 8
1.5.1 First Stage: Machine Translation (Before 1960s) �� 8
1.5.2 Second Stage: Early AI on NLP
from 1960s to 1970s�� 8
1.5.3 Third Stage: Grammatical Logic
on NLP (1970s–1980s) �� 9
1.5.4 Fourth Stage: AI and Machine Learning
(1980s–2000s)�� 9
1.5.5 Fifth Stage: AI, Big Data, and Deep Networks
(2010s–Present)�� 10
1.6 NLP and AI �� 10
1.7 Main Components of NLP�� 11
1.8 Natural Language Understanding (NLU) �� 12
1.8.1 Speech Recognition�� 13
1.8.2 Syntax Analysis�� 13
1.8.3 Semantic Analysis�� 13
1.8.4 Pragmatic Analysis�� 13
1.9 Potential Applications of NLP�� 14
1.9.1 Machine Translation (MT)�� 14
1.9.2 Information Extraction (IE)�� 15
1.9.3 Information Retrieval (IR)�� 15
1.9.4 Sentiment Analysis �� 15
1.9.5 Question-Answering (Q&A) Chatbots �� 16
References�� 16

xvii
xviii Contents

2 N-Gram Language Model �� 19

2.1 Introduction�� 19
2.2 N-Gram Language Model �� 21
2.2.1 Basic NLP Terminology�� 22
2.2.2 Language Modeling and Chain Rule�� 24
2.3 Markov Chain in N-Gram Model�� 26
2.4 Live Example: The Adventures of Sherlock Holmes�� 27
2.5 Shannon’s Method in N-Gram Model�� 31
2.6 Language Model Evaluation and Smoothing Techniques�� 34
2.6.1 Perplexity�� 34
2.6.2 Extrinsic Evaluation Scheme�� 35
2.6.3 Zero Counts Problems�� 35
2.6.4 Smoothing Techniques�� 36
2.6.5 Laplace (Add-One) Smoothing�� 36
2.6.6 Add-k Smoothing�� 38
2.6.7 Backoff and Interpolation Smoothing�� 39
2.6.8 Good Turing Smoothing �� 40
References�� 41
3 Part-of-Speech (POS) Tagging�� 43
3.1 What Is Part-of-Speech (POS)?�� 43
3.1.1 Nine Major POS in English Language�� 43
3.2 POS Tagging �� 44
3.2.1 What Is POS Tagging in Linguistics? �� 44
3.2.2 What Is POS Tagging in NLP? �� 45
3.2.3 POS Tags Used in the PENN Treebank Project�� 45
3.2.4 Why Do We Care About POS in NLP?�� 46
3.3 Major Components in NLU�� 48
3.3.1 Computational Linguistics and POS�� 48
3.3.2 POS and Semantic Meaning �� 49
3.3.3 Morphological and Syntactic Definition of POS�� 49
3.4 9 Key POS in English �� 50
3.4.1 English Word Classes�� 51
3.4.2 What Is a Preposition?�� 51
3.4.3 What Is a Conjunction?�� 52
3.4.4 What Is a Pronoun?�� 53
3.4.5 What Is a Verb? �� 53
3.5 Different Types of POS Tagset�� 56
3.5.1 What Is Tagset?�� 56
3.5.2 Ambiguous in POS Tags �� 57
3.5.3 POS Tagging Using Knowledge �� 58
3.6 Approaches for POS Tagging �� 58
3.6.1 Rule-Based Approach POS Tagging �� 58
3.6.2 Example of Rule-Based POS Tagging�� 59
Contents xix

3.6.3 Example of Stochastic-Based POS Tagging �� 60

3.6.4 Hybrid Approach for POS Tagging Using
Brill Taggers�� 61
3.7 Taggers Evaluations�� 63
3.7.1 How Good Is an POS Tagging Algorithm?�� 64
References�� 65
4 Syntax and Parsing�� 67
4.1 Introduction and Motivation �� 67
4.2 Syntax Analysis�� 68
4.2.1 What Is Syntax�� 68
4.2.2 Syntactic Rules�� 68
4.2.3 Common Syntactic Patterns�� 69
4.2.4 Importance of Syntax and Parsing in NLP �� 70
4.3 Types of Constituents in Sentences�� 70
4.3.1 What Is Constituent?�� 70
4.3.2 Kinds of Constituents�� 72
4.3.3 Noun-Phrase (NP)�� 72
4.3.4 Verb-Phrase (VP)�� 72
4.3.5 Complexity on Simple Constituents �� 73
4.3.6 Verb Phrase Subcategorization �� 74
4.3.7 The Role of Lexicon in Parsing�� 75
4.3.8 Recursion in Grammar Rules�� 76
4.4 Context-Free Grammar (CFG)�� 76
4.4.1 What Is Context-Free Language (CFL)?�� 76
4.4.2 What Is Context-Free Grammar (CFG)?�� 77
4.4.3 Major Components of CFG�� 77
4.4.4 Derivations Using CFG�� 78
4.5 CFG Parsing�� 79
4.5.1 Morphological Parsing�� 79
4.5.2 Phonological Parsing�� 79
4.5.3 Syntactic Parsing�� 79
4.5.4 Parsing as a Kind of Tree Searching �� 80
4.5.5 CFG for Fragment of English �� 80
4.5.6 Parse Tree for “Play the Piano” for Prior CFG �� 80
4.5.7 Top-Down Parser�� 81
4.5.8 Bottom-Up Parser �� 82
4.5.9 Control of Parsing �� 84
4.5.10 Pros and Cons of Top-Down vs. Bottom-Up Parsing �� 84
4.6 Lexical and Probabilistic Parsing�� 85
4.6.1 Why Using Probabilities in Parsing?�� 85
4.6.2 Semantics with Parsing �� 86
4.6.3 What Is PCFG? �� 87
4.6.4 A Simple Example of PCFG�� 87
xx Contents

4.6.5 Using Probabilities for Language Modeling�� 90

4.6.6 Limitations for PCFG �� 90
4.6.7 The Fix: Lexicalized Parsing�� 91
References�� 94
5 Meaning Representation�� 95
5.1 Introduction�� 95
5.2 What Is Meaning? �� 95
5.3 Meaning Representations�� 96
5.4 Semantic Processing �� 97
5.5 Common Meaning Representation �� 98
5.5.1 First-Order Predicate Calculus (FOPC)�� 98
5.5.2 Semantic Networks �� 98
5.5.3 Conceptual Dependency Diagram (CDD)�� 99
5.5.4 Frame-Based Representation�� 99
5.6 Requirements for Meaning Representation�� 100
5.6.1 Verifiability �� 100
5.6.2 Ambiguity �� 100
5.6.3 Vagueness�� 101
5.6.4 Canonical Forms �� 101
5.7 Inference �� 102
5.7.1 What Is Inference?�� 102
5.7.2 Example of Inferencing with FOPC �� 103
5.8 Fillmore’s Theory of Universal Cases�� 103
5.8.1 What Is Fillmore’s Theory of Universal Cases? �� 104
5.8.2 Major Case Roles in Fillmore’s Theory�� 105
5.8.3 Complications in Case Roles�� 106
5.9 First-Order Predicate Calculus�� 107
5.9.1 FOPC Representation Scheme�� 107
5.9.2 Major Elements of FOPC�� 107
5.9.3 Predicate-Argument Structure of FOPC �� 108
5.9.4 Meaning Representation Problems in FOPC�� 110
5.9.5 Inferencing Using FOPC�� 111
References�� 113
6 Semantic Analysis�� 115
6.1 Introduction�� 115
6.1.1 What Is Semantic Analysis?�� 115
6.1.2 The Importance of Semantic Analysis in NLP �� 116
6.1.3 How Human Is Good in Semantic Analysis?�� 116
6.2 Lexical Vs Compositional Semantic Analysis�� 117
6.2.1 What Is Lexical Semantic Analysis?�� 117
6.2.2 What Is Compositional Semantic Analysis? �� 117
6.3 Word Senses and Relations �� 118
6.3.1 What Is Word Sense?�� 118
6.3.2 Types of Lexical Semantics�� 119
Contents xxi

6.4 Word Sense Disambiguation�� 123

6.4.1 What Is Word Sense Disambiguation (WSD)? �� 123
6.4.2 Difficulties in Word Sense Disambiguation�� 123
6.4.3 Method for Word Sense Disambiguation�� 124
6.5 WordNet and Online Thesauri�� 126
6.5.1 What Is WordNet?�� 126
6.5.2 What Is Synsets? �� 126
6.5.3 Knowledge Structure of WordNet�� 127
6.5.4 What Are Major Lexical Relations Captured
in WordNet?�� 129
6.5.5 Applications of WordNet and Thesauri?�� 129
6.6 Other Online Thesauri: MeSH�� 130
6.6.1 What Is MeSH?�� 130
6.6.2 Uses of the MeSH Ontology �� 131
6.7 Word Similarity and Thesaurus Methods�� 131
6.8 Introduction�� 131
6.8.1 Path-based Similarity�� 132
6.8.2 Problems with Path-based Similarity�� 133
6.8.3 Information Content Similarity�� 134
6.8.4 The Resnik Method�� 135
6.8.5 The Dekang Lin Method�� 135
6.8.6 The (Extended) Lesk Algorithm �� 136
6.9 Distributed Similarity�� 137
6.9.1 Distributional Models of Meaning�� 137
6.9.2 Word Vectors�� 137
6.9.3 Term-Document Matrix�� 137
6.9.4 Point-wise Mutual Information (PMI)�� 139
6.9.5 Example of Computing PPMI on a Term-Context
Matrix�� 140
6.9.6 Weighing PMI Techniques�� 141
6.9.7 K-Smoothing in PMI Computation�� 142
6.9.8 Context and Word Similarity Measurement�� 144
6.9.9 Evaluating Similarity�� 145
References�� 146
7 Pragmatic Analysis and Discourse �� 149
7.1 Introduction�� 149
7.2 Discourse Phenomena�� 149
7.2.1 Coreference Resolution�� 150
7.2.2 Why Is it Important? �� 150
7.2.3 Coherence and Coreference�� 151
7.2.4 Importance of Coreference Relations �� 152
7.2.5 Entity-Based Coherence�� 153
7.3 Discourse Segmentation�� 154
7.3.1 What Is Discourse Segmentation?�� 154
xxii Contents

7.3.2 Unsupervised Discourse Segmentation�� 154

7.3.3 Hearst’s TextTiling Method�� 155
7.3.4 TextTiling Algorithm�� 157
7.3.5 Supervised Discourse Segmentation�� 158
7.4 Discourse Coherence�� 158
7.4.1 What Makes a Text Coherent?�� 158
7.4.2 What Is Coherence Relation?�� 159
7.4.3 Types of Coherence Relations�� 159
7.4.4 Hierarchical Structure of Discourse Coherence�� 160
7.4.5 Types of Referring Expressions�� 161
7.4.6 Features for Filtering Potential Referents �� 162
7.4.7 Preferences in Pronoun Interpretation�� 162
7.5 Algorithms for Coreference Resolution�� 163
7.5.1 Introduction�� 163
7.5.2 Hobbs Algorithm�� 163
7.5.3 Centering Algorithm �� 166
7.5.4 Machine Learning Method�� 169
7.6 Evaluation �� 171
References�� 172
8 Transfer Learning and Transformer Technology�� 175
8.1 What Is Transfer Learning?�� 175
8.2 Motivation of Transfer Learning�� 176
8.2.1 Categories of Transfer Learning �� 176
8.3 Solutions of Transfer Learning �� 178
8.4 Recurrent Neural Network (RNN)�� 180
8.4.1 What Is RNN?�� 180
8.4.2 Motivation of RNN �� 180
8.4.3 RNN Architecture �� 181
8.4.4 Long Short-Term Memory (LSTM) Network�� 183
8.4.5 Gate Recurrent Unit (GRU)�� 185
8.4.6 Bidirectional Recurrent Neural Networks (BRNNs)�� 186
8.5 Transformer Technology�� 188
8.5.1 What Is Transformer? �� 188
8.5.2 Transformer Architecture�� 188
8.5.3 Deep Into Encoder�� 189
8.6 BERT�� 192
8.6.1 What Is BERT? �� 192
8.6.2 Architecture of BERT �� 192
8.6.3 Training of BERT�� 192
8.7 Other Related Transformer Technology�� 194
8.7.1 Transformer-XL�� 194
8.7.2 ALBERT �� 195
References�� 196
Contents xxiii

9 Major NLP Applications �� 199

9.1 Introduction�� 199
9.2 Information Retrieval Systems�� 199
9.2.1 Introduction to IR Systems �� 199
9.2.2 Vector Space Model in IR �� 200
9.2.3 Term Distribution Models in IR�� 202
9.2.4 Latent Semantic Indexing in IR�� 207
9.2.5 Discourse Segmentation in IR�� 208
9.3 Text Summarization Systems�� 212
9.3.1 Introduction to Text Summarization Systems�� 212
9.3.2 Text Summarization Datasets�� 214
9.3.3 Types of Summarization Systems�� 214
9.3.4 Query-Focused Vs Generic Summarization Systems �� 215
9.3.5 Single and Multiple Document Summarization�� 217
9.3.6 Contemporary Text Summarization Systems�� 218
9.4 Question-and-Answering Systems�� 224
9.4.1 QA System and AI�� 224
9.4.2 Overview of Industrial QA Systems �� 228
References�� 236

Part II Natural Language Processing Workshops with Python

Implementation in 14 Hours
10
Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)�� 243
10.1 Introduction�� 243
10.2 What Is Natural Language Toolkit (NLTK)?�� 243
10.3 A Simple Text Tokenization Example Using NLTK�� 244
10.4 How to Install NLTK?�� 245
10.5 Why Using Python for NLP?�� 246
10.6 NLTK with Basic Text Processing in NLP �� 248
10.7 Simple Text Analysis with NLTK �� 249
10.8 Text Analysis Using Lexical Dispersion Plot �� 253
10.8.1 What Is a Lexical Dispersion Plot?�� 253
10.8.2 Lexical Dispersion Plot Over Context Using
Sense and Sensibility�� 253
10.8.3 Lexical Dispersion Plot Over Time Using
Inaugural Address Corpus�� 254
10.9 Tokenization in NLP with NLTK�� 255
10.9.1 What Is Tokenization in NLP?�� 255
10.9.2 Different Between Tokenize() vs Split() �� 256
10.9.3 Count Distinct Tokens�� 257
10.9.4 Lexical Diversity�� 258
10.10 Basic Statistical Tools in NLTK�� 260
10.10.1 Frequency Distribution: FreqDist()�� 260
xxiv Contents

10.10.2 Rare Words: Hapax �� 262

10.10.3 Collocations�� 263
References�� 265
11
Workshop#2 N-grams in NLTK and Tokenization in SpaCy
(Hour 3–4)�� 267
11.1 Introduction�� 267
11.2 What Is N-Gram?�� 267
11.3 Applications of N-Grams in NLP �� 268
11.4 Generation of N-Grams in NLTK �� 268
11.5 Generation of N-Grams Statistics �� 270
11.6 spaCy in NLP�� 276
11.6.1 What Is spaCy? �� 276
11.7 How to Install spaCy? �� 277
11.8 Tokenization using spaCy �� 278
11.8.1 Step 1: Import spaCy Module �� 278
11.8.2 Step 2: Load spaCy Module "en_core_web_sm". �� 278
11.8.3 Step 3: Open and Read Text File
"Adventures_Holmes.txt" Into file_handler
"fholmes"�� 278
11.8.4 Step 4: Read Adventures of Sherlock Holmes�� 278
11.8.5 Step 5: Replace All Newline Symbols�� 279
11.8.6 Step 6: Simple Counting �� 279
11.8.7 Step 7: Invoke nlp() Method in spaCy�� 280
11.8.8 Step 8: Convert Text Document Into Sentence
Object�� 280
11.8.9 Step 9: Directly Tokenize Text Document�� 282
References�� 284
12
Workshop#3 POS Tagging Using NLTK (Hour 5–6)�� 285
12.1 Introduction�� 285
12.2 A Revisit on Tokenization with NLTK �� 285
12.3 Stemming Using NLTK�� 288
12.3.1 What Is Stemming?�� 288
12.3.2 Why Stemming?�� 289
12.3.3 How to Perform Stemming? �� 289
12.3.4 Porter Stemmer �� 289
12.3.5 Snowball Stemmer�� 291
12.4 Stop-Words Removal with NLTK�� 292
12.4.1 What Are Stop-Words? �� 292
12.4.2 NLTK Stop-Words List �� 292
12.4.3 Try Some Texts �� 294
12.4.4 Create Your Own Stop-Words�� 295
12.5 Text Analysis with NLTK �� 296
Contents xxv

12.6 Integration with WordCloud �� 299

12.6.1 What Is WordCloud?�� 299
12.7 POS Tagging with NLTK�� 301
12.7.1 What Is POS Tagging?�� 301
12.7.2 Universal POS Tagset�� 301
12.7.3 PENN Treebank Tagset (English and Chinese)�� 302
12.7.4 Applications of POS Tagging �� 303
12.8 Create Own POS Tagger with NLTK�� 306
References�� 312
13
Workshop#4 Semantic Analysis and Word Vectors Using spaCy
(Hour 7–8)�� 313
13.1 Introduction�� 313
13.2 What Are Word Vectors?�� 313
13.3 Understanding Word Vectors�� 314
13.3.1 Example: A Simple Word Vector�� 314
13.4 A Taste of Word Vectors�� 316
13.5 Analogies and Vector Operations�� 319
13.6 How to Create Word Vectors? �� 320
13.7 spaCy Pre-trained Word Vectors �� 320
13.8 Similarity Method in Semantic Analysis�� 323
13.9 Advanced Semantic Similarity Methods with spaCy �� 326
13.9.1 Understanding Semantic Similarity�� 326
13.9.2 Euclidian Distance�� 326
13.9.3 Cosine Distance and Cosine Similarity�� 327
13.9.4 Categorizing Text with Semantic Similarity �� 329
13.9.5 Extracting Key Phrases �� 330
13.9.6 Extracting and Comparing Named Entities�� 331
References�� 333
14
Workshop#5 Sentiment Analysis and Text Classification
with LSTM Using spaCy (Hour 9–10) �� 335
14.1 Introduction�� 335
14.2 Text Classification with spaCy and LSTM Technology �� 335
14.3 Technical Requirements�� 336
14.4 Text Classification in a Nutshell �� 336
14.4.1 What Is Text Classification? �� 336
14.4.2 Text Classification as AI Applications�� 337
14.5 Text Classifier with spaCy NLP Pipeline�� 338
14.5.1 TextCategorizer Class �� 339
14.5.2 Formatting Training Data for the TextCategorizer�� 340
14.5.3 System Training�� 344
14.5.4 System Testing�� 346
14.5.5 Training TextCategorizer for Multi-Label
Classification�� 347
xxvi Contents

14.6 Sentiment Analysis with spaCy�� 351

14.6.1 IMDB Large Movie Review Dataset�� 351
14.6.2 Explore the Dataset �� 351
14.6.3 Training the TextClassfier �� 355
14.7 Artificial Neural Network in a Nutshell�� 357
14.8 An Overview of TensorFlow and Keras�� 358
14.9 Sequential Modeling with LSTM Technology�� 358
14.10 Keras Tokenizer in NLP�� 359
14.10.1 Embedding Words�� 363
14.11 Movie Sentiment Analysis with LTSM Using Keras
and spaCy�� 364
14.11.1 Step 1: Dataset�� 365
14.11.2 Step 2: Data and Vocabulary Preparation�� 366
14.11.3 Step 3: Implement the Input Layer �� 368
14.11.4 Step 4: Implement the Embedding Layer �� 368
14.11.5 Step 5: Implement the LSTM Layer �� 368
14.11.6 Step 6: Implement the Output Layer�� 369
14.11.7 Step 7: System Compilation �� 369
14.11.8 Step 8: Model Fitting and Experiment Evaluation �� 370
References�� 371
15
Workshop#6 Transformers with spaCy and TensorFlow
(Hour 11–12)�� 373
15.1 Introduction�� 373
15.2 Technical Requirements�� 373
15.3 Transformers and Transfer Learning in a Nutshell �� 374
15.4 Why Transformers?�� 375
15.5 An Overview of BERT Technology�� 377
15.5.1 What Is BERT? �� 377
15.5.2 BERT Architecture�� 378
15.5.3 BERT Input Format�� 378
15.5.4 How to Train BERT?�� 380
15.6 Transformers with TensorFlow �� 382
15.6.1 HuggingFace Transformers�� 382
15.6.2 Using the BERT Tokenizer �� 383
15.6.3 Word Vectors in BERT�� 386
15.7 Revisit Text Classification Using BERT �� 388
15.7.1 Data Preparation�� 388
15.7.2 Start the BERT Model Construction �� 389
15.8 Transformer Pipeline Technology�� 392
15.8.1 Transformer Pipeline for Sentiment Analysis�� 393
15.8.2 Transformer Pipeline for QA System �� 393
15.9 Transformer and spaCy�� 394
References�� 398
Contents xxvii

16
Workshop#7 Building Chatbot with TensorFlow and Transformer
Technology (Hour 13–14)�� 401
16.1 Introduction�� 401
16.2 Technical Requirements�� 401
16.3 AI Chatbot in a Nutshell �� 402
16.3.1 What Is a Chatbot?�� 402
16.3.2 What Is a Wake Word in Chatbot?�� 403
16.3.3 NLP Components in a Chatbot �� 404
16.4 Building Movie Chatbot by Using TensorFlow
and Transformer Technology�� 404
16.4.1 The Chatbot Dataset�� 405
16.4.2 Movie Dialog Preprocessing�� 405
16.4.3 Tokenization of Movie Conversation�� 407
16.4.4 Filtering and Padding Process�� 408
16.4.5 Creation of TensorFlow Movie Dataset
Object (mDS)�� 409
16.4.6 Calculate Attention Learning Weights�� 410
16.4.7 Multi-Head-Attention (MHAttention)�� 411
16.4.8 System Implementation�� 412
16.5 Related Works �� 430
References�� 431

Index�� 433
About the Author

Raymond Lee is the founder of the Quantum Finance Forecast System (QFFC)
(https://qffc.uic.edu.cn) and currently an Associate Professor at United International
College (UIC) with 25+ years’ experience in AI research and consultancy, Chaotic
Neural Networks, NLP, Intelligent Fintech Systems, Quantum Finance, and
Intelligent E-Commerce Systems. He has published over 100 publications and
authored 8 textbooks in the fields of AI, chaotic neural networks, AI-based fintech
systems, intelligent agent technology, chaotic cryptosystems, ontological agents,
neural oscillators, biometrics, and weather simulation and forecasting systems.
Upon completion of the QFFC project, in 2018 he joined United International
College (UIC), China, to pursue further R&D work on AI-Fintech and to share his
expertise in AI-Fintech, chaotic neural networks, and related intelligent systems
with fellow students and the community. His three latest textbooks, Quantum
Finance: Intelligent Forecast and Trading Systems (2019), Artificial Intelligence in
Daily Life (2020), and this NLP book have been adopted as the main textbooks for
various AI courses in UIC.

xxix
Abbreviations

AI Artificial intelligence
ASR Automatic speech recognition
BERT Bidirectional encoder representations from transformers
BRNN Bidirectional recurrent neural networks
CDD Conceptual dependency diagram
CFG Context-free grammar
CFL Context-free language
CNN Convolutional neural networks
CR Coreference resolution
DNN Deep neural networks
DT Determiner
FOPC First-order predicate calculus
GRU Gate recurrent unit
HMM Hidden Markov model
IE Information extraction
IR Information retrieval
KAI Knowledge acquisition and inferencing
LSTM Long short-term memory
MEMM Maximum entropy Markov model
MeSH Medical subject thesaurus
ML Machine learning
NER Named entity recognition
NLP Natural language processing
NLTK Natural language toolkit
NLU Natural language understanding
NN Noun
NNP Proper noun
Nom Nominal
NP Noun phrase
PCFG Probabilistic context-free grammar
PMI Pointwise mutual information

xxxi
xxxii Abbreviations

POS Part-of-speech
POST Part-of-speech tagging
PPMI Positive pointwise mutual information
Q&A Question-and-answering
RNN Recurrent neural networks
TBL Transformation-based learning
VB Verb
VP Verb phrase
WSD Word sense disambiguation
Part I
Concepts and Technology
Chapter 1
Natural Language Processing

Consider this scenario: Late in the evening, Jack starts a mobile app and talks with
AI Tutor Max.

1.1 Introduction

There are many chatbots that allow humans to communicate with a device in natural
language nowadays. Figure 1.1 illustrates dialogue between a student who had
returned to dormitory after a full day classes and initiated communication with a
mobile application called AI Tutor 2.0 (Cui et al. 2020) from our latest research on
AI tutor chatbot. The objective is to enable the user (Jack) not only can learn from
book reading but also can communicate candidly with AI Tutor 2.0 (Max) to provide
knowledge responses in natural language. It is different from chatbots that respond
with basic commands but is human–computer interaction to demonstrate how a user
wishes to communicate in a way like a student convers with a tutor about subject
knowledge in the physical world. It is a dynamic process consisting of (1) world
knowledge on simple handshaking dialogue such as greetings and general discus-
sions. This is not an easy task as it involves knowledge and common sense to con-
struct a functional chatbot with daily dialogues, and (2) technical knowledge of a
particular knowledge domain, or domain expert as it required to learn from author’s
book AI in Daily Life (Lee 2020) first which covers all basic knowledge on the sub-
ject to form a knowledge tree or ontology graph that can be served as a new type of
publication and interactive device between human and computer to learn new
knowledge.
Natural language processing (NLP) is related to several disciplines including
human linguistic, computation linguistic, statistical engineering, AI in machine

© The Author(s), under exclusive license to Springer Nature Singapore Pte 3

Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_1
4 1 Natural Language Processing

Max: Good evening, Jack. How are you?

Jack: Busy day. Just finished dinner after whole af-
ternoon lecture.
Max: I see. Anything I can help?
Jack: Yes. Today Dr Lee has talked about data min-
ing and search engine. Some concepts are not
so clear, might need your help.
Max: Sure thing. Which part are you not clear?
Jack: What is the limitation of search engine?
Max: Search engines today have one intrinsic prob-
lem, that is all the search results are searched
automatically and ranked according to similar
query keywords.
Jack: Why Hidden Markov Model is popular?
Max: The reason why HMM is popular because it
is: 1) reliable; 2) easy to implement technical-
ly; and 3) the Marko chains can be trained
automatically and computationally feasible to
use
Jack: What is Data Mining?
Max: Data mining is the process of discovering pat-
terns in large datasets involving methods at
the intersection of machine learning, statis-
tics, and database systems.
Jack: I see. Thank you very much. Have a nice even-
ing.
Max: You too. Good night!

Fig. 1.1 A snapshot of AI Tutor chatbot

learning, data mining, human voice processing recognition and synthesis, etc. There
are many genius chatbots initiated by NLP and AI scientists which become com-
mercial products in past decades.
This chapter will introduce this prime technology and components followed by
pertinent technologies in subsequent chapters.

1.2 Human Language and Intelligence

There is an old saying: The way you behave says more about who you are. It is
because we never know what people think, the only method is to evaluate and judge
their behaviors.
1.2 Human Language and Intelligence 5

Fig. 1.2 Turing test (Tuchong 2020a)

NLP core technologies and methodologies arose from famous Turing Test
(Eisenstein 2019; Bender 2013; Turing 1936, 1950) proposed by Sir Alan Turing
(1912–1954) in 1950s, the father of AI. Figure 1.2 shows a human judge convers
with two individuals in two rooms. One is a human, the other is either a robot, a
chatbot, or an NLP application. During a 20 min conversation, the judge can ask
human/machine technical/non-technical questions and require response on every
question so that the judge can decide whether the respondent is a human or a
machine. NLP in Turing Test is to recognize, understand questions, and respond in
human language. It remains a popular topic in AI today because we cannot see and
judge people’s thinking to define intelligence. It is the ultimate challenge in AI.
Human language is a significant component in human behavior and civilization.
It can be categorized into (1) written and (2) oral aspects generally. Written lan-
guage undertakes to process, store, and pass human/natural language knowledge to
next generations. Oral or spoken language acts as a communication media among
other individuals.
NLP has examined the basic effects on philosophy such as meaning and knowl-
edge, psychology in words meanings, linguistics in phrases and sentences forma-
tion, computational linguists in language models. Hence, NLP is cross-disciplinary
integration of disciplines such as philosophy in human language ontology models,
psychology behavior between natural and human language, linguistics in mathe-
matical and language models, computational linguistics in agents and ontology
trees technology as shown in Fig. 1.3.
6 1 Natural Language Processing

Fig. 1.3 Various discipline related to NLP

1.3 Linguistic Levels of Human Language

Linguistic levels (Hausser 2014) are regarded as functional analysis of human-

written and spoken languages. There are six levels in linguistics analysis (1) phonet-
ics, (2) phonology, (3) morphology, (4) syntax, (5) semantics, and (6) pragmatics
(discourse) classified in basic sound linguistic. The six-levels of linguistic are shown
in Fig. 1.4.
The basic linguistic structure of spoken language includes phonetics and phonol-
ogy. Phonetics refers to the physical aspects of sound, the study of production and
perception of sounds called phones. Phonetics governs the production of human
speech often without preceding knowledge of spoken language, organizes sounds,
and studies the phonemes of languages that can provide various meanings between
words and phrases.
Direct language structure is related to morphological and syntactic levels.
Morphology is the form and word level determined by grammar and syntax gener-
ally. It refers to the smallest form in linguistic analysis, consisting of sounds, to
combine words with grammatical or lexical function.
Lexicology is the study of vocabulary from a word form to a derived-form. Syntax
represents the primary level of clauses and sentences to organize meaning of differ-
ent words order, i.e. addition and subtraction of spoken language, and deals with
related sentence patterns and ambiguous analysis.
The advanced structure deals with actual language meaning at semantic and
pragmatic levels. Semantic level is the domain of meaning that consists of morphol-
ogy and syntax but is seen as a level that requires one’s own learning to assign cor-
rect meaning promptly with vocabulary, terminology form, grammar, sentence, and
discourse perspective. Pragmatics is the use of language in definitive settings. The
meaning of discourse does not have to be the same as abstract form in actual use. It
is largely based on concept of speech acts and the contents of statement with intent
and effect analysis of language performance.
1.4 Human Language Ambiguity 7

Fig. 1.4 Linguistic levels of human languages

1.4 Human Language Ambiguity

In many language models, cultural differences often produce identical utterance

with more than single meaning in conversation. Ambiguity are the capabilities to
understand sentence structures in many ways. There are (1) lexical, (2) syntactic, (3)
semantic, and (4) pragmatics ambiguities in NLP.
Lexical ambiguity arises from words where a word meaning depends on contex-
tual utterance. For instance, the word green is normally a noun for color. But it can
be an adjective or even a verb in different situations.
Syntactic ambiguity arises from sentences that are parsed differently, e.g. Jack
watched Helen with a telescope. It can describe either Jack watched Helen by using
a telescope or Jack watched Helen holding a telescope.
Semantic ambiguity arises from word meaning that can be misinterpreted, or a
sentence has ambiguous words or phrases, e.g. The van hits the boar while it is mov-
ing. It can describe either the van hits the boar while the van is moving, or the van
hits the boar while the boar is moving. It has more than a simple syntactic meaning
and required to work out the correct interpretation.
8 1 Natural Language Processing

Pragmatic ambiguity arises from a statement that is not clearly defined when the
context of a sentence provides multiple interpretations such as I like that too. It can
describe I like that too, other likes that too but the description of that is uncertain.
NLP analyzes sentences ambiguity incessantly. If they can be identified earlier,
it will be easier to define proper meanings.

1.5 A Brief History of NLP

There are several major NLP transformation stages in NLP history (Santilal 2020).

1.5.1 First Stage: Machine Translation (Before 1960s)

The concept of NLP was introduced in seventeenth century by philosopher and

mathematician Gottfried Wilhelm Leibniz (1646–1716) and polymath René
Descartes (1596–1650). Their studies of the relationships between words and lan-
guages formed the basis for language translation engine development (Santilal 2020).
The first patent for an invention related to machine translation was filed by inven-
tor and engineer Georges Artsrouni in 1933, but formal study and research was
rendered by Sir Alan Turing from his remarkable article Computing Machinery and
Intelligence published in 1950 (Turing 1936, 1950) and his famous Turing test offi-
cially used as an evaluation criterion for machine intelligence since NLP research
and development were mainly focused on language translation at that time.
The first and second International Conference on Machine Translation held in
1952 and 1956 used basic rule-based and stochastic techniques. The 1954
Georgetown-IBM experiment engaged wholly automatic machine translation of
more than 60 Russian sentences into English and was over optimistic that the whole
machine translation problem can be solved within a few years. However, break-
through on NLP was achieved by Emeritus Prof. Noam Chomsky on universal
grammar for linguistics in 1957, but since the ALPAC report published in 1966
revealed deficient progress for AI and machine translation in the past 10 years signi-
fied the first winter of AI.

1.5.2 Second Stage: Early AI on NLP from 1960s to 1970s

NLP major development was focused on how it can be used in different areas
such as knowledge engineering called agent ontology to shape meaning repre-
sentations following AI grew popular over time. BASEBALL system
1.5 A Brief History of NLP 9

(Green et al. 1961) was a typical example of Q&A-based domain expert system
of human and computer interaction developed in 1960s, but inputs were restric-
tive and language processing techniques remained in basic language processing.
In 1968, Prof. Marvin Minsky (1927–2016) developed a more powerful NLP
system. This advanced system used an AI-based question-answering inference
engine between humans and computers to provide knowledge-based interpretations
of questions and answers. Further, Prof. William A. Woods proposed an augmented
translation network (ATN) to represent natural language input in 1970. During this
period, many programmers started to transcribe codes in different AI languages to
conceptualize natural language ontology knowledge of real-world structural infor-
mation into human understanding mode status. Yet these expert systems were unable
to meet expectation signified the second winter of AI.

1.5.3 Third Stage: Grammatical Logic on NLP (1970s–1980s)

Research turned to knowledge representation, programming logic, and reasoning in

AI. This period was regarded as the grammatical logic phase of NLP in which pow-
erful sentence processing techniques such as SRI’s core language engine and dis-
course representation theory, a new pragmatic representation and discourse
interpretation with practical resources and tools such as parsers and Q&A chatbots.
Although R&D was hampered by computational power but lexicon in 1980s aimed
to expand NLP.

1.5.4 Fourth Stage: AI and Machine Learning (1980s–2000s)

The revolutionary success of Hopfield Network in the field of machine learning

proposed by Prof. Emeritus John Hopfield activated a new era of NLP research
using machine learning techniques as an alternative to complex rule-based and sto-
chastic methods in the past decades.
Computational technology upgrades in computational power and memory
complemented Chomsky’s theory of linguistics had enhanced language process-
ing from machine learning methods of corpus linguistics. This development
stage was also known as NLP lexical, and corpus referred to grammar emergence
in lexicalization method in late 1980s, which signified the IBM DeepQA project
led by Dr. David Ferrucci for their remarkable question-answering system devel-
oped in 2006.
10 1 Natural Language Processing

1.5.5 Fifth Stage: AI, Big Data, and Deep Networks

(2010s–Present)

NLP statistical technique and rule-based system R&D had evolved into cloud com-
puting technology on mobile computing and big data in deep network analysis, e.g.
recurrent neural networks using LSTM and related networks. Google, Amazon,
Facebook contributed to agent technologies and deep neural networks development
in 2010 to devise products such as auto-driving, Q&A chatbots, and storage devel-
opment are under way.

1.6 NLP and AI

NLP can be regarded as automatic or semi-automatic processing of human language

(Eisenstein 2019). It requires extensive knowledge of linguistics and logical theory
in theoretical mathematics, also known as computational linguistics. It is a multidis-
ciplinary study of epistemology, philosophy, psychology, cognitive science, and
agent ontology.
NLP is an area of AI which computer machines can analyze and interpret human
speech for human–computer interaction (HCI) to generate structural knowledge for
information retrieval operations, text and automatic text summarization, sentiment
and speech recognition analysis, data mining, deep learning, and machine transla-
tion agent ontologies at different levels of Q&A chatbots (Fig. 1.5).

Fig. 1.5 NLP and AI (Tuchong 2020b)

1.7 Main Components of NLP 11

1.7 Main Components of NLP

NLP consists of (1) Natural Language Understanding (NLU), (2) Knowledge

Acquisition and Inferencing (KAI), (3) Natural Language Generation (NLG) com-
ponents as shown in Fig. 1.6.
NLU is a technique and method devised to understand the meanings of human
spoken languages by syntax, semantic, and pragmatic analyses.
KAI is a system to generate proper responses after spoken languages are fully
recognized by NLU. It is an unresolved knowledge acquisition and inferencing
problem in machine learning and AI by conventional rule-based system due to the
intricacies of natural language and conversation, i.e. an if-then-else types of query-
response used in expert systems, most KAI systems strive to regulate knowledge
domain at a specific industry for resolution, i.e. customer service knowledge for
insurance, medical, etc. Further, agent ontology has achieved favorable outcome.
NLG includes answer, response, and feedback generation in human–machine
dialogue. It is a multi-facet machine translation process that converts responses into
text and sentences to perform text-to-speech synthesis from target language and
produce near human speech responses.

Fig. 1.6 NLP main components

12 1 Natural Language Processing

1.8 Natural Language Understanding (NLU)

Natural Language Understanding (NLU) is a process of recognizing and under-

standing spoken language in four stages (Allen 1994): (1) speech recognition, (2)
syntactic (syntax) analysis, (3) semantic analysis, and (4) pragmatic analysis as
shown in Fig. 1.7.

Spoken
Language

Speech
Recognition

Lexicon
Syntax
Analysis
Grammar

Semantic Semantic
Rules Analysis

Contextual Pragmatic
Information Analysis

Target Meaning
Representation

Fig. 1.7 NLU systematic diagram

1.8 Natural Language Understanding (NLU) 13

1.8.1 Speech Recognition

Speech recognition (Li et al. 2015) is the first stage in NLU that performs phonetic,
phonological, and morphological processing to analyze spoken language. The task
involves breaking down the stems of spoken words called utterances, into distinct
tokens representing paragraphs, sentences, and words in different parts. Current
speech recognition models apply spectrogram analysis to extract distinct frequen-
cies, e.g. the word uncanny can be split into two-word tokens un and canny. Different
languages have different spectrogram analysis.

1.8.2 Syntax Analysis

Syntax analysis (Sportier et al. 2013) is the second stage of NLU direct response
speech recognition, analyzing the structural meaning of spoken sentences. This task
has two purposes: (1) check syntax correctness of the sentence/utterance, (2) break
down spoken sentences into syntactic structures to reflect syntactic relationship
between words. For instance, the utterance oranges to the boys will be rejected by
syntax parser because of syntactic errors.

1.8.3 Semantic Analysis

Semantic analysis (Goddard 1998) is the third stage in NLU which corresponds to
syntax analysis. This task is to extract the precise meaning of a sentence/utterance,
or dictionary meanings defined by the text and reject meaningless, e.g. semantic
analyzer rejects word phrase like hot snowflakes despite correct syntactic words
meaning but incorrect semantic meaning.

1.8.4 Pragmatic Analysis

Pragmatic analysis (Ibileye 2018) is the fourth stage in NLU and a challenging part
in spoken language analysis involving high level or expert knowledge with common
sense, e.g. will you crack open the door? I’m getting hot. This sentence/utterance
requires extra knowledge in the second clause to understand crack is to break in
semantic meaning, but it should be interpreted as to open in pragmatic meaning.
14 1 Natural Language Processing

1.9 Potential Applications of NLP

After years of research and development from machine translation and rule-based
systems to data mining and deep networks, NLP technology has a wide range of
applications in everyday activities such as machine translation, information retrieval,
sentiment analysis, information extraction, and question-answering chatbots as in
Fig. 1.8.

1.9.1 Machine Translation (MT)

Machine translation (Scott 2018) is the earliest application in NLP since 1950s.
Although it is not difficult to translate one language to another yet there are two
major challenges (1) naturalness (or fluency) means different languages have differ-
ent styles and usages and (2) adequacy (or accuracy) means different languages may
present independent ideas in different languages. Experienced human translators
address this trade-off in creative ways such as statistical methods, or case-by-case
rule-based systems in the past but since there have been many ambiguity scenarios
in language translation, the goal of machine translation R&D nowadays strive sev-
eral AI techniques applications for recurrent networks, or deep networks backbox
systems to enhance machine learning capabilities.

Fig. 1.8 Potential NLP applications

1.9 Potential Applications of NLP 15

1.9.2 Information Extraction (IE)

Information extraction (Hemdev 2011) is an application task to extract key lan-

guage information from texts or utterances automatically. It can be structural, semi-
structural machine-readable documents or from users’ languages of NLP in most
cases. Recent activities in complex formats such as audio, video and even interac-
tive dialogue can be extracted from multiple medias. Hence, many commercial IE
programs become domain-specific like medical science, law, or AI Tutor specified
AI knowledge in our case. By doing so, it is easier to set up an ontology graph and
ontology knowledgebase to contain all the retrieved information can be referenced
to these domain knowledge graphs to extract useful knowledge.

1.9.3 Information Retrieval (IR)

Information retrieval (Peters et al. 2012) is an application for organizing, retrieving,

storing, and evaluating information from documents, source repositories, especially
textual information, and multimedia such as video and audio knowledge bases. It
helps users to locate relevant documents without answering any questions explicitly.
The user must make a request for IR system to retrieve the relevant output and
respond in document form.

1.9.4 Sentiment Analysis

Sentiment analysis (Liu 2012) is a kind of data mining system in NLP to analyze
user sentiment towards products, people, ideas from social media, forums, and
online platforms. It is an important application for extracting data from messages,
comments, and conversations published on these platforms; and assigning a labeled
sentiment classification as in Fig. 1.9 to understand natural language and utterances.
Deep networks are ways to analyze large amounts of data. In Part II: NLP
Implementation Workshop will explore how to implement sentiment analysis in
detail using Python spaCy and Transformer technology.
16 1 Natural Language Processing

Fig. 1.9 NLP on sentiment analysis

1.9.5 Question-Answering (Q&A) Chatbots

Q&A systems is the objective in NLP (Raj 2018). A process flow is necessary to
implement a Q&A chatbot. It includes voice recognition to convert into a list of
tokens in sentences/utterances, syntactic grammatical analysis, semantic meaning
analysis of whole sentences, and pragmatic analysis for embedded or complex
meanings. When enquirer’s utterance meaning is generated, it is necessary to search
from knowledge base for the most appropriate answer or response through inferenc-
ing either by rule-based system, statistical system, or deep network, e.g. Google
BERT system. Once a response is available, reverse engineering is required to gen-
erate natural voice from verbal language called voice synthesis. Hence, Q&A sys-
tem in NLP is an important technology that can apply to daily activities such as
human–computer interaction in auto-driving, customer services support, and lan-
guage skills improvement.
The final workshop will discuss how to integrate various Python NLP implemen-
tation tools including NLTK, spaCy, TensorFlow Keras, and Transformer Technology
to implement a Q&A movies chatbot system.

References

Allen, J. (1994) Natural Language Understanding (2nd edition). Pearson

Cui, Y., Huang, C., Lee, Raymond (2020). AI Tutor: A Computer Science Domain Knowledge
Graph-Based QA System on JADE platform. World Academy of Science, Engineering and
Technology, Open Science Index 168, International Journal of Industrial and Manufacturing
Engineering, 14(12), 543 - 553.
Eisenstein, J. (2019) Introduction to Natural Language Processing (Adaptive Computation and
Machine Learning series). The MIT Press.
Goddard, C. (1998) Semantic Analysis: A Practical Introduction (Oxford Textbooks in Linguistics).
Oxford University Press.
Green, B., Wolf, A., Chomsky, C. and Laughery, K. (1961). BASEBALL: an automatic question-
answerer. In Papers presented at the May 9-11, 1961, western joint IRE-AIEE-ACM com-
puter conference (IRE-AIEE-ACM ’61 (Western)). Association for Computing Machinery,
New York, NY, USA, 219–224.
Hausser, R. (2014) Foundations of Computational Linguistics: Human-Computer Communication
in Natural Language (3rd edition). Springer.
Hemdev, P. (2011) Information Extraction: A Smart Calendar Application: Using NLP,
Computational Linguistics, Machine Learning and Information Retrieval Techniques. VDM
Verlag Dr. Müller.
Ibileye, G. (2018) Discourse Analysis and Pragmatics: Issues in Theory and Practice.
Malthouse Press.
Lee, R. S. T. (2020). AI in Daily Life. Springer.
Li, J. et al. (2015) Robust Automatic Speech Recognition: A Bridge to Practical Applications.
Academic Press.
Liu, B. (2012) Sentiment Analysis and Opinion Mining. Morgan & Claypool Publishers.
Peters, C. et al. (2012) Multilingual Information Retrieval: From Research To Practice. Springer.
Raj, S. (2018) Building Chatbots with Python: Using Natural Language Processing and Machine
Learning. Apress.
Santilal, U. (2020) Natural Language Processing: NLP & its History (Kindle edition). Amazon.com.
Scott, B. (2018) Translation, Brains and the Computer: A Neurolinguistic Solution to Ambiguity
and Complexity in Machine Translation (Machine Translation: Technologies and Applications
Book 2). Springer.
Sportier, D. et al. (2013) An Introduction to Syntactic Analysis and Theory. Wiley-Blackwell.
Tuchong (2020a) The Turing Test. https://stock.tuchong.com/image/detail?imag
eId=921224657742331926. Accessed 14 May 2022.
Tuchong (2020b) NLP and AI. https://stock.tuchong.com/image/detail?imag
eId=1069700818174345308. Accessed 14 May 2022.
Turing, A. (1936) On computable numbers, with an application to the Entscheidungs problem. In:
Proc. London Mathematical Society, Series 2, 42:230–26
Turing, A. (1950) Computing Machinery and Intelligence. Mind, LIX (236): 433–460.
Chapter 2
N-Gram Language Model

2.1 Introduction

NLP entities like word-to-word tokenization using NTLK, spaCy technologies in

Workshop 1 (Chap. 10) analyzed words in insolation, but the relationship between
words is important in NLP. This chapter will focus on word sequences, its modeling
and analysis.
In many NLP applications, there are noises and disruptions effecting incorrect
words pronunciation regularly in applications like speech recognition, text classifi-
cation, text generation, machine translation, Q&A chatbots, Q&A conversation
machines or agents being used in auto-driving.
Humans experience mental confusion about spelling errors as in Fig. 2.1 often
caused by pronunciations, typing speeds, and keystroke’s location. They can be cor-
rected by looking up in a dictionary, a spell checker, and grammars usage.
Applying word prediction in a word sequence can provide automatic spell-check
corrections, its corresponding concept terminology can model words relationships,
estimate occurrence frequency to generate new texts with classification, and apply
in machine translation to correct errors.
Probability or word counting method can work on a large databank called corpus
(Pustejovsky and Stubbs 2012) which can be the collection of text documents, lit-
eratures, public speeches, conversations, and other online comments/opinions.

Fig. 2.1 Common spelling COMMON SPELLING ERRORS

errors
1. It’s “calendar”, not “calender”.
2. It’s “definitely”, not “definately”.
3. It’s “tomorrow”, not “tommorrow”.
4. It’s “noticeable”, not “noticable”.
5. It’s “convenient”, not “convinient”.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 19

Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_2
20 2 N-Gram Language Model

A text highlights spelling and grammatic errors in yellow and blue colors is
shown in Fig. 2.2. This method can calculate words probabilities occurrence fre-
quency to provide substitution of higher frequency probability but cannot always
present accurate options.
Figure 2.3 illustrates a simple scenario of next word prediction in sample utter-
ances I like photography, I like science, and I love mathematics. The probability of
I like is 0.67 (2/3) compared with I love is 0.33 (1/3), the probability of like photog-
raphy and like science is similar at 0.5 (1/2). Assigning probability to scenarios, I
like photography and I like science are both 0.67 × 0.5 = 0.335, and I love mathe-
matics is 0.33 × 1 = 0.33.

Fig. 2.2 Spelling and grammar checking tools

Fig. 2.3 Next word

prediction in simple
utterances
2.2 N-Gram Language Model 21

When applying probability on language models, it must always note (1) domain
specific verity of keywords togetherness and terminology knowledge varies accord-
ing to domains, e.g. medical science, AI, etc., (2) syntactic knowledge attributes to
syntax, lexical knowledge, and (3) common sense or world knowledge attributes to
the collection of habitual behaviors from past experiences, and (4) languages usage
significance in high-level NLP.
When applying probability on words prediction in an utterance, there are words
often proposed by rank and frequency to provide a sequential optimum estimation.
For example:
[2.1] I notice three children standing on the ??? (ground, bench …)
[2.2] I just bought some oranges from the ??? (supermarket, shop …)
[2.3] She stopped the car and then opened the ??? (door, window, …)
The structure of [2.3] is perplexed because word counting method with a sizeable
knowledge domain is adequate but common sense, world knowledge, or specific
domain knowledge are among the sources. It involves scenario syntactic knowledge
that attributes to do something with superior level at scene such as descriptive
knowledge to help the guesswork. Although it is plain and mundane to study pre-
ceding and words tracking but it is one the most useful techniques on words predic-
tion. Let us begin with some simple word counting methods in NLP, the N-gram
language model.

2.2 N-Gram Language Model

It was learnt that the motivations on words prediction can apply to voice recogni-
tion, text generation, and Q&A chatbot. N-gram language model, also called
N-gram model or N-gram (Sidorov 2019; Liu et al. 2020) is a fundamental method
to formalize words prediction using probability calculation. N-gram is statistical
model that consists of word sequence in N-number, commonly used N-grams
include:
• Unigram refers to a single word, i.e. N = 1. It is seldomly used in practice because
it contains only one word in N-gram. However, it is important to serve as the base
for higher order N-gram probability normalization.
• Bigram refers to a collection of two words, i.e. N = 2. For example: I have, I do,
he thinks, she knows, etc. It is used in many applications because its occurrence
frequency is high and easy to count.
• Trigram refers to a collection of three words, i.e. N = 3. For example: I noticed
that, noticed three children, children standing on, standing on the. It is useful
because it contains more meanings and not lengthy. Given a count knowledge of
first three words can easily guess the next word in a sequence. However, its
occurrence frequency is low in a moderate corpus.
22 2 N-Gram Language Model

• Quadrigram refers to a collection of four words, i.e. N = 4. For example: I noticed

that three, noticed that three children, three children standing on. It is useful with
literatures or large corpus like Brown Corpus because of their extensive words’
combinations.
A sizeable N-gram can present more central knowledge but can pose a problem.
If is too large, it means that probability and occurrence of word sequence is infre-
quent and even 0 in terms of probability counts.
Corpus volume and other factors also affect performance. N-gram model training
is based on an extensive knowledge base (KB) or databank from specific domains
such as public speeches, literatures, topic articles like news, finance, medical, sci-
ence, or chat messages from social media platforms. Hence, a moderate N-gram is
the balance by frequency and proportions.
The knowledge of counts acquired by a N-gram can assess to conditional prob-
ability of candidate words as the next word in a sequence, e.g. It is not difficult. It is
a bigram which means to count the occurrence of is given that it has already men-
tioned from a large corpus, or the conditional probability of it is given that it has
already mentioned or can apply it one by one to calculate the conditional probability
of an entire words sequence. It is like words and sentences formation of day-to-day
conversations which is a psychological interpretation in logical thinking. N-gram
progresses in this orderly fashion.
It serves to rank the likelihood of a sequence consisting of various alternative
hypotheses in a sentence/utterance for application like automatic speech recogni-
tion (ASR), e.g. [2.4] The cinema staff told me that popcorn/amelcorn sales have
doubled. It is understood that it refers to popcorn and not amelcorn because the
concept of popcorn is always attributed to conversations about cinema. Since the
occurrence of popcorn in a sentence/utterance has a higher rank than amelcorn, it is
natural to select popcorn as the best answer.
Another purpose is to assess the likelihood of a sentence/utterance for text gen-
eration or machine translation, e.g. [2.5] The doctor recommended a cat scan to the
patient. It may be difficult to understand what a cat scan is or how can a scan be
related to a cat without any domain knowledge. Since the occurrence of doctor is
attributed to medical domains, it is natural to search articles, literatures, websites
about medical knowledge to learn that CAT scan refers to a computerized axial
tomography scanner as in Fig. 2.4 instead of a cat. This type of words prediction is
often domain specific associated with the preceding word as guidance to select an
appropriate expression.

2.2.1 Basic NLP Terminology

Here is a list of common terminologies in NLP (Jurafsky et al. 1999; Eisenstein 2019):
• Sentence is a unit of written language. It is a basic entity in a conversation or
utterance.
2.2 N-Gram Language Model 23

Fig. 2.4 Computerized axial tomography scanner (aka. CAT scan) (Tuchong 2022)

• Utterance is a unit of spoken language. Different from the concept of sentence,

utterance is usually domain and culture specific which means it varies according
to countries and even within country.
• Word Form is an inflected form occurs in a corpus. It is another basic entity in
a corpus.
• Types/Word Types are distinct words in a corpus.
• Tokens are generic entities or objects of a passage. It is different from word form
as tokens can be meaningful words or symbols, punctuations, simple and distinct
character(s).
• Stem is a root form of words. Stemming is the process of reducing inflected, or
derived words from their word stem.
• Lemma is an abstract form shared by word forms in the same stem, part of speech,
and word sense. Lemmatization is the process of grouping together the inflected
forms of a word so that they can be analyzed as a single item which can be identi-
fied by the word’s lemma or dictionary form.
An example to demonstrate meaning representations between lemma and stem is
shown in Fig. 2.5. Lemmatization is the abstract form to generate a concept. It indi-
cated that stem or root word can be a meaningful word, or meaningless, or a symbol
such as inform or comput to formulate meaningful words such as information, infor-
mative, computer, or computers.
There are several corpora frequently used in NLP applications.
Google (2022) is the largest corpus as it contains words and texts from its search
engine and the internet. It has over trillion English tokens with over million mean-
ingful wordform types sufficient to generate sentences/utterances for daily use.
24 2 N-Gram Language Model

Fig. 2.5 Stemming vs. lemmatization

Brown Corpus (2022) is an important and well-known corpus because it is the

first well-organized corpus in human history founded by Brown University from
1961 with continuous updates. At present, it has over 583 million tokens, 293,181
wordform types and words in foreign languages. It is one of the most comprehen-
sive corpora for daily use, and a KB used in many N-grams, related NLP models and
applications.
Further, there are many domain specific corpora such as Wall Street Journal is
one of earliest corpora to discover knowledge from financial news, Associated Press
focus on news and world events, Hansard is a prominent Corpus of British
Parliament speeches, Boston University Radio News corpus, NLTK Corpora Library
and others etc. (Bird et al. 2009; Eisenstein 2019; Pustejovsky and Stubbs 2012).
Let us return to words prediction. A language model is a traditional word count-
ing model to count and calculate conditional probability to predict the probability
based on a word sequence, e.g. when applying to utterance it is difficult to… that
with a sizeable corpus like Brown Corpus, the traditional word counting method
may suggest either say/tell/guess based on occurrence frequency. This traditional
language model terminology is applied to predictions and forecasts at advanced
computer systems and research in specialized deep networks and models in
AI. Although there has been a technology shift, statistical model is always the fun-
damental model in many cases (Jurafsky et al. 1999; Eisenstein 2019).

2.2.2 Language Modeling and Chain Rule

Conditional probability calculation is to study the definition of conditional proba-

bilities and look for counts, given by

P ( A ∩ B)
P ( A|B ) = (2.1)
P ( B)

For example, to evaluate conditional probability: The garden is so beautiful that

given by the word sequence “The garden is so beautiful” will be
2.2 N-Gram Language Model 25

P ( The garden is so beautiful that )

P ( that|The garden is so beautiful ) =
P ( The garden is so beautiful )
(2.2)
Count ( The garden is so beautiful that )
=
Count ( The garden is so beautiful )

Although the calculation is straightforward but if the corpus or text collection is

moderate, this conditional probability (counts) will probably be zero.
Chain rules of probability is useful as an independent assumption to rectify this
problem.
By rewriting the conditional probability equation (2.1), it will be

P ( A ∩ B ) = P ( A|B ) P ( B ) (2.3)

For a sequence of events, A, B, C and D, the Chain Rule formulation will become

P ( A,B,C ,D ) = P ( A ) P ( B|A ) P ( C|A, B ) P ( D|,A|,B|,C ) (2.4)

In general:

P ( x1 ,x2 ,x3 , … xn ) = P ( x1 ) P ( x2 |x1 ) P ( x3 |x1 , x2 )… P ( xn |x1 … xn −1 ) (2.5)

If word sequence from position 1 to n as w1n is defined, the Chain Rule applied to
word sequence will become

P ( w1n ) = P ( w1 ) P ( w2 |w1 ) P ( w3 |w12 )… P ( wn |w1n −1 )

n (2.6)
= ∏P ( wk |w1k −1 )
k =1

So, the conditional probability for previous example will be

P ( the garden isso beautiful that ) = P ( the ) ∗ P ( garden|the )

∗

P ( is|the garden ) ∗ P ( so|the garden is ) ∗ P ( beautiful|the garden is so ) (2.7)

∗

P ( that|the garden isso beautiful )

Note: Normally, <s> and </s> are used to denote the start and end of sentence/
utterance for better formulation.
This method seems fair and easy to understand but poses two major problems.
First, it is unlikely to gather the right statistics for prefixes which means that not
knowing the starting point of the sentence. Second, the calculation for word
sequence probability is mundane. If it is a long sentence, conditional probability at
the end of this equation is complex to calculate.
Let us explore how genius Markov Chain is applied to solve this problem.
26 2 N-Gram Language Model

2.3 Markov Chain in N-Gram Model

Prof. Andrey Andrevevich Markov (1856–1922) is a renowned Russian mathemati-

cian and academician who made significant contribution to science studying the
theory of probability. His primary contribution called Markov chains or Markov
process had applied to biology, chemistry, computer science, and statistics (Ching
et al. 2013). Markov chains theory can be applied to speech recognitions, N-gram
language model, internet ranking, information, and queueing theories (Eisenstein
2019). There is a single dimension domain Markov chain modeling called Hidden
Markov Chain in handwritten characters and human voice recognitions. This model
has an important concept called Markov Assumption which assumes the entire pre-
fix history is not necessary, in other words, an event does not depend on its whole
history; it is only a fixed length nearby history is the essence of Markov chain theory.
Markov chain is a stochastic process to describe a sequence of possible events
which the probability of each event depends only on the state attained in previous
event. There are many kinds of Markov chain conventions. An important convention
called descriptive Markov chain is shown in Fig. 2.6. It revealed that an event of
Markov chain can be a list of relationship of every single event. Another concept is
that the previous state is important but not all previous sequences. Hence, this model
can apply in thermodynamics, statistical mechanics, physics, chemistry, economy,
finance, information theory, and NLP. A complete Markov Chain event is like a
conversation in a sentence/utterance, each word is equivalent to an object in Markov
chain. Although the whole chain of conditional probability can be calculated but the
last event is the most important one.
By applying Markov chain model, the conditional probability for N-gram prob-
ability of a word sequence w1n will be approximated by (assuming a prefix of
N words)

P ( wn |w1n −1 ) ≈ P ( wn |wnn−−1N +1 ) (2.8)

In general:
n
P ( w1n ) ≈ ∏P ( wk |wk −1 ) (2.9)
k =1

Fig. 2.6 Markov chain model

2.4 Live Example: The Adventures of Sherlock Holmes 27

In other words, the original complex conditional probability of a word sequence

stated in Eq. (2.6) can be easily evaluated by a sequence of bigram probability
calculations.
Let us look at an N-gram example The big white cat. Unigram probability is
P(cat). Bigram probability is the P(cat | white). Trigram probability is P(cat | big
white) = P(white | big) * P(cat | white) and quadrigram probability is P(cat | the big
white) = P(big | the) * P(white | big) * P(cat | white). All can be easily evaluated by
simple sequence of bigram probability multiplications applying Eq. (2.9).
However, it is cautious to note that the probability of a word formulation given
fixed prefixes may not always appropriate in many cases. They may be verifiable
events in real-time speeches as words uttered are often correlated to the previous but
in cases with pragmatic or embedded meanings at both right and left contexts, there
is no priori reason adhered to left contexts.

2.4 Live Example: The Adventures of Sherlock Holmes

N-gram probability calculation usually comes from a training corpus or knowledge

base (KB) in two extremes. One is an overly narrow corpus and the other one is an
overly general corpus. An overly narrow corpus is a restricted, specific corpus, can
be domain specific on a particular knowledge with significant counts to be found
during conditional probability counting. An overly general corpus cannot reflect a
specific domain but counting can always be found. Hence, a balance between the
two dimensions is required. Another consideration is a separate text corpus applied
to evaluate in standard metrics called held-out test set, or development test set.
Further, cross validations and results tested for statistical significance are also
required.
Let us begin with a corpus came from Project Gutenberg website (Gutenberg
2022) on The Adventures of Sherlock Holmes (Doyle 2019), a famous literature by
writer and physician Sir Arthur Canon Doyle (1859–1930). Gutenberg is a website
consisting of primarily copyrights clearance, free access, and download western
cultural tradition literatures available to public. This literature has 12 outstanding
detective stories of Sherlock Holmes ranging from A scandal in Bohemia to The
Adventure of the Copper Beeches with other statistics below. It is a domain specific
corpus with comprehensive detective knowledge to form a meaningful KB and per-
form N-gram modeling in NLP.
No. of pages : 424
No. of characters (exclude spaces) : 470,119
No. of words : 110,087
No. of tokens : 113,749
No. of sentences : 6830
No. of word types (V) : 9886
28 2 N-Gram Language Model

N-gram modeling in this example is to analyze an influential quote of Sherlock

Holmes: I have no doubt that I …. This quote does not occur often in other litera-
tures but because it is a detective story, the character has unique aptitude for deduc-
ing hypotheses and notions to solve cases. Applying Markov chain model can avoid
mundane conditional probability, the N-gram probability is given by

P ( I have no doubt that I ) = P ( I|s ) ∗ P ( have|I ) ∗ P ( no|have )

∗

(2.10)
P ( doubt|no ) ∗ P ( that|doubt ) ∗ P ( I|that )

Unigram checking on word counting for I have no doubt is necessary as basis to

calculate the conditional probability for all bigrams as shown in Fig. 2.7. So, given
the unigram count of I is 2755, the bigram probability of I have applying Markov
chain method will be 288/2755, which is 0.105 as shown in Fig. 2.8. It is a list of all
related bigram counts and probabilities for a given bigram such as I have, I had, I
am, I was, I knew, I hear, I don’t up to I should which are common words found in
many literatures. The probability also showed I have is the most frequent with 0.105
which means that I have no doubt that is quoted by the character regularly. The
occurrence of I think is high and general phrases such as I have, I had.
A bigram grammar fragments related to I have no doubt that … is shown in
Fig. 2.9 for the counting and probability occurrence frequency beginning with <s>I
, <s>I’d, <s>The, <s>It, I have, I had, I can, have no, have to, have been to com-
pare with several versions or combinations related to, I have no doubt that means to
compare occurrence frequency of I have with I had or I can, which is similar to
compare occurrence of no doubt, no sign, and no harm or that I, that he, that she,
that it. It is noted that the occurrence of I have no doubt that is high and distinct in
this literature.

Fig. 2.7 Unigram counts for words “I have no doubt that” from The Adventures of Sherlock Holmes

Fig. 2.8 Bigram grammar fragment from The Adventures of Sherlock Holmes
2.4 Live Example: The Adventures of Sherlock Holmes 29

Fig. 2.9 Bigram grammar fragment related to utterance “I have no doubt that I” from The
Adventures of Sherlock Holmes

Counting all conditional bigram probabilities based on unigram count in Fig. 2.7
showed I have no doubt that for I is at 0.138 which is very high, but it is interested
to note that no doubt is even higher at 0.167 but again since it is a detective story
with a restricted domain, doubt that is very high at 0.202 because the character
always involves guesswork and frequent grammar usage. Further, the probability of
bigram that I is much higher than other combination like that he, that she, and that
it. The occurrence frequency in other literatures is much lower but because the char-
acter is a self-assured and intelligent expert, so he said that I is more often than that
he or that she. That is the significance of a domain specific KB/corpus to check for
N-gram probability.
So, let us look at some N-gram probabilities calculation, e.g. the probability of
P(I have no doubt that I) given by Eq. (2.10) :

P ( I have no doubt that I ) = 0.138 × 0.105 × 0.040 × 0.167 × 0.202 × 0.129

= 0.000002526

It is compared with P(I have no doubt that he):

P ( I have no doubt that he ) = 0.138 × 0.105 × 0.040 × 0.167 × 0.202 × 0.079

= 0.000001540

This example test results led to several observations. It is noted that all these
probabilities are limited in general. Conditional probability is limited in a long sen-
tence and required for Markov chain. If applying traditional method on conditional
probability with complex calculation, most of the time the probability is dimin-
ished. Further, the probability seems to capture both syntactic facts and world
knowledge. Although that I or that he are often used in English grammar, the prob-
ability in this literature that I is more frequent. Hence, it is related to both syntactic
usage, common sense, and specific domain knowledge. It depends on KB domains
leading to diverse probability calculation results.
It is also noted that most of the conditional probabilities are limited because the
multiplication of all probability calculation in a long sentence becomes diminished,
so it is important to apply Markov chain and convert complex conditional
30 2 N-Gram Language Model

probabilities into bigram probabilities. Although the occurrence of bigram is infre-

quent but still exists. Nevertheless, if it is not sizeable KB or corpus, most of the
bigrams will be 0. Hence, the selection for corpus KB or corpus is important. An
effective N-gram is related to word counting, conditional probabilities calculation,
and normalization.
Another observation is that it showed all these conditional probabilities are lim-
ited and underflows as mentioned. A method is to convert them into natural log. By
applying natural log will become additions to calculate conditional probability with
Markov chain operation.
Maximum Likelihood Estimates (MLE) is another principal method to calculate
N-gram model. They are parameters of a model M from a training set T. It is the
estimate that maximizes the likelihood of training set T given the model M. Suppose
the word language, for example, occurred 380 times in a corpus with a million
words, e.g. Brown corpus, the probability of a random word from other text forms
with the same distribution will be language, which it will be 380/1,000,000 = 0.00038.
This may be a poor estimate for other corpora but this type of calculation is domain
specific as mentioned meaning that the calculation varies according to different
corpora.
Let us return to The Adventures of Sherlock Holmes’ famous quote I have no
doubt that example. This time the counting and probability calculation of these
words are tabulated as shown in Figs. 2.10 and 2.11, respectively. It showed that I
have has the most occurrence frequency with 288, that I is the next with 228 occur-
rences, no doubt with surprising high 46 occurrences, doubt that is 17 followed by
that no and so on. Another discovery is that most of the other combinations is 0. It
is intuitive because they are not grammatically or syntactically possible, e.g. no I or
I, I and many are infrequent in English usage.

Fig. 2.10 Bigram counts for “I have no doubt that I” in The Adventures of Sherlock Holmes

Fig. 2.11 Bigram probability (normalized) for “I have no doubt that I ” in The Adventures of
Sherlock Holmes
2.5 Shannon’s Method in N-Gram Model 31

Bigram normalization is achieved by the division of each bigram counts by

appropriate unigram counts for wn − 1. Here is the bigram normalization result for I
have no doubt that, e.g. computing bigram probability of no doubt is the counting
of no doubt which is 46 as shown in Fig. 2.10 against the counting of no which is
276 as in Fig. 2.7 which becomes 46/276 = 0.167. In fact, such bigram probability
P(no doubt) is much higher than P(I have) = 0.105, which is infrequent in other
corpora because not many corpora have frequency of no doubt as compared with I
have as I have is common in English usage. Since it is a detective literature and the
character is an expert at his field, it is unsurprised to identify the occurrence fre-
quency of no doubt is very high.
The overall bigram probability (normalized) findings are I have is 0.105, no
doubt is 0.167 the highest, that I is 0.129 as shown in Fig. 2.11. This is special
because the occurrence frequency of I is not high as compared with I have. doubt
that is 0.202 which is very high, and others are mostly 0. These findings showed
that, first, all conditional probabilities are limited because of N-gram calculation
characteristics comes from an extensive corpus. But it does not mean that there is no
comparison. It can be compared if they are not 0. Second, 0s are meaningful as most
of these words’ combinations are neither syntactically nor grammatically incorrect.
Third, these conditional probabilities and MLE are domain specific, which may not
be the same in other situations.

2.5 Shannon’s Method in N-Gram Model

Shannon’s method (Jurafsky et al. 1999) is another important topic in N-gram

model. Prof. Claude Shannon (1916–2001) is a renowned American mathematician,
electrical engineer, cryptographer, also known as the father of information theory
and a major founder of contemporary cryptography. He wrote his famous thesis at
age 21, a master’s degree student at MIT demonstrating Boolean algebra electrical
applications to construct any logical numerical relationship with meaning. One of
his most influential papers, A mathematical theory of communications (Shannon
1948) published in 1948 had defined a mathematical notion by which information
could be quantified and delivered reliably over imperfect communication channels
like phone lines or wireless connections nowadays. His groundbreaking innovation
had provided the tools for network communications and internet technologies. This
method showed that assigning probabilities to sentences are well but less informa-
tive for language generation in NLP. However, it has a more interesting task to turn
it around by applying N-gram and its probabilities to generate random sentences
like human sentences by which the model is derived.
There are four steps of Shannon’s Method for language generation as shown in
Fig. 2.12:
An example of four N-gram texts generation methods from The Complete Works
of Shakespeare by William Shakespeare (1564–1616) (Shakespeare 2021) applying
Shannon’s Method is shown in Fig. 2.13.
32 2 N-Gram Language Model

Fig. 2.12 Algorithm of Shannon’s method for language generation

Fig. 2.13 Sentence generation using Shannon method from “The Complete Works of Shakespeare”

In summary:
• Unigram results showed that the four random sentences are almost meaningless
because it used a single word to calculate probability that is mostly without
relations.
• Bigram results showed that the four random sentences have little meanings
because it used two words to calculate. It reflected its high occurrence probabil-
ity frequency but not grammatically correct.
• Trigram results showed that words relations are coherent because it used three
words to calculate. It reflected the conditional probability ranking had improved
grammar and meanings like human language.
2.5 Shannon’s Method in N-Gram Model 33

• Quadrigram results showed that the language of sentences is almost perfect per
original sentences since it used four words co-relation to calculate, but its high
occurrence conditional probability frequency are the words encountered with
low-ranking options due to copious information to search. It may not be benefi-
cial to text generation.
Although quadrigrams can provide realistic language, sentence/utterance but
lack freedoms to generate new sentences. Hence, trigram is often a suitable option
for language generation. Again, if corpus is not sizeable enough to accommodate
tokens and words volume like this literature, trigram will be unable to provide the
frequent words for N-gram may need to switch using bigram in this case. Hence,
quadrigram is unsuitable for text generation because it will be too close to the origi-
nal words or sentences.
Corpus used in this example is also domain specific from The Complete Works of
Shakespeare. It consists of 884,647 tokens and 29,066 distinct words that are
approximately 10 times more as compared with The Adventures of Sherlock Holmes.
It has approximately 300,000 bigram types out of all these tokens and the number
of bigram combinations will be 844 million possible bigrams. In other words, less
than 1% is used and other 99.96% of possible bigrams are never used. It makes
sense because most of these random bigram generations are grammatic, syntactic,
or even pragmatic meaningless, but pose a problem in N-gram calculations for text
generation.
For illustration purposes on how domain knowledge affects N-gram generation,
Fig. 2.14 shows some sample sentences generated by Wall Street Journal (WSJ)
articles as the corpus (Jurafsky et al. 1999). It showed that trigram has the best per-
formance in terms of sentence structure and meaningfulness on text generation.

Fig. 2.14 Sample sentence generation using Shannon method with Wall Street Journal articles
34 2 N-Gram Language Model

2.6 Language Model Evaluation and Smoothing Techniques

Language Model Evaluation (LME) (Jurafsky et al. 1999) is a standard method to

train parameters on a training set and to review model performance with new data
constantly. That is often occurred in real world to learn how the models perform
called training data (training set) on language model and see whether it works with
unseen information called test data (test set). A test set is a dataset completely dif-
ferent than the training set model but is drawn from the same source which is a
specific domain. Then apply an evaluation metric, e.g. perplexity to determine lan-
guage model effectiveness.
Unknown words are words unseen prior looking at test data regardless of how
much training data is available. It can be managed by an open vocabulary task with
steps below:
1. Create an unknown word token <UNK>
2. Train <UNK> probabilities
(a) Create a fix lexicon L, of size V from a dictionary or a subset of terms from
the training set
(b) A subnet of terms from the training set
(c) At text normalization phase, any training word not in L changed o <UNK>
(d) Now can count that like a normal word
3. Test
(a) Use <UNK> counts for any word not in training.

2.6.1 Perplexity

Perplexity (PP) is the probability of test set assigned by the language model, nor-
malized by the number of words as given by

1
PP (W ) = N (2.11)
P ( w1 w2 … wN )

By applying Chain rule, it will become

N
1
PP (W ) = N ∏ (2.12)
k =1 P ( wk |w1 w2 … wk −1 )

For bigrams, it will be given by

N
1
PP (W ) = N ∏ (2.13)
k =1 P ( wk |wk −1 )
2.6 Language Model Evaluation and Smoothing Techniques 35

Fig. 2.15 Perplexity values for WSJ from unigram to trigram

In general, minimizing perplexity is the same as maximizing probability for

model performance, which means the best language model is the one that can best
predict an unseen test set with minimized perplexity rate.
An example of perplexity values for WSJ is shown in Fig. 2.15 indicating that
trigram with minimized perplexity has performed better than bigram and unigram
supported this principle for text generation (Jurafsky et al. 1999).

2.6.2 Extrinsic Evaluation Scheme

An extrinsic evaluation is a popular method for N-gram evaluation, its theory is

straightforward as follows:
1. Put model A into an application, e.g. a speech recognizer or even a QA chatbot;
2. Evaluate application performance with model A;
3. Put model B into the application and evaluate;
4. Compare two models’ application performance.
The good thing about extrinsic evaluation is that it can perform exact testing at
two models which is fair and objective, but it is time-consuming for system testing
and implementations, i.e. take days to perform experiments if is a sophisticated
system. So, a temporary solution is to use intrinsic evaluation with an approxima-
tion called perplexity to evaluate N-gram. It is easier to implement if the same sys-
tem is used but perplexity is a poor approximation unless the test data looks identical
to the training data. Hence, it is generally useful in pilot experiments.

2.6.3 Zero Counts Problems

Next step is to manage zero counts problems. Let us return to The Adventures of
Sherlock Holmes example, this literature had produced 109,139 bigram types over
100 million of possible bigrams as recalled, so there is approximately 99.89% of
possible bigrams never seen that have zero entries in the bigram table. In other
words, most of these 0s conditional probability are bigrams that required to manage
especially in different NLP applications like text generation and speech
recognition.
There is a brief synopsis in such zero-count dilemma. Some of these 0s are truly
zeros which means that cannot or should not occur because it will not make
36 2 N-Gram Language Model

grammatical or syntactic sense, however, some are only rare events which means
they occurred infrequently, i.e. with an extensive training corpus.
Further, Zipf’s law (Saichev et al. 2010) stated that, a long tail phenomenon is
rare events occurred in a very high frequency, and large events numbers occurred in
a low frequency constantly. These are two extremes which means some popular
words always occurred in a high frequency, and most are bigrams in low frequency.
Hence, it is clear to collect statistics on high frequency events and may have to wait
for a long time until a rare event occurs, e.g. a bigram to take a count on this low
occurrence frequency event. In other words, high occurrence frequency events
always dominate the whole corpus. This phenomenon is essential because it always
occurs in website statistics or website counting. These high frequency websites and
N-grams are usually the top 100 and others with limited visit counts and occurrence,
so the estimate results are sparse as there are neither counts nor rare events that
required to estimate the likelihood of unseen or 0 count N-grams.

2.6.4 Smoothing Techniques

Every N-gram training matrix is sparse even with large corpora because of Zipf’s
law phenomenon. The solution is to use likelihood estimation for figures on unseen
N-grams or 0 count N-grams to judge the rest of corpus accommodated with these
phantom/shadow N-grams. It will affect the rest of corpus.
Let us assume that an N-gram is used, all the words are known and seen before-
hand. When assigning a probability to a sequence where one of these components is
0, the initial process is to search for a low N-gram order and backoff from a bigram
to unigram and replace 0 with something else, or a value with several methods to
resolve zero count problems based on this concept; these collective methods are
called smoothing techniques.
This section explores four commonly used smoothing techniques: (1) Laplace
(Add-one) Smoothing, (2) Add-k Smoothing, (3) Backoff and Interpolation
Smoothing, and (4) Good Turing Smoothing (Chen and Goodman 1999; Eisenstein
2019; Jurafsky et al. 1999).

2.6.5 Laplace (Add-One) Smoothing

Laplace (Add-one) Smoothing (Chen and Goodman 1999; Jurafsky et al. 1999)
logic is to consider all zero counts are rare events and add 1 into them. These rare
events are neither occurred nor sampled during corpus training.
For unigram:
1. Add 1 to every single word (type) count.
2. Normalize N token/(N (tokens) + V (types)).
2.6 Language Model Evaluation and Smoothing Techniques 37

3. Smooth count ci∗ (adjusted for additions to N) given by

N
ci∗ = ( ci + 1) (2.14)
N +V

4. Normalize N to obtain a new unigram probability p∗given by

ci + 1
p∗ = (2.15)
N +V

For bigram:
1. Add 1 into every bigram c(wn − 1wn) + 1.
2. Increase unigram count by vocabulary size c(wn − 1) + V.
Figure 2.16 showed a bigram count with and without Laplace Method for previ-
ous example I have no doubt that I from The Adventures of Sherlock Holmes. It
indicated that all 0s become 1 so that no I becomes 1, others like I have will come
from 288 to 289, the calculation is simple but effective.
For bigram probability calculation is given by

C ( wn −1 wn )
P ( wn |wn −1 ) = (2.16)
C ( wn −1)

So, the bigram probability with Laplace method will be given by

C ( wn −1 wn ) + 1 C ( wn −1 wn ) + 1
PLap ( wn |wn −1 ) = = (2.17)
∑ (C ( w
w n −1 w ) + 1) C ( wn −1 ) + V

Figure 2.17 shows the bigram probabilities with and without Laplace Method for
previous example I have no doubt that I from The Adventures of Sherlock Holmes.
Note: The bigram probability is calculated by the division of unigram originally
but now it will be the division by the count of unigram + total number of word type
(V) which is equal to 9886 e.g. P(have | I) = 288/2755 = 0.105. Applying Laplace
method, it becomes 289/(2755 + 9886) = 0.023. It showed that all zero cases will
become 1 which is simple for text generation, but the problem is, some probabilities

Orginal Bigram Table of "I have no doubt that I" Bigram Table of "I have no doubt that I" with Laplace Method
(By Bigram Count) (By Bigram Count)

I have no doubt that I I have no doubt that I

I 0 288 0 1 0 0 I 1 289 1 2 1 1
have 5 0 35 0 2 5 have 6 1 36 1 3 6
no 0 0 0 46 0 0 no 1 1 1 47 1 1
doubt 0 0 0 0 17 0 doubt 1 1 1 1 18 1
that 228 1 10 0 7 228 that 229 2 11 1 8 229

Fig. 2.16 Bigram counts with and without Laplace method

38 2 N-Gram Language Model

Orginal Bigram Table of "I have no doubt that I" Bigram Table of "I have no doubt that I" with Laplace Method
(By Bigram Probability) (By Bigram Probability)

I have no doubt that I I have no doubt that I

I 0.00000 0.10454 0.00000 0.00036 0.00000 0.00000 I 0.00008 0.02286 0.00008 0.00016 0.00008 0.00008
have 0.00577 0.00000 0.04037 0.00000 0.00231 0.00577 have 0.00056 0.00009 0.00335 0.00009 0.00028 0.00056
no 0.00000 0.00000 0.00000 0.16667 0.00000 0.00000 no 0.00010 0.00010 0.00010 0.00463 0.00010 0.00010
doubt 0.00000 0.00000 0.00000 0.00000 0.20238 0.00000 doubt 0.00010 0.00010 0.00010 0.00010 0.00181 0.00010
that 0.12903 0.00057 0.00566 0.00000 0.00396 0.12903 that 0.01965 0.00017 0.00094 0.00009 0.00069 0.01965

Fig. 2.17 Bigram probabilities with and without laplace method

have changed notably such as I have from 0.105 to 0.023, and no doubt has the high-
est change from 0.1667 to only 0.00463.
Although it is adequate to assign a number to all zero events but the one with
high frequency becomes insignificant because of copious word types in corpus base,
indicating that the performance of Laplace Add-one smoothing may not be effective
in many cases and required to look for alternatives.

2.6.6 Add-k Smoothing

Add-k Smoothing (Chen and Goodman 1999; Jurafsky et al. 1999) logic is to assume
that each N-gram is seen in k times, but the occurrence is too rare to be observed.
These zeros are rare events that are less than 1 and unnoticeable meaning that there
is a line between 0 and 1, it can be 0.1, 0.01, 0.2 or even smaller; so a non-integer
count is added instead of 1 to each count, e.g. 0.05, 0.1, 0.2, typically, 0 < k < 1
provided that k must be a small number less than 1 in practical applications. It is
because if k is too large, it will cause similar problem occurred in Laplace method.
By using the same logical as Add-1 method, Add-k Smoothing is given by

C ( wn −1 wn ) + k
− k ( wn |wn −1 ) =
∗
PAdd (2.18)
C ( wn ) + kV

where 0 < k < 1.

It is adequate to compare with Laplace method is that the whole V is not used if
V is very large such as 9886 in The Adventure of Sherlock Holmes. When the event
is, say 0.05, means that it will be even smaller, but the new number will not be too
small. Although add-k is useful for many tasks including text classification and
generation, but not for all language modeling, generating counts with poor variance
and often inappropriate discounts (Gale and Church 1994). Another add-k model
consideration is to select an appropriate k number through trial and error but that
will lead to problems in practical applications. Nevertheless, Add-k smoothing usu-
ally provides a better and viable solution as compared with Add-1 method.
2.6 Language Model Evaluation and Smoothing Techniques 39

2.6.7 Backoff and Interpolation Smoothing

Backoff and Interpolation (B&I) Smoothing (Chen and Goodman 1999; Suyanto
2020) logic is to look for a lower dimension N-gram if there is no example of a
particular N-gram. If N − 1 gram has insufficient number count (or does not exist),
then will switch to N − 2 gram and so on. Although it is not the perfect option but
at least it can produce some viable counting for words prediction. That is to estimate
a probability with a bigram instead of trigram if there is none to be found.
Furthermore, it can look up to unigram if no bigram either. This is a kind of backoff
method and by interpolation, can always weight and combine with quadrigram,
trigram, bigram, and unigram probabilities counts, e.g. when calculating trigram
probability with unigram, bigram, and trigram, each weighted by some λ values.
Note the sum of all λs must be 1 given by these equations:

PB & I ( wn |wn − 2 wn −1 ) = λ1 P ( wn )
+λ2 P ( wn |wn −1 ) (2.19)
+λ3 P ( wn |wn − 2 wn −1 )

For a sophisticated version of linear interpolation, each λ value can be calculated

by conditioning on the context which means it can be done by using conditional
probabilities as well. In this way, if a particular bigram has accurate numbers, it can
assume that the trigrams numbers are based on this bigram, which will be a robust
method to implement given by the following equation:

PB & I ( wn |wn − 2 wn −1 ) = λ1 ( wn − 2:n −1 ) P ( wn )

+λ2 ( wn − 2:n −1 ) P ( wn |wn −1 ) (2.20)
+λ3 ( wn − 2:n −1 ) P ( wn |wn − 2 wn −1 )

It is noted that by comparing with previous Eq. (2.19), this equation also consid-
ers conditional probability in all N-gram levels. Hence, both simple interpolation
and conditional interpolation methods are learnt from a held-out corpus. A held-out
corpus is an additional training corpus to set hyperparameters like λ values by
choosing λ values that can maximize the likelihood of held-out corpus. By adjusting
N-gram probabilities and search for λ value is to provide the highest probability of
held-out set. In fact, there are numerous approaches to find this optimal set of λ, a
simple way is applying EM algorithm which is an interactive learning algorithm to
converge locally optimal λ.
40 2 N-Gram Language Model

2.6.8 Good Turing Smoothing

Good Turing (GT) Smoothing (Chen and Goodman 1999; Gale and Sampson 1995)
logic is to use the total frequency of events that occurred only once to estimate how
much mass shift to unseen events, e.g. using a bag of green color beans to estimate
the probability of an unseen red color bean.
This technique uses the frequency of N-grams occurrence to reallocate probabil-
ity distribution in two criteria, e.g. N-gram statistics of The Adventures of Sherlock
Holmes in Fig. 2.17. It showed that the probability of have doubt = 0 without
smoothing, so by using bigrams frequency that occurred once, i.e. probability of I
doubt to represent the total number of bigrams for unknown bigrams given by

Count of bigrams that appeared once

Punknown ( wi |wi −1 ) = (2.21)
Count of total bigrams

It is an intuitive method because it only considers conditional probability of big-

rams that occurred once to represent unknown probabilities instead of adding 1 to
them. In other words, the conditional probability of unknown bigram of word will
be the count for bigram that occurred once over the count of total bigrams.
For known bigrams such as no doubt, the frequency of bigrams that occurred
more than one of the current bigram frequency Nc+1, frequency of bigrams that
occurred the same as the current bigram frequency Nc, and the total number of big-
ram N are given by

c∗
Pknown ( wi |wi −1 ) =
N
N c +1
where c∗ = ( c + 1) ∗ and c = count of input bigram. (2.22)
Nc

Exercise: Try to calculate these probabilities from data provided by Fig. 2.17.

Exercises
2.1 What is Language Model (LM)? Discuss the roles and importance of language
model in NLP.
2.2 What is N-gram? Discuss and explain the importance of N-gram in NLP and
text analysis.
2.3 State the Chain Rule and explain how it works for the formulation of N-gram
probabilities. Use trigram as example to illustrate.
2.4 What is a Markov Chain? State and explain how it works for the formulation
of N-gram probabilities.
2.5 Use The Adventures of Sherlock Holmes as corpus, calculate N-gram proba-
bility for sentence “I don’t believe in that” with Markov Chain and evaluate all
related bigram probabilities.
References 41

2.6 Repeat Exercise 2.5 by using another famous literature Little Women by
Louisa May Alcott (1832–1888) (Alcott 2017) to calculate N-gram probabil-
ity of sentence “I don’t believe in that” and compare with results in 2.5. What
is (are) the finding(s)?
2.7 Use Shannon’s text generation scheme on The Adventures of Sherlock Holmes
as corpus, generate sample sentences like Fig. 2.14 using unigram, bigram,
trigram, and quadrigram text generation methods.
2.8 Repeat Exercise 2.7 using literature Little Women (Alcott 2017) to generate
corresponding sample sentences and compare with results in 2.7. What is (are)
the finding(s)?
2.9 What is Perplexity (PP) in N-gram model evaluation? Use The Adventures of
Sherlock Holmes as corpus with sample test set, evaluate PP values from uni-
gram to trigram and compare with Fig. 2.15. What is (are) the finding(s)?
2.10 Use Little Women (Alcott 2017) as corpus and some sample test set. Compare
the performance of Add-1 smoothing against Add-k (k = 0.5). Which one is
better? Why?
2.11 What is Backoff and Interpolation (B&I) method in N-gram smoothing?
Repeat 2.10 using B&I smoothing method with λ1 = 0.4, λ2 = 0.3 and λ3 = 0.3.
Compare the performance with results obtained in 2.10.
2.12 What is Good Turing (GT) Smoothing in N-gram smoothing? Repeat Exercise
2.10 using GT Smoothing and compare performance results obtained in 2.10
and 2.11. Which one is better? Why?

References

Alcott, L. M. (2017) Little Women (AmazonClassics Edition). AmazonClassics.

Bird, S., Klein, E., and Loper, E. (2009). Natural language processing with python. O’Reilly.
Chen, S. F. and J. Goodman. 1999. An empirical study of smoothing techniques for language mod-
eling. Computer Speech and Language, 13:359–394.
Ching, W. K., Huang, X., Ng, M. K. and Siu, T. K. (2013) Markov Chains: Models, Algorithms
and Applications. Springer.
Doyle, A. C. (2019) The Adventures of Sherlock Holmes (AmazonClassics Edition).
AmazonClassics.
Eisenstein, J. (2019) Introduction to Natural Language Processing (Adaptive Computation and
Machine Learning series). The MIT Press.
Gale, W. A. and Church, K. W. (1994) What is wrong with adding one? In N. Oostdijk and P. de
Haan (eds), Corpus-Based Research into Language, pp. 189–198. Rodopi.
Gale, W. A. and Sampson, G. (1995). Good-Turing frequency estimation without tears. Journal of
Quantitative Linguistics, 2(3), 217-237.
Google (2022) Google official site. http://google.com. Accessed 12 July 2022.
Gutenberg (2022) Project Gutenberg official site. https://www.gutenberg.org/. Accessed 13
July 2022.
Jurafsky, D., Marin, J., Kehler, A., Linden, K., Ward, N. (1999). Speech and Language Processing:
An Introduction to Natural Language Processing, Computational Linguistics and Speech
Recognition. Prentice Hall.
42 2 N-Gram Language Model

Liu, Z., Lin, Y. and Sun, M. (2020) Representation Learning for Natural Language Processing.
Springer.
Pustejovsky, J. and Stubbs, A. (2012) Natural Language Annotation for Machine Learning: A
Guide to Corpus-Building for Applications. O’Reilly Media.
Saichev, A. I., Malevergne, Y. and Sornette, D. (2010) Theory of Zipf’s Law and Beyond (Lecture
Notes in Economics and Mathematical Systems, 632). Springer.
Shakespeare, W. (2021) The Complete Works of Shakespeare (AmazonClassics Edition).
AmazonClassics.
Shannon, C. (1948). A Mathematical Theory of Communication. Bell System Technical Journal.
27 (3): 379–423.
Sidorov, G. (2019) Syntactic n-grams in Computational Linguistics. Springer.
Suyanto, S. (2020). Phonological similarity-based backoff smoothing to boost a bigram syllable
boundary detection. International Journal of Speech Technology, 23(1), 191-204.
Tuchong (2022) Computerized Axial Tomography Scanner (“Cat scan”). https://stock.tuchong.
com/image/detail?imageId=902001913134579722. Accessed 12 July 2022.
Chapter 3
Part-of-Speech (POS) Tagging

3.1 What Is Part-of-Speech (POS)?

Part-of-Speech (PoS or POS) is a category of words normally in lexical terms that

have similar grammatic behaviors or properties (Bender 2013; Jurafsky et al. 1999).
These are words assigned to the same POS exhibited in syntactic or functional
behaviors and roles in grammatic structure sentence, e.g. English verbs and nouns.
They have close morphological sometimes that can undergo inflection for similar
properties and semantic behaviors. To explore how POS works, it is necessary to
understand the concept of inflection.
Inflection can be considered as the process of word formation in which items are
added to the baseform of a word to convey grammatical meanings. The word inflec-
tion comes from Latin word inflectere which means to bend, e.g. (1) inflection -s of
cats signifies the noun is plural, (2) the same -s inflection of gets signifies the sub-
ject is a third-person singular, e.g. [3.1] He gets the book, and (3) inflection of -ed
often signifies past tense, e.g. arrive → arrived, close → closed etc. Thus, inflec-
tions are to express grammatical types such as persons, quantities, and tenses. There
are several types of POS to define inflection characteristics.

3.1.1 Nine Major POS in English Language

Every word in English sentences fall into nine major POS types. They are (1) adjec-
tives, (2) verbs, (3) pronouns, (4) conjunctions, (5) prepositions, (6) articles (deter-
miners), (7) adverbs, (8) nouns, and (9) interjections as shown in Fig. 3.1. Some
linguists include only first eight as major POS and leave interjections as an indi-
vidual category.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 43

Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_3
44 3 Part-of-Speech (POS) Tagging

Fig. 3.1 Major POS in English language

POS is important to study:

1. Word classes categorization and usage in linguistics
2. Grammars in English usage
3. Word functions categorization in NLP and
4. POS tagging

3.2 POS Tagging

3.2.1 What Is POS Tagging in Linguistics?

Part-of-Speech Tagging (Khanam 2022; Sree and Thottempudi 2011), also called
POS tagging, POST, or grammatical tagging is the operation of labelling a word in
a text, or corpus according to a particular POS based on definition and contexts in
linguistics. A simplified format is usually learnt by students to identify word types
such as adjectives, adverbs, nouns, verbs, etc. Grammars vary in foreign languages
leading to several POS tagging categorization.
3.2 POS Tagging 45

3.2.2 What Is POS Tagging in NLP?

Tagging is a kind of classification process that may be defined as automatic descrip-

tion assignment to words or tokens in NLP (Eisenstein 2019). They are called POS
tags or tags to represent one of the POS, semantic information in a s entence/utter-
ance. Some words may have different meanings and roles in POS, e.g. book can be
used as a noun or booking a table as a verb.
In NLP, POS tagging is the operation of converting a sentence/utterance to forms,
or list of words and list of tuples, where each tuple has a word or tag form to signify
noun, verb, adjective, pronoun, conjunction, and their subcategories. Figure 3.2
shows how tagging is applied to sample sentence/utterance: [3.2] She sells seashells
on the seashore.
Machine learning and rule-based models can produce POS tags in NLP. They
generally fall into (1) Rule-based POS tagging, (2) Stochastic POS tagging, and (3)
Hybrid POS tagging using advanced technology like Transformation-based tagging
(Jurafsky et al. 1999; Khanam 2022; Pustejovsky and Stubbs 2012). We will study
how they work with NLTK and spaCy technologies at workshops in Part II. First, let
us look at some realistic POS databanks.

3.2.3 POS Tags Used in the PENN Treebank Project

PENN Treebank is a frequently used POS tag databank provided by the PENN
Treebank corpus (Marcus et al. 1993). It is an English corpus marked by a TreeTagger
tool developed by Prof. Helmut Schmid at University of Stuttgart in Germany. It
classifies 9 major POS into subclasses that has a total of 45 POS tags with punctua-
tion and examples as shown in Fig. 3.3, its English Penn Treebank (PTB) corpus has
a comprehensive section of Wall Street Journal (WSJ) articles to be used on sequen-
tial labelling models’ evaluation as well as characters and word levels language
modelling.
A POS tagging table for sentence [3.3] David has purchased a new laptop from
Apple store in Fig. 3.4 showed that Apple is a proper noun because it can be differ-
entiated by capital letter A as a product brand name.

Fig. 3.2 POS example for utterance “She sells seashells on the seashore”
46 3 Part-of-Speech (POS) Tagging

Fig. 3.3 Penn Treebank POS Tags (with punctuation)

Fig. 3.4 Penn Treebank POS tags of sample sentence “David has purchased a new laptop from
Apple store”

3.2.4 Why Do We Care About POS in NLP?

POS is a fundamental concept to understand proper use of language, e.g. English.

Without this, we cannot differentiate usages or roles of different words in a sentence
whether it is a noun, verb, adjective, and determiners. The major concerns include:
1. Pronunciation often differs from the same word with different roles, e.g. [3.4]
Here are the students’ records vs. [3.5] The teacher records his lecture.
2. Prediction of the following word, e.g. (a) they should use will instead of shall, (b)
word after to is not past tense. It is natural in grammar rules as compared with
N-gram solely relied on counting words relationship.
3. Stemming is within a restricted tagset, e.g. comput for computer.
4. Syntactic parsing base and then meaning extraction, e.g. [3.6] Better get going
or you will be late.
5. Machine translation for the same word with different POS classes most likely
has different translation in other languages, e.g. translation from English
to French.
(E) book + N → (F) acheter + N (Buy a book → Achète un livre)
(E) book + VB → (F) réserver + VB (Book a room → Réserver une chambre)
3.2 POS Tagging 47

A proper POS tagging can provide correct translation between foreign languages.
Further, it is to stress different accents and avoid confusion of the same word (word
type) with different POS in a sentence/utterance. There are three types:
1. Noun vs. Verb confusion, e.g. ABstract (noun) vs. abstRACT (verb)
2. Adjective vs. Verb confusion, e.g. PERfect (adjective) vs. perFECT (verb)
3. Adjective vs. Noun confusion, e.g. miNUTE (adjective) vs. MInute (noun)
Figure 3.5 shows some common examples of English words from CELEX online
dictionary with different stress accents and meanings often occurred when dealing
with noisy channels to differentiate every word’s role in a sentence/utterance. They
can be solved by applying statistical probabilistic N-gram method or stochastic
techniques and corpora to facts analysis. Nevertheless, POS tagging is the initial
step for resolution.

Fig. 3.5 Common example of same English word with different stress accents
48 3 Part-of-Speech (POS) Tagging

3.3 Major Components in NLU

Natural Language Understanding (NLU) (Allen 1994; Mitkov 2005) is a critical

component in various NLP applications including text summarization, sentiment
analysis, information retrievals to Q&A Chatbot systems. It composes of five basic
modules: (1) morphology, (2) POS tagging, (3) syntax, (4) semantics, and (5) dis-
course integration as shown in Fig. 3.6.
Morphology is the understandings of shapes and patterns for every word of a
sentence/utterance.
POS tagging is key process to provide functions and categories of words.
Syntax is syntactic analysis to understand the syntactic role and usage of every
word or word pattern.
Semantics is an analysis to understand semantic meaning of a sentence/utterance
and its overall meaning.
Discourse integration is to understand the relationship between different sen-
tences and its contents.

3.3.1 Computational Linguistics and POS

Computational linguistics (CL) (Bender 2013; Clark et al. 2012; Mitkov 2005) can
be considered as the understanding of written or spoken language from computa-
tional and scientific perspective. It focuses on building artifacts to process and ana-
lyze language. Language is like a mirror of mind to reflect of what humans think. A
computational interpretation of language provides a new insight to how human
thinks and intelligence works.
As human language is natural and the most polytropic means of communication
either person-to-person or person-to-machine, linguistically enabled computer sys-
tems provide a new era of NLP applications. There are two major issues to address
in computational linguistics: (1) linguistic itself refers to facts about language and
(2) algorithmic refers to effective computational procedures dealing with these facts.

Fig. 3.6 Major

components in NLU
3.3 Major Components in NLU 49

The major goals of computational linguists include:

1. Construction of grammatical and semantic frameworks/models for languages
characterization,
2. Realization of learning models for the exploration of both structural and distri-
butional properties of language, and
3. Exploration of neuroscience and cognitive oriented computational models of
how language processing and learning works in our brains.
Thus, POS and POS tagging can be considered as the fundamental process in
computational linguistics to understand and model human languages.

3.3.2 POS and Semantic Meaning

The elementary level of language semantics (Goddard 1998) is to describe actual

meaning of word forms. For example, a noun may be a category of words for people,
locations, and things. Adjective may be the category of words for properties of nouns.
Consider: [3.7] green book in which green is an adjective while book is a noun.
In fact, the word book can have two meanings: (1) description of word book from
dictionary, (2) noun in a sentence which is an object. For the word green, it has an
in-depth interpretation of (1) an adjective to describe the book in green color and (2)
semantic meaning to describe the book in green.
Now consider: [3.8] book worm???
[3.9] This green is very smoothing???
Here the word book has the same spelling and pronunciation as [3.8] but it
becomes an adjective instead of a noun because of the semantic meaning of book
worm. In [3.9], green becomes a noun instead of an adjective because of semantic
meaning consideration in the whole sentence/utterance. So, POS of every word/
word pattern can be varied when considering the role in overall semantic meaning
of sentence/utterance.

3.3.3 Morphological and Syntactic Definition of POS

Based on morphological well-defined grammatic rules’ structure, when there is an

adjective that can fill in the blank, e.g. [3.10] It’s so _____, it can be difficult, expen-
sive, small etc. This rules’ structure gives shape to appropriate POS tag for descrip-
tion, e.g. when a noun is a word that can be labelled as plural means it can be
defined in either singular or plural form with s, or the other way round which is a
two-way process. Thus, when a tagger tags a word with s, it gives hints that the word
may contain s or a noun in plural, e.g. cat or cats.
Conversely, when there is a noun that can fill in the blank, e.g. [3.11] the _____
is so pretty, it can be decoration, house, painting etc. and conscious of not using a
proper noun, e.g. the Tesla.
50 3 Part-of-Speech (POS) Tagging

Consider the following situations, what is the POS for word purple:
[3.12] It’s so purple.
[3.13] Both purples should be okay for the room.
[3.14] The purple is a bit odd for the white carpet.
In [3.12] purple is an adjective. However, in [3.13] is a particular noun in plural
forms. Same notion for purple in [3.14] is also an indifferent noun to classify as a
group against uncountable objects in purple.

3.4 9 Key POS in English

There are nine key POS in English: (1) pronoun, (2) verb, (3) adjective, (4) interjec-
tion, (5) noun, (6) adverb, (7) conjunction, (8) preposition, and (9) article as shown
in Fig. 3.7. Some linguists consider interjections as separate POS category to
express strong feeling or emotion in a single word or a phrase, e.g. [3.15] Hooray!
It’s the last day of school. It is distinct compared with other POS.

Fig. 3.7 Nine major POS in English language with description

3.4 9 Key POS in English 51

3.4.1 English Word Classes

There are two types of English word classes: (1) closed-class and (2) open-class.
Both classes are important to understand proper sentences in different languages.
Closed-class words are also known as functional/grammar words. They are
closed since new words are seldom created in the class. For example, conjunctions,
determiners, pronouns, and prepositions are closed-class. Conversely, new items
are added to open classes regularly. As closed-class words are usually used with a
particular grammatical structure, it cannot be interpreted in isolation, e.g. [3.16] the
style of this painting, both the and this have no special meaning as compared with
painting that has a specific meaning in usual knowledge.
Open-class words are also known as lexical/content words. They are open since
the meaning of an open-class word to be found in dictionary so the meaning can be
interpreted in isolation. For example, noun, verb, adjective, and adverbs are open-
class that made up of the entire subclass of words. These connective words are
restrictive and used frequently to describe different scenarios or meanings about
spatial positions of two object nouns, e.g. [3.17] The cat sits by/under/above the
piano. Further, there are new types of open-class objects created from scratch or
combination of existing word according to contemporary times, e.g. fax, telex,
internet, iPhone, hub, bitcoin, metaverse, etc.

3.4.2 What Is a Preposition?

Preposition (PP) is POS with a word (group of words) being used before a noun,
pronoun, or a noun phrase to indicate direction, location, spatial relationships, time;
or to describe an object; or information to the recipient. There are approximately
80–100 prepositions in English to generate functional sentences/utterances.
This information can include where something takes place, e.g. [3.18] before
dinner, or general descriptive information e.g. [3.19] the girl with ponytail. The
target of preposition is the noun that followed the preposition. It is also the ending
point for each preposition phrase. For instance, [3.20] to the supermarket. The word
to is a preposition and supermarket is the target of preposition, and [3.21] over the
rainbow, the word over is the preposition and rainbow is the target of preposition.
A list of top 40 preposition from CELEX online dictionary (CELEX 2022) of
COBUILD 16-million-word corpus is shown in Fig. 3.8. It showed of, in, for, to and
with are the top five prepositions to correlate with ideas and additional information
of a sentence/utterance.
52 3 Part-of-Speech (POS) Tagging

Fig. 3.8 TOP 40 commonly used prepositions extracted from CELEX online dictionary

3.4.3 What Is a Conjunction?

Conjunction (CONJ or CNJ) is POS to connect words, clauses, or phrases that are
known as conjuncts. This definition may sometime overlap with other POS so that
the constitute of a conjunction must be defined for each foreign language. For
instance, a word in English may have several senses and meanings. It can be consid-
ered as either a conjunction or preposition highly dependable on syntax of the sen-
tence/utterance, e.g. after is a preposition in [3.22] Jane left after the show but is a
conjunction in [3.23] Jane left after she finished her homework.
Co-ordinating conjunction allows joining words, clauses, or phrases of equal
grammatic rank in a sentence/utterance. Common coordinating conjunctions are
and, but, for, nor or yet which include logical meaning at times.
3.4 9 Key POS in English 53

Subordinating conjunctions join independent and dependent clauses. A subordi-

nating conjunction can present a causation relationship, or some kinds of relation-
ship between different words, clauses, or phrases. Common subordinating
conjunctions are as, although, because, since, though, while, and whereas. In many
situations, a conjunction is a non-inflected grammatical item, it may or may not link
up the items being conjoined, e.g. [3.24] the book is so difficult that is hard for chil-
dren to read, that is to describe about the book to connect two ideas and [3.25] this
painting is very beautiful but is expensive. In this case but is to explain an initial idea
to correlate with second idea. A list of top 50 commonly used coordinating and
subordinating conjunctions from CELEX online dictionary is shown in Fig. 3.9. It
showed and, that, or, and as are used frequently to convey more than one concept at
the same time or further explanation.

3.4.4 What Is a Pronoun?

Pronoun (PRN or PN) is POS that can be considered as a word (phrase) to serve as
substitution for a noun or noun phrase. It is also called the pronoun’s antecedent.
Pronouns are usually appeared as short words to replace a noun (noun phrase) for
the construction of a sentence/utterance. Commonly used pronouns are I, he, she,
you, me, we, us, this, them, that.
A pronoun can be served as a subject, direct (indirect) object, object of preposi-
tion and more to substitute any person, location, animal, or thing. It can replace a
person’s name in a sentence/utterance, e.g. [3.26] Jack is sick today, he cannot
attend the evening seminar. Pronoun is also a powerful tool to simplify the contents
of a dialogue and conversation by replacing with simple token. A list of top 50 com-
monly used pronouns extracted from CELEX online dictionary is shown in Fig. 3.10.
It showed it, I, he, you, and his are used frequently.
The truth is, without pronouns, nouns become repetitive and cumbersome in
speech and writing. However, pronoun may cause ambiguity, e.g. [3.27] Jack
blamed Ivan for losing the car key, he felt sorry for that. He normally refers to the
first person which is Jack but make sense in pragmatic meaning for Ivan to feel sorry
because Jack blamed him for the loss.

3.4.5 What Is a Verb?

Verb (VB) can be considered as a word syntax to conduct an action, process, occur-
rence, or state-of-being. In general, verbs are inflected to encode tense, aspect,
mood, and voice in many languages, but are interchangeable with nouns of a word
in some foreign languages. In English, a verb may also conform with gender, per-
son, or numbers of arguments such as its subject or object.
54 3 Part-of-Speech (POS) Tagging

Fig. 3.9 TOP 50 commonly used conjunctions extracted from CELEX online dictionary

English verbs have tenses consideration: (1) present tense to notify that an action
is being carried out, (2) past tense to notify that an action has been completed, (3)
future tense to notify that an action to be happened in future, and (4) future perfect
tense to notify an action will be completed in future.
A modal verb is a category of verb that contextually indicates a modality such as
ability, advice, capacity, likelihood, order, obligation, permission, request, or
3.4 9 Key POS in English 55

Fig. 3.10 TOP 50 commonly used pronouns extracted from CELEX online dictionary

suggestion. It is usually accompanied by the base (infinitive form) of another word

with semantic contents. Common modal verbs are can, could, may, might, shall,
should, will, would, and must. A list of top 25 commonly used verbs from CELEX
online dictionary is shown in Fig. 3.11. It showed can, will, may, would, and should
are used frequently. They also express significance in subsequent verb, e.g. verb
following can and will must use present tense, not past tense.
56 3 Part-of-Speech (POS) Tagging

Rank VB Frwq. Rank VB Frwq.

1 can 70,930 14 won’t 3,100

2 will 69,206 15 ’d 2,299
3 may 25,802 16 ought 1,845
4 would 18,448 17 will 862
5 should 17,760 18 shouldn’t 858
6 must 16,520 19 mustn’t 332
7 need 9,955 20 ’II 175
8 can’t 6,375 21 needn’t 148
9 have 6,320 22 mightn’t 68
10 might 5,580 23 oughtn’t 44
11 couldn’t 4,262 24 mayn’t 3
12 shall 4,118 25 dare 3
13 wouldn’t 3,548

Fig. 3.11 TOP 25 commonly used modal verbs extracted from CELEX online dictionary

3.5 Different Types of POS Tagset

3.5.1 What Is Tagset?

There are nine POS in English, pronoun, verb, adjective, interjection, noun, adverb,
conjunction, preposition, and article learnt as students but there are clearly more
subcategories that can be further divided. For example, in nouns, the plural, posses-
sive, and singular forms can be distinguished and further classified.
A Tagset is a batch of POS tags (POS tags or POST) to indicate the part of speech
and sometimes other grammatical categories such as case, tense for the classifica-
tion of each word in a sentence/utterance.
Brown Corpus Tagset (Brown 2022), PENN Treebank Tagset (Treebank 2022),
and CLAWS (CLAWS7 2022) are commonly used. Brown Corpus was the first well-
organized corpus of English for NLP analysis developed by Profs Emeritus Henry
Kučera (1925–2010) and W. Nelson Francis (1910–2002) at Brown University,
USA in mid-1960s. It consists of over 1 million of English words which extracted
from over 500 samples of randomly chosen publications. Each sample consists of
over 2000 words with 87 tags defined (Brown 2022).
The English PENN Treebank Tagset originated by English corpora is annotated
with TreeTagger tool. PENN Treebank Tagset is developed by Prof. Helmud Schmid
in the University of Stuttgart, Germany. It consists of 45 distinct tags (Abeillé 2003;
Treebank 2022).
3.5 Different Types of POS Tagset 57

English CLAWS part-of-speech Tagset version 7, also called C7 Tagset is avail-

able in English corpora annotated with tools using CLAWS (Constituent Likelihood
Automatic Word-tagging System). C7 Tagset is developed by the University Centre
for Computer Corpus Research on Language at Lancaster University. It bases on
Hidden Markov model to determine the likelihood of sentences, sequences of words
in anticipating each POS label. It consists of 146 distinct tags (CLAWS7 2022).

3.5.2 Ambiguous in POS Tags

It may wonder the necessity of tagset databank against dictionary to check out
POS. A reason is that there are ambiguities in POS tags for many words:
1. Noun-verb ambiguity
For example: record: [3.28] records the lecture vs. [3.29] play CD records.
2. Adjective-verb ambiguity
For example: perfect: [3.30] a perfect plan vs. [3.31] Jack perfects the
invention.
3. Adjective-noun ambiguity
For example: complex: [3.32] a complex case vs. [3.33] a shopping complex.

Figure 3.12 shows an ambiguous analysis of words in Brown corpus (DeRose

1988). One tag refers to a word tagged with single POS type, 2–7 tags refer to a
word tagged with several POS types. For example, a 3 POS ambiguous tag for
green: (a) [3.34] colour green (noun), (b) [3.35] a green apple (adjective) and (c)
[3.36] the roof was greening with leaves (verb). A 7 POS ambiguous tag for still: (a)
[3.37] the still status (adjective), (b) [3.38] the still of the night (noun), (c) [3.39] it
was still snowing (adverb), and (d) [3.40] Her quiet words stilled the animal (verb)

Fig. 3.12 Ambiguous

analysis of words in Brown Unambiguous (1 tag) 35,340
corpus
2 tags 3,760

3 tags 264

4 tags 61

5 tags 12

6 tags 2

7 tags 1

Ambiguous (2-7 tags) 4,100

Ambiguous % 10.40%
58 3 Part-of-Speech (POS) Tagging

(Note: As an exercise, find out the other three POS tag usages for still). Overall
speaking, there is a total of 10.4% ambiguous word types often used in language in
which over 40% ambiguous words are easy to disambiguate.

3.5.3 POS Tagging Using Knowledge

There are four methods to acquire knowledge from POS tagging: (1) dictionary, (2)
morphological rules, (3) N-gram frequencies, and (4) structural relationships
combination.
Dictionary is the basic method for tag usage, but it may not be fully reliable
because there are ambiguous words meaning that the same word can have more than
single POS tagging in diverse scenarios.
Morphological rules are to identify well-known words shapes and patterns, e.g.
the inflection -ed for past tense, verb + -ing for continuous form, -tion for noun
description, -ly for adjective, and capitalization such as New York for proper noun.
N-gram frequencies checking, also called next word prediction, e.g. grammatic
pattern to ___. When there is a to, if the next word is a verb, it must be in present
and not past tense. If it is a determiner, the next word must be a noun.
Structural relationships combination method means to combine several methods
to acquire tag information. e.g. [3.41] She barely heard the foghorns knelling her
demise vs. [3.42] The hunter’s horn sounded the final knell. If there is no under-
standing on what knell means, there is an -ing pattern to indicate that is a verb in
continuous tense, and final is an adjective description to indicate that knell is
likely a noun.

3.6 Approaches for POS Tagging

There are three basic approaches of POS Tagging: (1) Rule-based, (2) Stochastic-
based, and (3) Hybrid Tagging.

3.6.1 Rule-Based Approach POS Tagging

Rule-based is classical approach in linguistic (Sree and Thottempudi 2011). The

grammars knowledge learnt in primary schools are in fact grammatic rules which
means that rule-based approach is the transfer of linguistic rule base usage into POS
tagging.
It is a two stages process: (1) dictionary consists of all possible POS tags for
basic concepts of words as abovementioned, (2) words with more than single tag
ambiguity applied handwritten or grammatic rules to assign the correct tag(s)
3.6 Approaches for POS Tagging 59

according to surrounding words. The obtained rule sets directly affect tagging
results accuracy. The lexicon is used initially for basic segmentation and tagging of
the corpus, listing all possible lexical properties of the object, and combine rule-
base with contextual information to disambiguate and retain the only suitable lexi-
cal properties.
The rule generation can be achieved by (1) hand creation and (2) training from a
corpus with machine learning. The advantages of hand creation are that it is more
sensible and explainable to humans, but manual construction of rules is usually
labor intensive. Also, if rules are described with too many details, the coverage of
rules will be greatly reduced and difficult to adjust according to actual situation.
Conversely, if rules are not based on contexts but rather on the lexical nature of
rules, ambiguity may arise, i.e. If the preceding of a word is an article, then the word
must be a noun.
For example, consider: [3.43] a book. a is an article as per possible tags that can
assign directly, but a book can either be a noun or a verb. If consider a book, a is an
article and follow rules above, book should be a noun because article is often fol-
lowed by a noun, so a tag of noun is assigned to book. Word structures are often
complex leading to more ambiguities and rules are required for differentiation.

3.6.2 Example of Rule-Based POS Tagging

Step 1: Assign each word with a list of possible tags based on a dictionary.
Step 2: Work out unknown and ambiguous words with two approaches: (1) rules
that specify what (1) to do; and (2) not to do.
Figure 3.13 shows a sample adverbial that rule (Jurafsky et al. 1999):
It showed that:
–– The first two statements of this rule verify the word that is directly precedes a
sentence/utterance’s final adjective, adverb, or quantifier.
–– For all other cases, the adverb reading is eliminated.
–– The last clause eliminates cases which are preceded by verbs like consider or
believe which can take a noun and an adjective.

Fig. 3.13 Sample rule for adverbial “that” rule

60 3 Part-of-Speech (POS) Tagging

–– The logic behind is to avoid tagging the following instance of that as an adverb
such as [3.44] It isn’t that odd.
–– The other rule is used to verify if the previous word is a verb which expects a
complement (like think or hope), and if that is followed by the beginning of a
noun phrase, and a finite verb such as [3.45] I consider that a win or more com-
plex structure such as [3.46] I hope that she is confident.
Stochastic-based approach (Dermatas and Kokkinakis 1995) is different from
rule-based approach in which it is a supervised model using frequencies or proba-
bilities of tags appeared in the training corpus to assign a tag to a new word. This
tagging method depends on tag occurrence statistics, i.e. probability of the tags.
Stochastic taggers are further categorized into two parts: (1) word frequency and (2)
tag sequence frequency to determine a tag.
Word frequency is to identify the tag that has a notable occurrence of the word,
e.g. based on the counting from a corpus, the word list occurs ten times in which six
times as noun and four times as verb, and the word cloud will always be assigned as
noun since it has a notable occurrence in the training corpus. Hence, a word fre-
quency approach is not very reliable in certain scenario.
Tag sequence frequency, also called N-gram approach is assigned the best tag to
a word evaluated by the probability of N previous words tags. Although it provides
better outcomes than word frequency approach, it may be unable to provide accu-
rate outcomes for some rare words and phrases.
Stochastic POS tag model allows features to be non-independent and allows for
the addition of various granularities features. Hidden Markov Model (HMM) Tagger
is a common stochastic-based approach, its Maximum Entropy Markov Model
(MEMM) (Huang and Zhang 2009) is a stochastic POS tagging model that deter-
mines an exponential algorithm for each state as the conditional probability of the
next state given the current state, which has the advantages of a stochastic POS tag-
ging model. However, it also suffers from label bias problems. Unlike MEMM
model, the Conditional Random Field (CRF) model uses only one model as the joint
probability of the entire label sequence given the observations sequence. Lafferty
et al. (2001) verified that this model can effectively solve the tagging bias problems.

3.6.3 Example of Stochastic-Based POS Tagging

Let us use HMM Tagger as example. The rationale of HMM tagger is applying
N-gram frequencies to determine the best tag for a given word, like the same con-
cept to investigate N-gram with Markov Chain. Mathematically, all is needed to
maximize the conditional probability. The conditional probability wi is tag ti in the
context given wi by

P ( wi |ti in context ) P ( ti in context )

P ( ti in context|wi ) = (3.1)
P ( wi )
3.6 Approaches for POS Tagging 61

In other words, given a sentence/utterance or word sequence, HMM taggers

select tag sequence that maximizes the following formula given by

P ( word|tag ) ∗ P ( tag|previous n tags ) (3.2)

For bigram-HMM tagger, select tag ti for wi, that is most probable given the pre-
vious tag ti−1, and the current word wi in this equation by

ti = argmax
j P ( t j |,ti −1 |,wi ) (3.3)

By simplifying Markov assumptions, the previous equation is applied to give

basic HMM equation for a single tag as follows:

ti = argmax
j P ( t j |ti −1 ) P ( wi |t j ) (3.4)

3.6.4 Hybrid Approach for POS Tagging Using Brill Taggers

Hybrid approach is the integration of rule-based and stochastic with high-level

methods including neural networks such as LSTM and other machine learning
related methods often applied in NLP nowadays. Let us study an important hybrid
approach for POS Tagging—Transformation-based tagging, also called Brill
Taggers invented by Dr. Eric Brill in 1995 (Brill 1995). It is a direct Transformation-
Based Learning (TBL) implementation based on the integration of these two
approaches.

3.6.4.1 What Is Transformation-Based Learning?

There are five steps in TBL by comparison to analogue on oil painting with layering-
and-refinement approach.
1. Start with background theme such as sky or household background.
2. Paint background first, e.g. if sky is the background scheme, paint clouds over it.
3. Paint the main theme or object over the background, e.g. landscape, birds.
4. Refine the main theme or object over background to make it more precise, e.g.
paint landscape, add trees and animals layer-by-layer.
5. Further refine objects or main theme until perfect, e.g. apply layering process or
refinement for every single tree and animal (Fig. 3.14).
62 3 Part-of-Speech (POS) Tagging

Fig. 3.14 Oil painting analog to Brill Tagger transformation technique (Tuchong 2022)

3.6.4.2 Hybrid POS Tagging: Brill Tagger

Brill Tagger is a type of hybrid TBL. Hybrid refers to integrate rule-based and
stochastic-based methods in a Brill’s algorithm.
Rule 1: Label each word of the tag that is mostly likely given on contextual infor-
mation, e.g.

Race : P ( NN | race ) = 0.98; P ( VB | race ) = 0.02

Rule 2: Apply transformation rule based on the context established.

Example:
Race: change NN to VB when the previous tag is TO.
[3.47] Secretariat is expected to race tomorrow. -- change tag race from NN to VB.
[3.48] The race is already over. -- no change, race remains as NN.
For [3.47] race has a higher probability of a noun, it will be treated as such by
applying rule 1 initially. However, when there is a verb prior to, it should apply rule
2 to change into a verb instead of a noun according to grammatic rules.
For [3.48] race again has a high probability of a noun but due to grammatic rule
is invalid, it remains as a noun. Thus, TBL is often applied to identify stochastic
probabilities of tag frequencies for initial guesswork followed by grammatic rules
for refinement.
3.7 Taggers Evaluations 63

Fig. 3.15 Sample rules used in Brill’s TBL scheme

3.6.4.3 Learning Brill Tagger Transformations

There are three stages to learn Brill tagger transformations:

1. Label every word with its best tag with stochastic method,
2. Examine every possible transformation to select one with the most improved
tagging, and
3. Retag data according to tagging rules.
These three stages are repetitive until a stopping criterion with no more rules to
apply. TBL output is an ordered list of transformations which constitute a POS tag-
ging procedure to a new corpus. Sample rules of a Brill’s TBL model are shown in
Fig. 3.15 (Jurafsky et al. 1999).
There are many NLP applications applying Brill tagger’s model because TBL is
a well integration of rule-based model to provide detailed refinement, and stochastic
model to provide an efficient tagging solution. Further, it would be effortless to
implement Brill’s models either in world or knowledge domains, e.g. medical
knowledge domain that may have specific rules or terminologies for a corpus.

3.7 Taggers Evaluations

There are several considerations when POS taggers have implemented (Padro and
Marquez 1998):
1. Evaluate algorithm adequacy
2. Identify errors origin
3. Repair and solve
A confusion matrix suggests that current taggers face with major problems:
1. Noun-single or mass vs. proper noun-singular vs. adjective (NN vs. NNP vs. JJ).
These are hard to distinguish as proper noun is crucial for information extrac-
tion, retrieval, and machine translation for different languages have diverse tag-
ging algorithms or classification schemes.
64 3 Part-of-Speech (POS) Tagging

IN JJ NN NNP RB VBD VBN

IN 0.18 0.56
JJ 0.32 4.35 3.21 2.25 0.31 2.54
NN 7.56 0.35
NNP 0.31 3.12 5.23 0.15
RB 2.45 3.21 0.43
VBD 0.56 0.52 4.31
VBN 3.21 2.12

Fig. 3.16 Confusion matrix from HMM of The Adventures of Sherlock Holmes

2. Adverb vs. adverb vs. preposition-sub-conjunction (RP vs. RB vs. IN). All of
these can appear in satellite sequences following a verb immediately.
3. Verb-base form vs. verb-past participle vs. adjective (VB vs. VBN vs. JJ). They
are crucial to distinguish for partial parsing, i.e. participles to identify passives,
and to label the edges of noun phrases correctly.
The confusion matrix from HMM error analysis of The Adventures of Sherlock
Holmes (Doyle 2019) is shown in Fig. 3.16. For example, mis-tagging of (1) NN by
JJ is 7.56%, (2) NNP by NN is 5.23%, and (3) JJ by NN is 4.35%. Hence, mistaking
NN by JJ is occurred more often than JJ by NN in English texts but it may vary in
other foreign languages.

3.7.1 How Good Is an POS Tagging Algorithm?

A satisfied POS tagging algorithm depends on the maximum performance it can

achieve. It must be realistic, naturally the higher the better but there are limitations.
For instance, a POS Tagging system with over 90% accuracy should be considered
satisfactory. But how to define satisfactory? E.g. (1) a speech dialogue system 97%
assigns a correct meaning to user’s input is satisfactory because ambiguity often
occurred in noisy backgrounds with incorrect pronunciations; or (2) an OCR system
determines 97% words correctly is satisfactory. Hence, it depends on scenario, envi-
ronment, complexity, domain problems, and applications to achieve.

Exercises
3.1 What is Part-of-Speech (POS)? How it is critical for NLP systems/applica-
tions implementation?
3.2 State and explain NINE basic types of POS in English Language. For each
POS type, give an example for illustration.
3.3 What is POS Tagging in NLP? How is it important to NLP systems/applica-
tions implementation? Give two examples of NLP systems/applications for
illustration.
References 65

3.4 State and explain THREE types of POS Tagging methods in NLP.
3.5 What is PENN Treebank tagset? Perform POS Tagging for the following sen-
tences/utterance using PENN Treebank tagset.
[3.47] POS tagging is a very interesting topic.
[3.48] It is not difficult to learn PENN Treebank tagset provided that we
have sufficient examples.
3.6 What is Natural Language Understanding (NLU)? State and explain FIVE
major components of NLU in NLP.
3.7 Why semantic meaning is an important factor in POS tagging? Give two
examples to support your answer.
3.8 What is ambiguous in POS tags? Give two examples word with three and four
ambiguous of POS tags.
3.9 What is Rule-based approach in POS Tagging? Give an example of POS tag-
ging rule to illustrate how it works.
3.10 What is Stochastic-based approach in POS Tagging? Give a live example to
explain how word frequency and tag sequences frequency are applied for POS
tagging.
3.11 State and explain Transformation-based Learning (TBL). Give a live example
to support your answer.

References

Abeillé, A. (ed) (2003) Treebanks: Building and Using Parsed Corpora (Text, Speech and Language
Technology Book 20). Springer.
Allen, J. (1994) Natural Language Understanding (2nd edition). Pearson
Bender, E. M. (2013) Linguistic Fundamentals for Natural Language Processing: 100 Essentials
from Morphology and Syntax (Synthesis Lectures on Human Language Technologies). Morgan
& Claypool Publishers
Brill, E. (1995). Transformation-based error-driven learning and natural language processing: A
case study in part-of-speech tagging. Computational Linguistics, 21(4), 543–566.
Brown (2022) Brown corpus tagset. https://web.archive.org/web/20080706074336/http://www.
scs.leeds.ac.uk/ccalas/tagsets/brown.html. Accessed 15 July 2022.
CELEX (2022) CELEX corpus official site. https://catalog.ldc.upenn.edu/LDC96L14. Accessed
15 July 2022.
Clark, A., Fox, C. and Lappin, S. (2012) The Handbook of Computational Linguistics and Natural
Language Processing. Wiley-Blackwell.
CLAWS7 (2022) UCREL CLAWS7 Tagset. https://ucrel.lancs.ac.uk/claws7tags.html. Accessed
15 July 2022.
DeRose, S. J. (1988). Grammatical category disambiguation by statistical optimization.
Computational Linguistics, 14, 31–39.
Dermatas, E. and Kokkinakis, G. (1995). Automatic stochastic tagging of natural language texts.
Computational Linguistics, 21(2), 137–164.
Doyle, A. C. (2019) The Adventures of Sherlock Holmes (AmazonClassics Edition).
AmazonClassics.
Eisenstein, J. (2019) Introduction to Natural Language Processing (Adaptive Computation and
Machine Learning series). The MIT Press.
66 3 Part-of-Speech (POS) Tagging

Goddard, C. (1998) Semantic Analysis: A Practical Introduction (Oxford Textbooks in Linguistics).

Oxford University Press.
Huang H, Zhang X. (2009) Part-of-speech tagger based on maximum entropy model. 2009 2nd
IEEE International Conference on Computer Science and Information Technology. IEEE;
pp 26-29. https://doi.org/10.1109/ICCSIT.2009.5234787.
Jurafsky, D., Marin, J., Kehler, A., Linden, K., Ward, N. (1999). Speech and Language Processing:
An Introduction to Natural Language Processing, Computational Linguistics and Speech
Recognition. Prentice Hall.
Khanam, H. M. (2022) Natural Language Processing Applications: Part of Speech Tagging.
Scholars’ Press.
Lafferty, J., McCallum, A. and Pereira, F. (2001). Conditional random fields: Probabilistic mod-
els for segmenting and labeling sequence data. Proc. 18th International Conf. on Machine
Learning. Morgan Kaufmann. pp. 282–289.
Marcus, M., Santorini, B. and Marcinkiewicz, M. A. (1993). Building a large, annotated corpus of
English: The Penn Treebank. In Computational Linguistics, volume 19, number 2, pp. 313–330.
Mitkov, R. (2005) The Oxford Handbook of Computational Linguistics. Oxford University Press.
Padro, L. and Marquez, L. (1998). On the evaluation and comparison of taggers: The effect of noise
in testing corpora. Cornell University Library, arXiv.org. https://arxiv.org/abs/cs/9809112.
Pustejovsky, J. and Stubbs, A. (2012) Natural Language Annotation for Machine Learning: A
Guide to Corpus-Building for Applications. O’Reilly Media.
Sree, R. and Thottempudi, S. G. (2011) Parts-of-Speech Tagging: A hybrid approach with rule
based and machine learning techniques. LAP Lambert Academic Publishing.
Treebank (2022) Penn Treebank Tagset. https://www.ling.upenn.edu/courses/Fall_2003/ling001/
penn_treebank_pos.html. Accessed 15 July 2022.
Tuchong (2022) Oil painting analog to Brill Tagger transformation technique. https://stock.
tuchong.com/image/detail?imageId=965034062997356555. Accessed 15 July 2022.
Chapter 4
Syntax and Parsing

4.1 Introduction and Motivation

This chapter will explore syntax analysis and introduce different types of constitu-
ents in English language followed by the main concept of context-free grammar
(CFG) and CFG parsing. We will also study different major parsing techniques,
including lexical and probabilistic parsing with live examples for illustration.
Linguistic and grammatical aspects are addressed in NLP to identify patterns
that govern the creation of language sentences like English. They include the inves-
tigation of Part-of-Speech (POS) mentioned in Chap. 3 and grammatic rules to cre-
ate sentences/utterances with syntactic rules. These syntactic rules relied on
effective computational procedures such as rule-based, stochastic-based, techniques
and machine learning to deal with language syntax (Bender 2013; Gorrell 2006).
Another motivation is to study syntax and parsing methods or algorithms so that
they can fall into an automatic system like forming a parser to understand syntactic
structure during the construction process. Figure 4.1 illustrates the relationship
between grammar, syntax, and corresponding parse tree of a sentence/utterance
with four tokens: Tom pushed the car. Syntax level analysis is to analyze structure
and the relationship between tokens to create a parse tree accordingly.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 67

Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_4
68 4 Syntax and Parsing

Fig. 4.1 Grammar, syntax, and parse tree

4.2 Syntax Analysis

4.2.1 What Is Syntax

Syntax can be considered as the rules that manage group of words are combined to
create phrases, clauses, and sentences/utterances in linguistics (Bender 2013; Brown
and Miller 2020). The term syntax come from Greek word σύνταξη meaning
arrange words togetherness. Syntax provides a proper and organized way to form
meaningful phrase or sentence. It is a vital tool in technical writing and sentence
construction.
The fact is all native speakers learn proper syntax of their mother languages by
nature. The complexity sentences by a writer or speaker create formal or informal
level, or phrase and clauses presentation to audiences.
Syntax can be considered as the proper ordering of word tokens in written/spo-
ken sentences/utterances so that computer systems can understand how to process
these tokens without knowing the exact meaning from NLP perspective.

4.2.2 Syntactic Rules

POS in English often follows patterns orders in sentences and clauses (Khanam
2022; Jurafsky et al. 1999). For instance, compound sentences are combined by
conjunctions like and, or, with or multiple adjectives transformation of the same
noun based on order(s) according to their classes, e.g. [4.1] The big black dog.
Syntactic rules also described to assist language parts make sense. For example,
sentences/utterances in English usually begin with a subject followed by a predicate
4.2 Syntax Analysis 69

(i.e. a verb in the simplest form) and an object or a complement to show what’s
acted upon, e.g. [4.2] Jack chased the dog is a typical sentence with a subject-verb-
object pattern of syntactic rule in English. However, [4.3] Jack quickly chased the
dog at lush green field contains adverbs and adjectives to take their places in front
of the sentence transformation (quickly chased, lush green field) with informative
description.

4.2.3 Common Syntactic Patterns

There are seven common syntactic patterns:

1. Subject → Verb
For example: [4.4] The cat meowed.
This syntactic pattern is a standardized pattern containing only minimum sub-
ject and verb requirements. The topic always comes first in usual situation.
2. Subject → Verb → Direct Object
For example: [4.5] The cat plays the ball.
When verb is transitive with a direct object, the direct object usually goes
after the verb in this syntactic pattern.
3. Subject → Verb → Subject Complement
For example: [4.6] The cat is playful.
Subject complement usually goes after the verb in this syntactic pattern.
Linking verbs such as be, is, like or seem are usually used with subject
complement.
4. Subject → Verb → Adverbial Complement
For example: [4.7] The cat paced slowly.
Adverbial complement usually goes after the verb like previous case (3).
5. Subject → Verb → Indirect Object → Direct Object
For example: [4.8] The cat gave me the ball.
This syntactic pattern contains direct and indirect objects. Direct object usu-
ally goes after indirect object and the indirect object usually goes right after verb.
For example, [4.8] can be rephrased as [4.9] The cat gave the ball to me.
6. Subject → Verb → Direct Object → Direct Complement
For example: [4.10] The cat made the ball dirty.
Object complement usually goes after direct object and the direct object is
usually followed by verb in this syntactic pattern.
7. Subject → Verb → Direct Object → Adverbial Complement
For example: [4.11] The cat perked its ears up.
up is the adverbial complement to describe how the cat behaves. Direct com-
plement is replaced by adverbial complement like previous case (6).
70 4 Syntax and Parsing

Fig. 4.2 Major

components in NLU

4.2.4 Importance of Syntax and Parsing in NLP

There are five major components in NLU as recalled in Fig. 4.2, syntax and parsing
components have central roles to link up natural language with its syntactic struc-
ture prior understanding its semantic or embedded (discourse pragmatic) meanings
in NLP (Allen 1994; Eisenstein 2019). It is the first tier to analyze whether sen-
tences/utterances make sense or not. In other words, if a sentence/utterance or dia-
log has syntactic error, e.g. Jack buys (buys what?), it will not make sense, let alone
to study semantic meaning.
Syntax and parsing are sole process beneficial to:
1. Check grammar by word-processing applications such as Microsoft Word.
2. Speech recognizer at human speech real-time syntactic level in noisy environment.
It has significance in high-level NLP applications such as machine translation
and Q&A chatbot systems.

4.3 Types of Constituents in Sentences

4.3.1 What Is Constituent?

A constituent is considered as the linguistic component of a language (Bender 2013;

Brown and Miller 2020). For example, words or phrases that combine into a sen-
tence/utterance are constituents. It can be a word, morpheme, clause, or phrase.
Parsing is a kind of sentence analysis to identify the subject or predicate with differ-
ent POS and parse sentences/utterances into corresponding constituents, e.g. There
are several ways to describe the cat in Fig. 4.3.
[4.12] The milky cat with long tail (as a constituent of a clause) is meowing.
4.3 Types of Constituents in Sentences 71

Fig. 4.3 The milky cat

with long tail is meowing
(Tuchong 2022)

A single pronoun it to replace the identified constituent. This makes sense as it

described the milky cat with long tail is meowing, it is meowing, or a name to the
cat Coco, like:
[4.13] Coco is meowing or
[4.14] Coco with long tail is meowing.
which means a word or phrase form can be replaced by a simple token, or complex
constituents with additional description:
[4.15] The milky cat with long tail is meowing in the late afternoon.
[4.16] The milky cat with long tail is meowing the late afternoon while Jack is asleep.
Constituents can also be a time unit with usage variations instead of an object
unit in noun phrase (NP), they are syntactically acceptable, but some are not:
[4.17] Jane wants to go to Greece late this winter.
[4.18] Late this winter Jane wants to go to Greece.
[4.19] Jane wants late this winter to go to Greece.
It makes sense wherever the location of late this winter as it is a constituent
describing a particular time in syntax, but there are syntactic errors as below:
[4.20] Late Jane wants to go to Greece this winter.
– Cannot separate time unit into two parts.
[4.21] Jane wants late to go to Greece this winter.
– Senseless meaning
[4.22] The late this winter Jane wants to Greece.
– Incorrect syntactic pattern
72 4 Syntax and Parsing

4.3.2 Kinds of Constituents

Constituents exist in every sentence, phrase, and clause. Every sentence, in other
words, is constructed by the combination of all these components into meaningful
sentences/utterances (Bender 2013; Brown and Miller 2020). The commonly used
constituent types include: (1) noun phrase, (2) verb phrase, and 3) preposition
phrase. For instance:
[4.23] My cat Coco scratches the UPS courier on the table.
– These constituents are made up of noun phrase (my cat Coco), predicate and
verb phrase (scratches the UPS courier on the table).

4.3.3 Noun-Phrase (NP)

An NP consists of a noun and its modifiers. Modifiers that go before the noun such
as adjectives, articles, participles, possessive nouns, or possessive pronouns; or go
after the noun such as adjective clauses, participle phrases, or prepositional phrases.
For example, in [4.23] My cat Coco is a NP consists of determiner (DT) My + noun
(NN) cat + proper noun (NNP) Coco.
There are other NPs appear as objects of prepositions or objects of verbs:
[4.24] The milky cat with long tail is meowing.
[4.25] Very few cats wore a collar.
[4.26] The long tail is brought to room.
[4.27] Many places hear meowing.
[4.28] A cat with a long tail and a collar is meowing.
[4.29] Jane saw so many cats in the room.

4.3.4 Verb-Phrase (VP)

A VerbPhrase (VP) consists of a main verb followed by other linking verbs or modi-
fiers that act as a sentence’s verb. Modifiers in VP are words that can change, adapt,
limit, expand, or help to define a certain word in a sentence. They are usually auxil-
iary verbs such as is, has, am, and are that work with the main verb. A main verb in
VP holds information about an event or activity being referred to, and auxiliary
verbs add meaning by relating to time or aspect of the phrase.
There are nine common VP types:
1. Singular main verb
[4.30] Jack catches a deer.
2. Auxiliary verb (to be) + main verb -ing form
4.3 Types of Constituents in Sentences 73

When the main verb is used in -ing form, e.g. walking, talking, it expresses a
continuous aspect to show whether is in the past, present, or future.
[4.31] Jack is singing.
3. Auxiliary verb (have) + main verb (past participle form)
When verb to have (i.e. have, has, had) and the main verb in past partici-
ple form.
[4.32] Jack has broken the vase.
4. Modal verb + main verb
When a modal verb is combinedly used with a main verb, it includes things
such as possibility, probability, ability, permission, and obligation. Examples of
modal words include must, shall, will, should, would, can, could, may, and might.
[4.33] Jack will leave.
5. Auxiliary verb (have + been) + main verb (-ing form)
When both continuous and perfect aspects are expressed, the continuous
aspect comes from -ing verb and the perfect aspect comes from auxiliary verb
have been.
[4.34] Jack has been washing the car.
6. Auxiliary verb (to be) + main verb (past participle form)
When a verb to be is combined with main verb in past participle form to
express a passive voice. The passive voice is used to indicate an action is happen-
ing to the subject of sentence than the subject performing the action.
[4.35] The lunch was served.
7. Negative and interrogative verb phrases
VP gets separated when sentences have negative or interrogative nature.
[4.36] Jack is not answering the exam questions.
8. Emphasize verb phrases
Use auxiliary verbs, e.g. do, does, did to emphasize a sentence.
[4.37] Jack did enjoy the vacation.
9. Composite VP
When it consists of other VP or NP
[4.38] My cat Coco scratches the UPS courier on the table.
– Scratch is the main verb in VP to describe action/event happens to object
UPS courier, on the table is auxiliary information to further explain the event. It
still makes sense with/without it. It includes scratch VP + UPS courier NP + on
the table PP.

4.3.5 Complexity on Simple Constituents

Single word constituents are POS studied in Chap. 3, types of single word constitu-
ent types depend on tagset sizes. There are other complex design decisions:
[4.39] Jane bought the big red handbag ☑ vs.
[4.40] Jane bought the red big handbag ☒
74 4 Syntax and Parsing

Although there are two POS correct syntactic meanings but incorrect syntax, e.g.
red before big. There are also incomplete simple constituent types.
[4.41] The cat with a long tail meowing a collar. ☒
– Does not make sense although NP is correct, collar is an incorrect description.
[4.42] Jane imagined a cat with a long tail. ☑
[4.43] Jane decided to go. ☑
– Both make sense without further description in syntactic structure.
[4.44] Jane decided a cat with a long tail. ☒
– Does not make sense again in syntactic correctness.
[4.45] Jane decided a cat with a long tail should be her next pet. ☑
– Syntactic correct although the sentence structure is slightly complex.
[4.46] Jane gave Lily some food. ☑
– Syntactic correct although most of the time it describes food.
[4.47] Jane decided Lily some food. ☒
– Although syntactic structure is the same, but they have different designs to
further describe food types and purposes.

4.3.6 Verb Phrase Subcategorization

There is universal pattern or structure to classify verbs in VPs. Subcategories repre-

sent the ability of lexical items (usually verbs) to recognize the existence and types
of syntactic arguments they co-occur in linguistics (Brown and Miller 2020; Gorrell
2006). Although traditional English grammar subcategorizes verbs into transitive
and intransitive subcategories, modern English grammars distinguish over 100 sub-
categories. Subcategorization frames can be considered as a set of rules to generate
syntactic structures out of the base form. There are five major frame rules as shown
in Fig. 4.4.

Frame Rule Description Examples

ϕ VP with single verb as member. talk, sleep, eat, laugh, etc.

The verbal phrase requires a noun phrase (NP) as a specifier

VS(NP); find, see, leave, get, etc.
(VS) - intransitive verbs

The verbal phrase requires a noun phrase (NP) as a specifier

VS(NP)VC(NP); (VS) and a noun phrase (NP) as a complement (VC) show, make, read, write, etc.
(direct transitive verbs) - direct transitive verbs

The verbal phrase requires a noun phrase (NP) as a specifier

VS(NP)VC(PH([on])); (VS) and a prepositional phrase (PP) headed by “on” as a depend, insist, operate,
complement (VC) - indirect transitive verbs governing “on” suggest etc.

The verbal phrase requires a noun phrase as a specifier (VS),

VS(NP)VC(NP)VC(PH([to])); a noun phrase as a complement (VC), and a prepositional give, mean, think, etc.
phrase headed by “to” as a complement (VC) - ditransitive verbs

Fig. 4.4 Examples of verbs with different frames of subcategorization in VP syntax

4.3 Types of Constituents in Sentences 75

1. VP with single verb as member

[4.48] He talks. (VP → VB)
[4.49] I laugh. (VP → VB)
2. Verbal phrase requires a noun phrase (NP) as a specifier (VS)—intransitive verbs
[4.50] He finds a clue. (VP → VB + NP)
[4.51] She sees Jack. (VP → VB + NP)
3. Verbal phrase requires a noun phrase (NP) as a specifier (VS) and a noun phrase
(NP) as a complement (VC)—direct transitive verbs
[4.52] Please show me the map. (VP → VB + NP + NP)
4. Verbal phrase requires a noun phrase (NP) as a specifier (VS) and a prepositional
phrase (PP) headed by on as a complement (VC)—indirect transitive verbs gov-
erning on:
[4.53] This ingredient can make six muffins depending on size.
(VP → VB + PP + NP)
5. Verbal phrase requires a noun phrase (NP) as a specifier (VS), as a complement
(VC), and a prepositional phrase headed by to as a complement (VC)—ditransi-
tive verbs:
[4.54] Do you mean that I need to attend the exam? (VP-VB + S)

4.3.7 The Role of Lexicon in Parsing

A lexicon is the vocabulary of a language or branch of knowledge (Bender 2013;

Brown and Miller 2020), e.g. medical, computer science domain. It is a language’s
inventory of lexemes in linguistics. The word lexicon derives from Greek word
λεξικόν meaning of or for words.
Linguists believe that all human languages compose of two major components:
(1) lexicon as the list of a language’s words and vocabulary and (2) grammar as the
set of rules to allow words combinations into meaningful sentences.
Items within a lexicon are called lexemes, and groups of lexemes are called lem-
mas, often used to describe the size of a lexicon.
Lexical analysis is the process to understand what words mean, intuit contexts,
and note the relationship of one word to others. It analyzes and converts the sequence
of words into a list of lexical tokens. A program that performs such lexical analysis
is called tokenizer, lexer, or scanner. A lexer is combined with a parser generally to
analyze the syntax of sentences, texts, or dialogues.
The roles of lexicon in parsing are to:
1. Treat as the starting point for POS tagging.
2. Provide extra information such as subcategorization with frames and syntac-
tic rules.
For verbs, lexicon refers to several types of subcategorizations such as think
vs. laugh.
76 4 Syntax and Parsing

For adjectives:
[4.55] Jack is angry with Sophia vs. [4.56] Jack is angry at Sophia.
[4.57] Jack is mad at Sophia vs. [4.58] Jack is mad with Sophia. ☒
There are patterns and rules. Both are correct for [4.55] and [4.56] , but for the
verb mad, [4.57] is correct while incorrect for [4.58] which means subcategoriza-
tion is acceptable for some pattern but not only on syntax.
For nouns: [4.59] Janet has a passion for classical music vs. [4.60] Janet has an
interest in classical music.
They have different patterns of syntactic rules.

4.3.8 Recursion in Grammar Rules

English sentences can be very large and complex in structure. A concise sentence
usually consists of limited set of constituent types, i.e. NP, VP, and PP to recursion
and construct grammar rules as follows:
S → NP VP [4.61] My good friend Jack buys a flat.
VP → V NP [4.62] buys a flat.
NP → NP PP [4.63] My good friend.
NP → NP S [4.64] The boy who come early today won the game.
PP → prep NP [4.65] The cupcake with sprinkles is yours.

4.4 Context-Free Grammar (CFG)

4.4.1 What Is Context-Free Language (CFL)?

Context-free language (CFL) is a superset of Regular Language (RL) generated by

context-free grammar (CFG) which means every RL is a CFL but not all CFL is a
RL (Eisenstein 2019; Jurafsky et al. 1999). In short, CFL is:
1. Recursively enumerable language as superset of language model
2. Context-sensitive language, a subset of recursively enumerable language
3. Subsets of context-sensitive language.
The four levels of human language are shown in Fig. 4.5.
The set of all CFLs are identical to the set of languages accepted by pushdown
automata, and RL is a CFL subset. An input language is accepted by a computa-
tional model if it runs through the model and ends in an acceptable final state, most
arithmetic expressions generated by context-free grammar (CFG) is CFL.
CFL is closed in a specific operation if applying a CFL language operation on its
results. These operations include union, concatenation, Kleene closure, substitution,
4.4 Context-Free Grammar (CFG) 77

Fig. 4.5 Level of

languages

prefix, cycle, reversal, quotient, union, intersection, difference with RL, and homo-
morphism. CFL and CFG have NLP and computer language designs in computer
science and linguistics.

4.4.2 What Is Context-Free Grammar (CFG)?

CFG is to describe CFL as a set of recursive rules for generating string patterns,
because the application of production rules in a grammar is context-independent,
meaning they do not depend on other symbols with the rules (Bender 2013; Brown
and Miller 2020).
CFG is commonly applied in linguists and compiler design to describe program-
ming languages and parsers that can be created automatically.

4.4.3 Major Components of CFG

CFG consists of four major components (Bender 2013; Jurafsky et al. 1999):
1. A set of nonterminal symbols N are placeholders for patterns of terminal sym-
bols created by nonterminal symbols. These symbols usually located at the LHS
(left-hand-side) of production rules (P). The strings generated by CFG usually
consist of symbols only from nonterminal symbols.
2. A set of terminal symbols Σ (disjoint from N) are characters appear in strings
generated by grammar. Terminal symbols usually located only at RHS (right-
hand-side) of production rules (P).
3. A set of production rules P: A → α,where A is a nonterminal symbol and α is a
string of symbols from the infinite set of strings (Σ ∪ N).
4. A designated start symbol S is a start symbol of the sentence/utterance.
Σ is a set of POS and N is the set of constituent types, i.e. NP, VP, and PP men-
tioned in Chap. 3 and previous section, respectively.
78 4 Syntax and Parsing

4.4.4 Derivations Using CFG

The standard formulation of CFG is given by

Assume LG generated by grammar G is a set of strings composed of terminal
symbols which is generated from S:

{
LG = w|w is in Σ∗ and S ⇒ w } (4.1)

Let Σ be the set of POS, so CFG in Eq. (4.1) can create grammar like this:

N V det N (4.2)

The definition of CFG is given by

s | w is in Σ∗ and S ⇒ w and s can be derived from w by substituting 

LG =   (4.3)
words for POSas licensed by the lexicon 

This definition can generate numerous productions like this format:

S → NP VP (4.4)

Equation (4.4) is the most basic grammar rule where a sentence is generated
from a NP and a VP that can be further decomposed recursively as shown in Fig. 4.6.
It shows CFG rules and its corresponding parse tree for sentence/utterance [4.66]
Jane plays the piano. There are four tokens in this sentence/utterance to form a well-
defined syntactic structure generated by NP and VP. NP can be designated to a name
pointed to token Jane, and for VP, is decomposed into verb or NP as shown in four
production rules shown in the top left corner at Fig. 4.6. In this case, verb is pointed
to plays, NP can be decomposed into a determiner and a noun pointed to the and
piano, respectively.

CFG Production Rule

S ® NP VP Parse Tree
NP ® Name S
NP ® Det N
VP ® V NP
NP VP

Name V NP

(S (NP (NAME Jane)) Jane plays Det N

(VP (V plays)
(NP (ART the)
(N piano)))) the piano

Fig. 4.6 CFG rules and corresponding parse tree for sentence [4.66] Jane plays the piano
4.5 CFG Parsing 79

4.5 CFG Parsing

There are three CFG parsing levels: (1) morphological, (2) phonological, and (3)
syntactic (Grune and Jacob 2007; Jurafsky et al. 1999).

4.5.1 Morphological Parsing

Morphological parsing is the initial level to determine morphemes of a word being

constructed. For example, a morphological parser can reveal that the word mice is
the plural form of noun stem mouse, while cats is the plural form of noun stem cat.
Given the string cats as input, the morphological parser will interpret cats as cat N
PL. By using FSA (Finite State Automata), FST (Finite State Transducer), a mor-
phological parser can produce an output with their stems and modifiers.
Originally, FST is generated by algorithmic parsing of word source such as a
complete dictionary with modifier markups but can be realized by recurrent neural
networks with training corpus upon advancement in machine learning and artificial
neural networks.

4.5.2 Phonological Parsing

Phonological parsing is the second level using the sounds of a language, i.e. pho-
nemes to process sentences/utterances (Wagner and Torgesen 1987).
Phonological processing includes (1) awareness, (2) working memory, and (3)
phonological retrieval. All three components are important to speech production
and written language skills development. Hence, it is necessary to observe chil-
dren’s spoken and written language development with phonological processing
difficulties.
Phonological parsing is to interpret sounds into words and phrases to gener-
ate parser.

4.5.3 Syntactic Parsing

Syntactic parsing is the third level to identify relevant components and correct
grammar of a sentence. Abstract meaning representation is assigned to define legal
strings of a language like CFG without recognizing the structure.
Parsing algorithms are applied to analyze sentences/utterances within language
and assign appropriate syntactic structures into them. Parse trees are useful to study
grammar, semantic analysis, machine translation, speech recognition, Q&A chat-
bots in NLP.
80 4 Syntax and Parsing

4.5.4 Parsing as a Kind of Tree Searching

Syntactic parsing can be considered as search within a set of parse trees, its main
purpose is to identify the right path and space through automation in an FSA system
structure.
CFG is a process to determine the right parse tree among all possible options. If
there is more than one possible parse tree, stochastic method (or other machine
learning methods) will be applied to locate a probable one. In other words, it is a
process to identify search space defined by grammatical rules so that their con-
straints can become inputs to perform automatic parsing and study grammars.

4.5.5 CFG for Fragment of English

English grammar and lexicon simplified domains are applied to reveal CFG rules in
an example of musical instruments as shown in Fig. 4.7. It consists of production
rules from several categories S → NP VP, S → Aux NP VP, S → VP as well as pro-
duction rules for NP, Nom, and VP with components Det, N, V, Prep, and PropN.

4.5.6 Parse Tree for “Play the Piano” for Prior CFG

A parse tree of sentence/utterance [4.67] play the piano is shown in Fig. 4.8. It has
three tokens play—Verb, the—Det, and piano—Noun to construct a parse tree from
top node S to generate VP, VP to generate Verb and NP, and NP to decompose into
Det Nom, and Nom to generate Noun.

Fig. 4.7 A simplified example on English grammar and lexicon

4.5 CFG Parsing 81

Fig. 4.8 Parse tree for the S

simple sentence “Play the
piano”

Nom

Verb Det Noun

Play the piano

4.5.7 Top-Down Parser

There are (1) top-down and (2) bottom-up parser approaches to construct a parse
tree. Top-down parser constructs from rootnode S down to leavenodes (words in the
sentence/utterance). The first step is to identify all trees with root S, the next step is
to expand all constituents in these trees based on the given production rules. The
whole process is operated in level-by-level process until parse trees reach the leaves,
i.e. POS tokens of the sentence/utterance. For candidate parse trees that cannot
match the leave nodes, i.e. POS tokens are discarded and considered as failed parse
tree(s). Figure 4.9 shows first three-level construction all possible parse trees apply-
ing Top-Down parser.
It showed that the parse tree construction started from base level with S tag (root
node). The second level has generated an additional layer with three possible pro-
duction rules: S → NP & VP, S → Aux & NP & VP, and S → VP. The third level is
complex because it has decomposed into three levels, S → NP & VP is the first
variation to decompose into Det and Nom. NP is the second variation to decompose
into PropN. It is noted that LHS is the expanded part for demonstration purpose, but
both LHS and RHS are required expansion. S → Aux & NP & VP are the second
variation where NP to decompose into Det & Nom, and a NP decompose into
PropN. VP decomposition in the first four parse tree is not shown as they all failed
to match the leave nodes except only the fifth case is correct to form a complete play
the piano parse tree.
Top-down approach by CFG on terminals and nonterminals is shown in Fig. 4.10.
It showed rule 3 as the first one to apply and rule 2 for VP decomposed into V NP
and V to decompose play and then NP to Det and Nom, rule 4 and rule 5 are Det
points to the, and rule 6 Nom points to an and final rule points to end, and rule 7
82 4 Syntax and Parsing

Base level: S

S S S
Second level:

NP VP Aux NP VP VP

S S S S S S
Third level:
NP VP NP VP Aux NP VP Aux NP VP VP VP

Det Nom PropN Det Nom PropN V NP V

Fig. 4.9 A three-level expansion of parse tree generation using top-down approach

Fig. 4.10 CFG rules and terminal/nonterminal nodes being used with top-down approach parsing

points to piano. This will complete a top-down approach parsing with the fifth parse
tree end-up as valid solution. Readers can base on these seven-step process to com-
plete the construction of parse tree for the fifth case as an exercise.

4.5.8 Bottom-Up Parser

Bottom-up parser, however, starts from token words of the sentences/utterances to

construct parser tree upward by applying the same set of production rules and try to
generate from right-hand-side (RHS) of the production rule in reverse order. In
example [4.67] play the piano has two variations to start in which the word play can
be considered either as a noun (N) or a verb (V). So, there are two options: one with
4.5 CFG Parsing 83

the play the piano as N Det or as V Det N. Since this approach cannot indicate
which one is the correct option so the parsing operation will continue to grow until
they can reach the root node S, and if they cannot match the root node, the tree(s)
will be discarded.
Figure 4.11 shows the first three-level expansion of a parse tree using bottom-up
approach. So, in this case play the piano has two variations either play is N or
V. There are two parts one is play consider as N and other as V from base level. So,
at second level is to further expand the line pointed to play and tried to expand N
pointed to play into N in the first case. In second case is to further expand N pointed
to Nom in second layer. In the third level, second case is further expanded into two
options, one is Nom → V and the other is VP → V & NP, NP → Det & Nom, and
further up to S → VP to complete the whole parsing, in which other two parse tree
options endedup with invalid parsing as shown in Fig. 4.11.
Figure 4.12 shows CFG rules for terminal and nonterminal nodes applying bot-
tom-up approach parsing. Again, it consists of seven steps. Rule 1 is V pointed to
play, rule 2 is Det pointed to the, rule 3 is N pointed to piano, rule 4 is Nom pointed
to N, rule 5 is NP pointed to Det & Nom, rule 6 is VP pointed to NP, and rule 7 is S
point to VP to complete the whole parse tree until it can finally match the root/
source node S.

NP NP

Nom Nom Nom Nom Nom

Third Level: Noun Det Noun Verb Det Noun Verb Det Noun

Play the piano Play the piano Play the piano

Nom Nom Nom

Noun Det Noun Verb Det Noun

Second Level:

Play the piano Play the piano

Noun Det Noun Verb Det Noun

Base Level: Play the piano Play the piano

Play the piano

Fig. 4.11 A three-level expansion of parse tree generation using bottom-up approach
84 4 Syntax and Parsing

Fig. 4.12 CFG rules and terminal/nonterminal nodes being used with bottom-up approach parsing

4.5.9 Control of Parsing

Although both top-down and bottom-up parsing are straightforward, the control of
parsing is still needed to consider (1) which node that need to expand first and (2)
select grammatical rules sequence wisely to save time as most of the parse tree gen-
eration are dead-end and wastage of resources.

4.5.10 Pros and Cons of Top-Down vs. Bottom-Up Parsing

4.5.10.1 Top-Down Parsing Approach

Pros

Since it starts from root/source node S, it can always generate a correct parse tree
unless the sentence has a syntactic error. In other words, it never explores the parse
that will not end up to root/source node S which means it will always find a solution.

Cons

This approach does not consider final word/token tags during parsing from the very
beginning, it will waste a lot of time to generate tree(s) that may be totally unrelated
to the correct result. play should parse as V instead of N as shown in Fig. 4.9, this
approach showed that all first fourth parts of parse tree using play as N are invalid
and waste of time to parse tree generation.
4.6 Lexical and Probabilistic Parsing 85

4.5.10.2 Bottom-Up Parsing Approach

Pros

Since it starts from sentence tokens/POS, it can always generate parse tree with all
tokens/POS in the sentence considered and reduced time on rules unrelated to these
tokens which means it can sort out problems occurred in top-down approach for all
production rules without POS tags.

Cons

This approach may often end up with broken tree(s) that cannot match the root node
S to complete parse tree as it starts from leave node instead of root/source node S. It
makes sense because although there are many ways to match production rules, the
variations of most parse trees are syntactic incorrect so they cannot match the root/
source node S. All parse trees in Fig. 4.11 showed that except the last one (also the
correct one), others ended up with broken trees and failed to match the root/source
node S again wasted time to parse tree generation.
Let us look at lexicalized and probabilistic parsing as alternative.

4.6 Lexical and Probabilistic Parsing

4.6.1 Why Using Probabilities in Parsing?

There are two reasons using probabilities parsing (Eisenstein 2019; Jurafsky et al.
1999): (1) resolve ambiguity and (2) word prediction in voice recognition. For
instance:
[4.68] I saw Jane with the telescope. (Jane with telescope or I use telescope to
see Jane?)
[4.69] I saw the Great Pyramid flying over Giza plateau vs.
[4.70] I saw UFO flying over Giza plateau
Although both situations have pragmatic problems in which [4.69] is incorrect
because Great Pyramid is an unmovable architecture. It can be solved using proba-
bilities in parsing from a large corpus and knowledge base (KB) to identify the
frequencies of a particular term or constituent is used correctly without pragmatical
analysis.
For example, in voice recognition:
[4.71] Jack has to go vs.
[4.72] Jack half to go vs.
[4.73] If way thought Jack wood go
86 4 Syntax and Parsing

Note: when analyzing N-gram probabilities in Chapter 2 on N-gram Language

Model, I have, I should, I would usage and bigram probabilities from The Adventures
of Sherlock Holmes, they provided directions for one that is more probable and used
frequently instead of understanding their exact semantic or pragmatic meanings. So,
such probabilistic method can also be applied to parsing.

4.6.2 Semantics with Parsing

The following examples show how semantic meanings (Bunt et al. 2013; Goddard
1998) affect/determine the validness of sentence/utterance in parsing:
[4.74] Jack drew one card from a desk [?] vs.
[4.75] Jack drew one card from a deck.
Note: drew → deck is clearly a semantic concern.
[4.76] I saw the Great Pyramid flying over Giza plateau. [?] vs.
[4.77] I saw a UFO flying over Giza plateau.
Note: movable vs. unmovable objects.
[4.78] The workers dumped sacks into a pin. [?] vs.
[4.79] The workers dumped sacks into a bin.
Note: dump looks for a locative complement.
[4.80] Tom hit the ball with the pen. [?] vs.
[4.81] Tom hit the ball with the bat.
Note: which object can use to hit the ball?
[4.82] Visiting relatives can be boring. [?] vs.
[4.83] Visiting museums can be boring.
Note: Visiting relatives are genuinely ambiguous. Visiting museums are evident
as only animate bodies can visit. There is no need for abstraction with enough data,
in other words, sufficient large corpus, databank, or dialogue databank can sort out
ambiguity problems to work out correct syntax with semantic meaning in many cases.
There are two classical approaches to add semantics into parsing: (1) cascade
systems to construct all parses and use semantics for rating tedious and complex; (2)
do semantics incrementally.
A modern approach is to forget the meaning and only based on KB and corpus.
If a corpus contains sufficient sentences and knowledge, facts about meaning
emerge in the probability of observed sentences themselves. It is modern because
constructing world models are harder than early researchers realized but there are
huge text corpora to construct useful statistics. Here comes lexical and probabilistic
approach of parsing.
4.6 Lexical and Probabilistic Parsing 87

4.6.3 What Is PCFG?

A probabilistic context-free grammar (PCFG) is a context-free grammar that asso-

ciates each of its production rule with a probability. It creates the same set of parses
for a test that traditional CFG performs but assigns a probability value to each parse.
In other words, the probability of a parse generated by a PCFG is the product of
probability’s rules.
The general format of PCFG production rule is given by

A → β [ p] (4.5)

Another way to interpret it as:

P ( A → β |A ) (4.6)

Note: the sum of all probabilities of rules with LHS A must be 1.

PCFG extends CFG like how Markov models extend regular grammars. Each
production rule is assigned with a probability. The probability of a parse is the prod-
uct of probabilities of productions used in that derivation. These probabilities can be
regarded as parameters of the model, and for large NLP problems it is convenient to
learn these parameters via machine learning methods. A probabilistic grammar’s
validity is constrained by context of its training dataset.
An efficient PCFG design must weigh scalability, generality factors and issues
such as grammar ambiguity must be resolved to improve system performance.

4.6.4 A Simple Example of PCFG

This section used sentences/utterances [4.84] buy coffee from Starbucks as example
to illustrate how PCFG works. It has simple CFG rules and probabilities in a seg-
ment of an AI chatbot dialogue for food ordering at campus as shown in Fig. 4.13.
The probability of each production rule type must sum up to 1 is one of the most
important basic criteria of PCFG as shown. For instance, three production rules of
S: S → NP VP (0.82), S → Aux NP VP (0.12) and S → VP (0.06) must sum-up to
1. It is the same as other production rules for NP, Nom, VP, Det, N, V, Aux, Proper-N,
and Pronoun. Of course, if the corpus is very large, some of these probability values
will be very small, just like N-gram probability evaluation discussed in Chap. 2.
It can apply either top-down parser or bottom-up parser approach to generate
parse tree with the following PCFG probability evaluation scheme:

P (T ) = ∏p ( r ( n ) ) (4.7)
n∈T

where p(r(n)) is the probability that rule r will be applied to expand the non-
terminal n.
88 4 Syntax and Parsing

Fig. 4.13 Sample CFG rules and their probabilities in AI Chatbot dialogues (food ordering
at campus)

So, what required to achieve is

arg max P ( T )
Tˆ ( S ) = (4.8)
T ∈τ ( S )

where τ(S) denotes the set of all possible parses for S.

Figure 4.14 depicts different meanings and two parse trees for utterances [4.85]
can you buy Starbucks coffee? The first interpretation regards Starbucks and coffee
are two standalone NPs with equal significance. The second interpretation is to
combine Starbucks and coffee into a single NP constituent which is a brand name,
in this case, can you buy Starbucks coffee can interpret to buy coffee or non-coffee
items. Hence, parse tree probability calculation is also different.
Figure 4.15 shows all CFG rules and associated probabilities of these two parse
trees. So, PCFG probabilities for parse trees 1 and 2 are

P ( PT1 ) = 0.12 × 0.36 × 0.06 × 0.06 × 0.37 × 0.72 × 0.43 × 0.36 × 0.41 × 0.63 × 0.75
= 1.242 × 10 −6

P ( PT2 ) = 0.12 × 0.36 × 0.36 × 0.06 × 0.05 × 0.72 × 0.43 × 0.36 × 0.41 × 0.63 × 0.75
= 1.007 × 10 −6
4.6 Lexical and Probabilistic Parsing 89

PT1: Can you buy Starbucks coffee? PT2: Can you buy Starbucks coffee?

S S

Aux NP VP Aux NP VP

V NP NP V NP

Nom

Nom Nom

Pro PNoun Noun Pro PNoun Noun

Can you buy Starbucks coffee Can you buy Starbucks coffee

Fig. 4.14 Two possible parse trees for utterance “Can you buy Starbucks coffee”?

Fig. 4.15 CFG rules and associated probabilities for two possible parse trees PT1 vs. PT2

CFG probability algorithm parse tree 1 has a high probability. In other words, it
is more possible the meaning is to buy coffee other than buy other things from
Starbucks. It also shows an efficient solution to differentiate which parse tree is
more probable, when there are ambiguities in two or more parse trees provided with
sufficient lexical probabilities and corpus to calculate probabilities.
90 4 Syntax and Parsing

4.6.5 Using Probabilities for Language Modeling

Probability parsing can be considered as the integration of N-gram probability con-

cept with parse tree formation. Since there are few grammar rules than word
sequence for N-gram generation, applying this calculation method one can calculate
results efficiently instead of N-gram frequencies regardless of syntactic meaning
and rules.
Based on this method, the probability of S is the sum of probabilities of all pos-
sible parses given by

P (S ) = ∑ P (T ) (4.9)
T ∈τ ( S )

against N-gram probability calculation with Markov model.

P ( S ) = P ( w1 ) ∗ P ( w2 |w1 ) ∗ P ( w3 |w1 w2 ) ∗ P ( w4 |w1 w2 w3 )… (4.10)

4.6.6 Limitations for PCFG

In many situations, it is adequate to know that one rule is used more frequently than
another, e.g.
[4.86] Can you buy Starbucks coffee? vs. [4.87] Can you buy KFC coffee?
But often it matters what the context is.
For example:

S → NP VP
NP → Pronoun [0.80] (4.11)
NP → LexNP [0.20]
For example, when NP is the subject, the probability of a pronoun may be higher
at 0.91. When NP is the direct object, the probability of a pronoun may be lower at
0.34 which means it depends on NP position in a sentence/utterance. In other words,
the probabilities also often depend on lexical options as shown in the following
examples:
[4.88] I saw the Great Pyramid flying over Giza Plateau. vs.
[4.89] I saw a UFO flying over Giza Plateau.
[4.90] Farmer dumped sacks in the bin. vs.
[4.91] Farmer dumped sacks of apples.
[4.92] Jack hit the ball with the bag. vs.
[4.93] Jack hit the ball with the bat.
[4.94] Visiting relatives can be boring. vs.
4.6 Lexical and Probabilistic Parsing 91

[4.95] Visiting museums can be boring.

[4.96] There were boys in park and girls vs.
[4.97] There were boys in park and shops.
For instance, there are two interpretations of utterances [4.98] boys in park and
girls as shown in Fig. 4.16 showing their syntax ambiguities.
Figure 4.16 shows two possible parse trees for utterance [4.98]. The first is boys
in park is a noun clause with conjunction NP girls. The second is park and girls
belong to a single NP with boys as single NP. Although structures are different but
the mathematization result for two parse trees is identical which means CFG prob-
ability calculation cannot differentiate which is better or popular. How to fix this
problem?

4.6.7 The Fix: Lexicalized Parsing

The lexicon can be considered as an estimation of a knowledge base (KB) a possible

solution to the above ambiguous problem.
Figure 4.17 shows [4.90] applying lexical parsing as example. Each constituent
is a head word, i.e. S using dumped as Head word. NP and VP are signified by
farmer and the other signified by head word dumped at second tier. From farmer it
comes up with NNS farmer. VP from dumped because will come up with VBD, NP
and PP and VDB is signified by dumped as head word and NP is sack and PP is into.
So sack further decomposes in NNS which points to sacks, for PP to further decom-
poses into to P and NP into bin. This will provide information to further decomposi-
tion by combining keywords. So, for NP bin it will further decompose into the and
the bin as head words for DT and NN, respectively.

PT1: (boys in park) NP and girls PT2: boys in (park and girls) NP

NP NP

NP Conj NP NP PP

NP PP Noun Noun Prep NP

Noun Prep NP NP Conj NP

Noun Noun Noun

boys in park and girls boys in park and girls

Fig. 4.16 Two interpretations of the utterance “boys in park and girls”
92 4 Syntax and Parsing

S (dumped)

NP (farmer) VP (dumped)

NNS (farmer) VBD(dumped) NP(sacks) PP (into)

NNS(sacks) P (into) NP (bin)

DT (the) NN (bin)

farmer dumped sacks into the bin

Fig. 4.17 Lexical tree for the utterance “workers dumped sacks into a bin”

By adding lexical items with production rules:

VP ( dumped ) → VBD ( dumped ) NP ( sacks ) PP ( into ) = 8 × 10 −10 (4.12)

VP ( dumped ) → VBD ( dumped ) NP ( cats ) PP ( into ) = 1 × 10 −10 (4.13)

VP ( dumped ) → VBD ( dumped ) NP ( stones ) PP ( into ) = 2 × 10 −10 (4.14)

VP ( dumped ) → VBD ( dumped ) NP ( sacks ) PP ( above ) = 1 × 10 −12 (4.15)

with lower probabilities means infrequency appeared in the corpus.

This determination method is more efficient as compared with N-gram probabil-
ity calculation, sample sentences/utterances such as:
[4.99] The farmer dumped sacks of apples into a bin. vs.
[4.100] The famer dumped sacks of peaches into a bin. vs.
[4.101] The farmer dumped all the sacks of apples into a bin.
But there will be situations that many lexical probabilities comeup with 0 values
like N-gram probability evaluation.
A short-cut by considering the following lexical rule as replacement instead of
considering the whole lexical rule such as Eq. (4.12) can sort out this problem:

VP ( dumped ) → VBD NP PP p ( r ( n ) | n, h ( n ) ) (4.16)

By doing so, lexical probability of certain nodes n with heads h is considered

based on two conditions: (1) syntactic type of node n and (2) head of node’s mother
h(m(n)), so lexical rule of Eq. (4.16) is split into the following:
4.6 Lexical and Probabilistic Parsing 93

(
Given P h ( n ) = word i |, n|, h ( m ( n ) ) )
VP ( dumped ) → PP ( into ) , p = p1 (4.17)
VP ( dumped ) → PP ( of ) , p = p2
NP ( sacks ) → PP ( of ) , p = p3

Now the original lexical probability Eq. (4.7) becomes

(
P ( T ) = ∏ p ( r ( n ) |, n|, h ( n ) ) ∗ p h ( n ) |, n|, h ( m ( n ) )
n∈T
) (4.18)

Using Brown corpus as example, the probability of:

p ( VP → VBD NP PP|, VP|, dumped ) = 0.67

p ( VP ) → VBD NP# VP, dumped ) = 0.0
(4.19)
p ( into|, PP|, dumped ) = 0.22
p ( into|, PP|, sacks ) = 0

parse contribution of this part to the total scores for two candidates will be

[dumped into] 0.67 × 0.22 = 0.147

(4.20)
[sacks into ] 0 × 0 =0

So, we should consider dumped into instead of sacks into in this case.

Exercises
4.1 What is syntax and parsing in linguistic? Discuss why they are impor-
tant in NLP?
4.2 What is syntactic rule? State and explain SEVEN commonly used syntactic
patterns in English language, with an example each to illustrate.
4.3 Answer (4.2) by applying to other language such as Chinese, French, or
Spanish. What is (are) the different of the syntactic rules between these two
languages with example to illustrate.
4.4 What are constituents in English language? State and explain three commonly
used English constituents, with an example each to illustrate how it works.
4.5 What is context-free grammar (CFG)? State and explain the importance of
CFG in NLP.
4.6 State and explain FOUR major CFG components in NLP. Use an example
sentence/utterance to illustrate.
4.7 What are TWO major types of CFG parsing scheme? Use an example sen-
tence/utterance [4.102] Jack just brought an iPhone from Apple store to illus-
trate how these parsers work.
94 4 Syntax and Parsing

4.8 What is PCFG in NLP parsing? Use same example [4.102] Jack just brought
an iPhone from Apple store to illustrate how it works. Compare with parsers
used in (4.7), which one is better?
4.9 What are the advantages and limitations of PCFG in NLP parsing? Use some
sample sentences/utterances to support your answers.
4.10 What is lexical parsing in NLP parsing? Discuss and explain how it works by
using sample sentence [4.102] Jack just brought an iPhone from Apple store
for illustration.

References

Allen, J. (1994) Natural Language Understanding (2nd edition) Pearson

Bender, E. M. (2013) Linguistic Fundamentals for Natural Language Processing: 100 Essentials
from Morphology and Syntax (Synthesis Lectures on Human Language Technologies). Morgan
& Claypool Publishers
Brown, K. and Miller, J. (2020) Syntax: A Linguistic Introduction to Sentence Structure. Routledge.
Bunt, H. et al. (2013) Computing Meaning: Volume 4 (Text, Speech and Language Technology
Book 47). Springer.
Eisenstein, J. (2019) Introduction to Natural Language Processing (Adaptive Computation and
Machine Learning series). The MIT Press.
Goddard, C. (1998) Semantic Analysis: A Practical Introduction (Oxford Textbooks in Linguistics).
Oxford University Press.
Gorrell, P. (2006) Syntax and Parsing (Cambridge Studies in Linguistics, Series Number 76).
Cambridge University Pres.
Grune, D. and Jacob, C. (2007) Parsing Techniques: A Practical Guide (Monographs in Computer
Science). Springer.
Khanam, H. M. (2022) Natural Language Processing Applications: Part of Speech Tagging.
Scholars’ Press.
Jurafsky, D., Marin, J., Kehler, A., Linden, K., Ward, N. (1999). Speech and Language Processing:
An Introduction to Natural Language Processing, Computational Linguistics and Speech
Recognition. Prentice Hall.
Tuchong (2022 The milky cat with long tail meowing. https://stock.tuchong.com/image/detail?im
ageId=896801601105166424. Accessed 20 July 2022.
Wagner, R. K., & Torgesen, J. K. (1987). The nature of phonological processing and its causal
role in the acquisition of reading skills. Psychological Bulletin, 101(2), 192–212. https://doi.
org/10.1037/0033-2909.101.2.192.
Chapter 5
Meaning Representation

5.1 Introduction

The understanding of sentences/utterances in terms of structure, grammar, and the

relationship between words using N-gram models or simple syntactic rules has hith-
erto studied, but not on actual meaning of sentences or different words in a sentence.
This chapter will explore how to interpret meaning and scientific/logical methods to
process meaning called meaning representation. It is difficult to interpret advance
NLP analysis which involves semantic meaning, pragmatic meaning, and discourse
in subsequent chapters without it. Let us start with meaning representations that
amalgamate knowledge linguistic forms of real-world to the world of linguistics.

5.2 What Is Meaning?

Language is prodigious in recognizing humans encode or decode world description

from experiences to ideas and interpret others’ opinions. It is natural but difficult to
utter word strings that match the world into expressions. A way to enrich this activ-
ity is to transform essences that wish to convey into meaningful words, clauses,
phrases, or sentences/utterances by verbal or written forms for others to listen,
understand, inference, and even respond.
Meaning is the message to convey by words, phrases, and sentences/utterances
with context in linguistics. It is also called lexical or semantic meanings. Prof. W
Tecumseh Fitch described semantics meaning in The Evolution of Language (Fitch
2010) as a branch of language study that consistently related with philosophy. This
is because the study of semantic meaning raises many fundamental philosophical
problems that needed to be solved and explained by philosophers.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 95

Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_5
96 5 Meaning Representation

A good dictionary provides meaning explanation of a single word in details and

many dictionaries on concept/language translation. Nevertheless, the meanings of
sentences/utterances are not simply the combination of individual word’s meaning,
but usually appeared as phrasal words with specific meanings at pragmatic level,
e.g. [5.1] off the wagon.
Semantic meaning is the study of meaning assignment to minimal meaning-bear-
ing elements to form complex and meaningful ideas. Some basic word groups may
be aggregated in content relationship called thematic groups, and lexical or seman-
tic fields related to common sense or world knowledge, e.g. the concept of doctor in
English constitutes the lexical semantic field in two senses: a medical doctor, or a
person with PhD title. Once the meaning of a word (word group) is decrypted or
analyzed, reaction is formed as response to the event it represents. Words and their
meanings are significant informational cues to understand languages. Further, a per-
son’s life experience and cultural difference are relevant to linguistic meaning
development in communication process.

5.3 Meaning Representations

This chapter will adopt similar approach as per syntax and morphology analysis
(Bender 2013) to create linguistics inputs representations and capture their mean-
ings. These linguistic representations are meanings characterization of sentences
and state-of-affairs in real-world situation.
Unlike parse trees, these representations are not primarily descriptions of input
structure, but is a kind of representation of how humans understand, mean anything
such as actions, events, and objects, etc. and try to make sense of it in our environ-
ment—the meaning of everything.
There are five types of meaning representation: (1) categories, (2) events, (3)
time, (4) aspect, and (5) beliefs, desires, and intentions.
1. Categories refer to specific objects and entities, e.g. company names, locations,
objects.
2. Events refer to actions or phenomena experienced, e.g. eating lunch, watching a
movie. They are relevant to verbs or verb phases expressed in POS.
3. Time refers to exact or reference moment, e.g. 9:30 am, next week, 2023.
4. Aspects refer to:
(a) Stative—to state facts.
For example: [5.2] Jane knows how to run.
(b) Activity—to describe action.
For example: [5.3] Jane is running.
(c) Accomplishment—to describe completed action without ending terms.
For example: [5.4] Jane booked the room.
(d) Achievement—to describe terminated action.
For example: [5.5] Jane found the book.
5.4 Semantic Processing 97

5. Beliefs, desire, and intention refer to principles such as:

[5.6] I think what you are saying is totally correct.
[5.7] Jane wants to know why she failed in the test.
[5.8] I believe everything happens for a reason.
These principles are complex as they involve several thoughts in philosophy.
Nevertheless, it is important to devise suitable, logical, and computational represen-
tations in NLP to facilitate semantic processing and present ideas in sentences/
utterances.

5.4 Semantic Processing

Semantic processing (Bender and Lascarides 2019; Best et al. 2000; Goddard 1998)
undertakes meaning representation to encode and interpret meanings. These repre-
sentations allow to:
1. Reason relations with the environment
For example: [5.9] Is Jack inside the classroom?
2. Answer questions based on contents
For example: [5.10] Who got the highest grade in the test?
3. Perform inference based on knowledge and determine the verity of unknown
fact(s), thing(s), or event(s),
For example: [5.11] If Jack is in the classroom, and Mary is sitting next to him,
then Mary is also in the classroom.
Semantic processing applied to typical applications includes Q&A chatbot sys-
tems, it is necessary to understand meanings, i.e. the ability to answer questions
about context or discourse with knowledge, literal or even embedded meanings for
implementation. The following shows live examples in our AI Tutor chatbot (Cui
et al. 2020) which involve different degrees of semantic processing:
[5.12] What is the meaning of NLP?
– Basic level of semantic processing for the meaning of certain concept.
[5.13] How does N-gram model work?
– Requires understandings on facts and meanings to respond.
[5.14] Is Turing Test still exist?
– Involves high-level query and inference from previous knowledge.
[5.15] Why do we need to study self-awareness in AI?
– Involves high-level information such as world knowledge or common
sense aside AI terminology knowledge base to respond.
[5.16] Should I study AI?
– Involves the highest-level information about user’s common sense and
world knowledge aside AI concepts learnt by the book.
98 5 Meaning Representation

5.5 Common Meaning Representation

There are four common methods of meaning representation scheme: (1) First-Order
Predicate Calculus (FOPC), (2) Semantic Networks (semantic net), (3) Conceptual
Dependency Diagram (CDD), and (4) Frame-based Representation. A sample sen-
tence [5.17] Jack drives a Mercedes is used to illustrate how they perform.

5.5.1 First-Order Predicate Calculus (FOPC)

First-Order Predicate Logic (FOPL) (Dijkstra and Scholten 1989; Goldrei 2005) is
also known as predicate logic, or first-order predicate calculus. It is a robust lan-
guage representation scheme to express the relationship between information
objects as predicates. For example, FOPC meaning for [5.17] is given by

x, y Driving x Driver Jack,x DriveThing y,x

CarBrand Merceedes,y (5.1)

This FOPC formulation consists of four predicate calculus segments (predicates)

in logical terms.

5.5.2 Semantic Networks

Semantic networks (semantic nets) (Jackson 2019; Sowa 1991) are knowledge rep-
resentation technique used for propositional information. They convey knowledge
meanings in a two-dimensional representation. A semantic net can be represented as
a labelled directed graph. The logic behind is that a concept meaning is connected
to other concepts and can be represented as a graph. The information in semantic net
is characterized as a set of concept nodes to link up with each other by set of labelled
arcs which characterized the relationship as illustrated in Fig. 5.1 for example sen-
tence [5.17].

Fig. 5.1 Semantic net for Driving

example sentence [5.17]

Driver DriveThing

Jack Mercedes
5.5 Common Meaning Representation 99

Driving is the core concept connected to two nodes (concepts): Driver and
DriveThing which links to Jack as the driver and Mercedes as drivething respectively.

5.5.3 Conceptual Dependency Diagram (CDD)

Conceptual Dependency Diagram (CDD) is a theory to describe how to represent

the meaning of sentences/utterances to draw inferences. It has been argued that
CDD representation is independent of the language in which sentences are stated
originally.
Schank (1972) proposed Conceptual Dependency (CD) theory as a part of a nat-
ural language comprehension project. Sentences/utterances applying CD can trans-
late and express basic concepts as a small set of semantic primitives, which can be
integrated to represent complex meanings—conceptualizations. Figure 5.2 shows a
CD diagram for example sentence [5.17].
Mercedes and Jack are two concepts linkup by main concept Drive-by using CD
representation.

5.5.4 Frame-Based Representation

Frame-based systems use frames and notions as basic components to characterize

domain knowledge introduced by Prof. Marvin Minsky in his remarkable work A
framework for representing knowledge published in 1975 (Minsky 1975).
A frame is a knowledge configuration to characterize a concept such as a car or
driving a car attached to certain definitional and descriptive information. There are
several constructed knowledge representation systems based on the original model.
The vital successor of frame-based representation schemes are description logs that
encode the declarative part of frames using semantic logics. Most of these semantic
logics are components in first-order logic that related to feature logics. A frame-
based representation for [5.17] is shown in Fig. 5.3.

Fig. 5.2 Conceptual

dependency diagram for
Mercedes
example sentence [5.17]

Drive-by

Jack
100 5 Meaning Representation

Fig. 5.3 Frame-based Driving

diagram for example
Driver: Jack
sentence [5.17]
DriveThing: Mercedes

The frame-based representation is also invariance to language(s) being used like

other meaning representation model.
In summary, these meaning representations indicated that the linguistics mean-
ing for [5.17] describes certain state-of-affairs happened in a real-world. Different
meaning representation models are just different ways to represent the same sce-
nario. For example, FOPC is a kind of mathematical and logical representation of
meanings, while semantic nets are graphical representation of such meaning in the
form of directed graphs.

5.6 Requirements for Meaning Representation

There are three factors to fulfill a meaning representation (Bunt 2013; Butler 2015;
Potts 1994): (1) Verifiability, (2) Ambiguity, and (3) Vagueness considerations.

5.6.1 Verifiability

Verifiability is to determine sentence/utterance is literally meaningful (it expresses a

proposition) if and only if it is either analytic or empirically verifiable, which means
that it must provide ways for comparison of meaning representations to facts against
knowledge base, world knowledge, or common sense such as:
For example: [5.18] Does Jack drive a Mercedes?
A verifiable meaning representation asserts to prove the correctness of this state-
ment with comparison, matching, or inferencing operations.
The answer is yes according to statement [5.17].

5.6.2 Ambiguity

Ambiguity is a word, statement, or phrase that consists of more than one meaning.
Ambiguous words or phrases can cause confusion, misunderstanding, or even
humor situations.
For example: [5.19] Jack rode a horse in brown outfit.
5.6 Requirements for Meaning Representation 101

This clause may drive readers to wonder that the horse wore brown outfit instead
of the rider. Likewise, same words with different meanings induce ambiguity, e.g.
Jack took off his gun at the bank. It is diverting to confuse the meaning of bank
refers to a building or the land alongside of a river or lake. Context meaning is
important to resolve ambiguity.

5.6.3 Vagueness

Vagueness is to describe borderline cases, e.g. tall is a vague term in the sense that
a person who is 1.6 m in height is neither tall nor short since there is no amount of
conceptual analysis or empirical investigation can settle whether a 1.6 m person is
tall or not without any frame of reference. Here is another live example:
[5.20] He lives somewhere in the south of US.
– is also vague as to the meaning of location.
Ambiguity and vagueness are two varieties of uncertainty which are often dis-
cussed together but are distinct in essential features and significances in semantic
theory. Ambiguity involves uncertainty about mapping between representation lev-
els which have more than a single meaning with different structural characteristics,
while vagueness involves uncertainly about the actual meaning of terms. Hence, a
good meaning representation system should resolve vagueness and avoid ambiguity.

5.6.4 Canonical Forms

5.6.4.1 What Is Canonical Form?

A canonical form refers to entities of resources which can be determined in more

than one way, and one of them can be considered as a favorable canonical (stan-
dard) form.
The canonical form of a mathematical entity is a standard way of determining
that quantity in mathematical expression. For example, the canonical form of a posi-
tive integer in decimal form is a number sequence which is not started from zero. It
is a class of entity in which an equivalence relation is defined. For example, a Row
Echelon Form (REF) and Jordan Normal Form are typical canonical forms for
matrices interpretation in Linear Algebra.
There are many methods to represent canonical form of the same entity in com-
puter science. For instance: (1) computer algebra represents mathematical objects
and (2) path concept in a hierarchical file system, where a single file can be refer-
enced in several ways.
102 5 Meaning Representation

5.6.4.2 Canonical Form in Meaning Representation

Canonical form of meaning representation in NLP refers to the phenomena of a

single sentence/utterance can be assigned with multiple meanings leading to same
meaning representation. For example:
[5.21] Jack eats KitKat.
[5.22] KitKat, Jack likes to eat.
[5.23] What Jack eats is KitKat?
[5.24] It’s KitKat at that Jack eats.
All these sentences/utterances have similar meanings with minor variations in
tones and thematic issues.
FOPC, semantic net, conceptional dependency diagram, and frame-based repre-
sentation are good elaborations of how canonical form performs and store such
representations in a knowledge base.

5.6.4.3 Canonical Forms: Pros and Cons

Advantages

1. Simplify reasoning and storage operations.

2. Needless to generate inference rules for all different variations with same
meaning.

Disadvantages

Nevertheless, it may complicate semantic analysis for sentences/utterances with

similar meanings, but each has variance in phonemes or high-level semantic mean-
ings like examples [5.21–5.24].

5.7 Inference

5.7.1 What Is Inference?

Inference (Blackburn and Bos 2005) is divided into deduction and induction with
origin dated back to Ancient Greece from Aristotle 300s BCE. Deduction refers to
use available information to guess or draw conclusion about facts such as legendary
Sherlock Holmes’ deductive reasoning methods (Doyle 2019). Examples of infer-
ence by deduction reasoning:
5.8 Fillmore’s Theory of Universal Cases 103

[5.25] Jack is a pilot; he travels a lot.

[5.26] Jane’s hair is totally soaked; it might be raining outside.
[5.27] Mary has been very busy at work and may not be able to come for gathering
this evening.
Induction is inference from evidence to a universal conclusion. An important fact
is that the conclusions may be correct or incorrect.
Examples of inference by inductive reasoning:
[5.28] The sun rose in the morning every day for the past 30 years. The sun rises
every day (in human history).
[5.29] The first two kids I met at my new school were kind to me. The students at this
school are kind.
[5.30] Our teacher allows us to pick a piece of object out of a box. The first four
students got candies. The box must be full of candies.
An inference is valid in general if it is conformed to sound evidence(s) and the
conclusion follows logically from related premises.

5.7.2 Example of Inferencing with FOPC

Inferencing with FOPC is to come up with valid conclusions which leaned on inputs
meaning representation and knowledge base. For example:
[5.31] Does Jack eat KitKat?
It consists of two FOPC statements:

Thing KitKat (5.2)

Eat Jack,x Thing x (5.3)

Given the above two FOPC statements are true, it can infer the saying [5.31] as
yes by using inductive or deductive reasoning.

5.8 Fillmore’s Theory of Universal Cases

Case grammar (Fillmore 2020; Mazarweh 2010) is a linguistic system which

focuses on the association between number of subjects, objects, or valence, etc. of a
verb and grammatic context used in linguistic analysis. This theory is proposed by
American linguistic Prof. Charles J. Fillmore (1929–2014) in his remarkable work
The Case for Case in semantic analysis published in 1968, also known as Fillmore’s
104 5 Meaning Representation

Theory of Universal Cases (Fillmore 1968). He believed that there are only a
restricted number of semantic roles, called case roles appeared in every sentence/
utterance being constructed with the verb.

5.8.1 What Is Fillmore’s Theory of Universal Cases?

The Fillmore’s Theory of Universal Cases (Fillmore 2020; Mazarweh 2010) ana-
lyzes fundamental syntactic structure of sentences/utterances by exploring the asso-
ciation of semantic roles such as: agent, benefactor, location, object, or instrument,
etc. which are required by the verb in sentence/utterance. For instance, the verb pay
consists of semantic roles such as agent (A), beneficiary (B), and object (O) for
sentence construction. For example:
[5.32] Jane (A) pays cash (O) to Jack (B).
According to Fillmore’s Case Theory, each verb needs a certain number of case
roles to form a case-frame. Thus, case-frame determines the vital aspects of seman-
tic valency of verbs, adjectives, and nouns. Case-frames are conformed to certain
limitations such as a particular case role can appear only once per sentence. There
are mandatory and optional cases. Mandatory cases cannot be deleted; otherwise,
it will produce ungrammatical sentences. For example:
[5.33] This form is used to provide you.
This sentence/utterance makes no sense without an additional role that explain
provide you to or with what matter or notion. One possible solution is:
[5.34] This form is used to provide you with the necessary information.
The association between nouns and their structures contains both syntactic and
semantic importance. The syntactic positional relationship between forms in a sen-
tence varies from language to language, so grammarians can observe, examine
semantic values in these nouns, and provide information to consider case role in a
specific language.
One of the major tasks of semantic analysis in Fillmore’s Theory is to offer a
possible mapping between syntactic constituents of a parsed clause and their seman-
tic roles associated with the verb. The term case role is widely used for purely
semantic relations, including theta and thematic roles. The theta role (θ-role) refers
to a formal device for representing syntactic argument structure required syntacti-
cally by a particular verb. For instance:
[5.35] Jack gives the toy to Ben.
Statement [5.35] shows the verb give has three arguments, whereas Jack is deter-
mined as the external theta role of agent, toy is determined as the theme role, and to
Ben is determined as the goal role.
5.8 Fillmore’s Theory of Universal Cases 105

Thematic role, also called semantic role, refers to case role that a noun phase
(NP) may deploy with respect to action or state used by the main verb. For example:
[5.36] Jack gets a prize.
Statement [5.36] shows Jack is the agent as he is doer to get, the prize is the
object being received, so it is a patient.

5.8.2 Major Case Roles in Fillmore’s Theory

There are six major roles in Fillmore’s Theory:

1. Agent—doer of action, attributes intention.
2. Experiencer—doer of action without intention.
3. Theme—thing that undergoes change or being acted upon with.
4. Instrument—tool being used to perform the action.
5. Beneficiary—person or thing for which the action being acted on or performed to.
6. To/At/From Loc/Poss/Time—to possess thing(s), place, location, or time.
For examples:
[5.37] Jack cut the meat with a knife.
[5.38] The meat was cut by Jack.
[5.39] The meat was cut with a knife.
[5.40] A knife cut the meat.
[5.41] The meat is cut.
[5.42] Jack lent Jane the CD.
[5.43] Jack lent the CD to Jane.
These examples can conclude that:
1. Agent—Jack is the doer revealed in [5.37, 5.38, 5.42, and 5.43] that performs
the action.
2. Theme—meat and CD are things (objects) being acted upon or undergoing
change as revealed in [5.38–5.43] accordingly.
3. Instrument—knife is the tool to complete an action as revealed in [5.37, 5.39,
and 5.40].
4. To-Poss—Jane is the one that possesses the CD as revealed in [5.42 and 5.43]
driven by Jack the giver.
Syntactic choices intuition is largely a reflection of underlying semantic relation-
ships which means that identical meanings can descend to articles, e.g. [5.37] can
also be presented in [5.38–5.39], or simplified versions presented in [5.40–5.41].
Syntax can have several syntactic options but are related to same meanings in
semantic meanings. Semantic analysis is a major task to offer a suitable linkage
between constituent of a parsed clause and associated semantic roles related to the
main verb.
106 5 Meaning Representation

5.8.3 Complications in Case Roles

There are four types of complications in case roles analysis:

1. Syntactic constituents’ ability to indicate semantic roles in several cases, e.g.
subject position: agent vs. instrument vs. theme
[5.44] Jack cut the fish.
[5.45] The knife cut the fish.
[5.46] The fish is cut.
2. Syntactic expression options availability, e.g. agent and theme in different con-
figurations such as:
[5.47] Jack cut the fish.
[5.48] It was the fish that Jack cut.
[5.49] The fish was cut by Jack.
3. Prepositional ambiguity not always introduces the same role, e.g. proposition by
may indicate either agent or instrument such as:
[5.50] The meat was cut by Jack.
[5.51] The meat was cut by a knife.
4. Role options in a sentence:
[5.52] Jack cut the fish with a knife.
[5.53] The fish was cut by Jack.
[5.54] The fish was cut with a knife.
[5.55] A knife cut the fish.
[5.56] The fish was cut.
It seems that semantic roles act like a musical conductor in an orchestra with old
syntactic constituents and left them out at times, but it is not as bad as it seems.
There are regularities to consider sets of rules which is the beauty of human lan-
guages to describe the same idea in different styles and configurations.
There are possible rules in case role, such as:

If ∃ Agent, it becomes Subject

Else if ∃ Instrument it becomes Subject
Else if ∃ Theme it becomes Subject (5.4)
Agent preposition is BY
Instrument preposition is BY if no agent, else WITH

Note that:
1. They are general rules, some verbs may have exception.
2. Every syntactic constituent can only fill-in one case at a time.
3. No case role can appear twice in the same rule.
4. Only NPs of same case role can be co-joined in the rule.
5.9 First-Order Predicate Calculus 107

5.8.3.1 Selectional Restrictions

Selectional restrictions are methods to restrict types of certain roles to be used for
semantic consideration. For instance:
1. Agents must be animate, i.e. a living thing such as a person, Jack.
2. Instruments must be inanimate objects, i.e. non-living things such as rock.
3. Themes are types that may be dependent on verbs, e.g. window relates to the
verb break.
Such constraints can be applied to the following examples to check whether they
make sense or not:
[5.57] Someone assassinated the President vs.
[5.58] The spider assassinated the fly. ☒
Nevertheless, additional rules can be deployed to state that assassinate has inten-
tional or political killing such that [5.58] may be incorrect. In fact, such method is
usually applied for semantic analysis to be discussed in Chap. 6.

5.9 First-Order Predicate Calculus

5.9.1 FOPC Representation Scheme

First-Order Predicate Calculus (FOPC) (Dijkstra and Scholten 1989; Goldrei

2005) can be used as a framework to derive semantic representation of a sentence/
utterance. Although it is imperfect, it is still the most straightforward mechanism to
interpret meanings as other alternatives are finite and complex for implementation.
In most cases, they become notational variants in which the quintessential parts are
the same regardless of variant to select.
FOPC supports:
1. Reasoning in truth conditions analysis to respond yes or no questions.
2. Variables in general cases through variable binding at responses and storage.
3. Inference to respond beyond knowledge base storage on new knowledge.
This choice is neither arbitrary, nor determined by practical application. FOPC
reflects natural languages semantics as it was designed by humans.

5.9.2 Major Elements of FOPC

FOPC consists of four major elements: (1) terms, (2) predicates, (3) connectives,
and (4) quantifiers.
108 5 Meaning Representation

1. Terms
Terms are objects names with three representations: (a) constants, (b) func-
tions, and (c) variables.
Constants refer to specific object described in sentence/utterance, e.g.
Jack, IBM.
Functions refer to concepts expressed as genitives such as brand name, loca-
tion, e.g. Brandname(Mercedes), LocationOf(KFC) can be regarded as single-
argument predicate.
Variables refer to objects without reference which object is referred to, like
variables x, y and z used in a mathematical equation x + y = z(e. g. a, b, c, x, y, z, etc.).
They are frequently used in FOPC for query and inferencing operations.
2. Predicates
Predicates (Epstein 2012) refer to a predicate notion in traditional grammar
traces back to Aristotelian logic (Parry and Hacker 1991). A predicate is regarded
as the property of a subject has or is characterized by. It can be considered as the
expression of fact to the relations that link up some fixed number of objects in a
specific domain, e.g. he talks, she cries, Jack plays football, etc. Predicates are
often represented with capital letters like Buy or Play in FOPC and combine with
object-names to form a proposition, e.g. Drive(Mercedes), Drive(Mercedes,
Jack), Drive(Mercedes, x), Drive(Mercedes, Jack, UIC, Starbucks), Drive(car, x,
org, dest), etc.
3. Connectives
Connectives refer to proposition combinations. Conjunctions (and as in
English, written as & or Λ), disjunctions (or as in English, written V), and impli-
cations (as if-then in English, written → or ⊃). Negation (as not in English, writ-
ten ¬ or ~ ) is also regarded a connective, even though it operates on a single
proposition.
4. Quantifiers
Quantifiers refer to generalizations. There are two major kinds of quantifiers:
universal (as all in English, written ∀) and existential (as some in English, writ-
ten ∃). The term first-order in FOPC means that this logic only uses quantifiers
to generalize objects, but never onto predicates.
A FOPC Context-Free Grammar (CFG) specification is shown in Fig. 5.4.

5.9.3 Predicate-Argument Structure of FOPC

The semantics of human languages usually exhibit certain predicate-argument

structure by variables, e.g. indefinites in generic cases and inferencing. It also uses
quantifiers, e.g. every, some to create FOPC flexibility for sentence structures and
partial compositional semantics e.g. sort of.
Predicate-argument structure refers to actions, events, and relations that can be
determined and represented by predicates and arguments. Languages exhibit
5.9 First-Order Predicate Calculus 109

Fig. 5.4 Context-free Formula → AtomicFormula

grammar (CFG)
specification of FOPC
Formula Connective Formula
Quantifier Variable; : : : Formula
¬ Formula
(Formula)
AtomicFormula → Predicate(Term, ...)

Term → Function(Term, ...)

Constant
Variable

Connective → ņ| Ģ →
Quantifier → 
Constant → IBM Tesla USA Jack A ...
Variable → x y z
Predicate → Drive Buy Find ...
Function → LocationOf Brandname ...

certain division-of-labor in which words/constituents are served as predicates and

arguments, e.g. predicates to manifest verb, and arguments to manifest different
cases of the verb.
Predicates are primarily verbs (V), verb phrases (VPs), prepositions, adjectives,
sentences/utterances and sometimes can be nouns and even noun phrases (NPs).
For instance:
[5.59] Helen cries.
[5.60] Helen speaks to Mary.
[5.61] Helen speaks loudly.
[5.62] Helen speaks loudly in the classroom.
Arguments are primarily nouns, nominals, and noun phrases (NPs), but can be
other constituents which rely upon the actual context of sentence/utterance. For
instance:
[5.63] Jack goes to the bank vs.
[5.64] He goes to the bank.
The following shows an FOPC formulation example:
[5.65] Jack gave a pen to Jane.

Giving Jack,Jane,Pen (5.5)

Note that the corresponding FOPC formulation (5.5) is precisely in Fillmore’s

case role theory that gives conveys a three-argument predicate: (1) Agent which is
Jack as the giver; (2) Possess which is Jane as the recipient, and (3) Theme which is
the pen as the direct object.
110 5 Meaning Representation

It can have other configurations to describe the same predicate logic for example:

Giving Jack,Pen,Jane (5.6)

Gave John,Pen,Jane (5.7)

Here are some complex cases with additional constituents:

[5.66] Jack gave Jane a pen for Susan.

Giving Jack,Jane,Pen,Susan (5.8)

[5.67] Jack gave Jane a pen for Susan on Monday.

Giving Jack,Jane,Pen,Susan,Monday (5.9)

[5.68] Jack gave Jane a pen for Susan in class on Monday.

Giving Jack,Jane,Pen,Susan,in class,Monday (5.10)

Note that all these predicates should be treated individually as their arguments
have different overall meanings.

5.9.4 Meaning Representation Problems in FOPC

A predicate that represents a verb meaning, e.g. give has the same argument num-
bers present as its syntactic categorization frame. It is still difficult to (1) determine
the correct role numbers for an event, (2) manifest facts about case role(s) associ-
ated with the event, and (3) ensure correct inference(s) is/are derived from meaning
representation.
According to above considerations, the FOPC formulation stated in Eq. (5.5) is
not as useful as it seems, it would be preferable if roles or cases are separated and
flexible when deciding the whole FOPC statement like this:

x, y Borrowing x Borrower Jack,x Borrowed y,x Borrow _ to

) 1Jaa1ne,.x 5 Isa
( y,Pen

Note: Corresponding to Fillmore’s case role theory, Borrower = Agent,

Borrowed = Theme, Borrow_to = To-Poss.
Although the notion of predicate relation becomes more complicated, it allows
more flexibility for sentence/utterance construction.
5.9 First-Order Predicate Calculus 111

It may further generalize Eq. (5.11) into the following formulation:

x, y, z Borrowing x Borrower w,x Borrowed y,x Borrow _ to z,xx (5.12)

By doing so, it can generate other complicated clauses by applying different

predicates combinations. The semantics of NPs and PPS in a sentence plug into
slots provided by the template can allow flexibility to variable arguments number
associated with an event (predicate).
This event has many roles to cement input with specific category (e.g. pen) for
categories and instances declaration. For example:

Isa MobyDick,Novel , AKO Novel,Literature (5.13)

Note: Just like Isa() to serve as predicate Is a, AKO() is a useful predicate to serve
as the meaning a kind of. In fact, FOPC materializes events so that they can be quan-
tified, related to other events and objects through a defined set of relationships, and
logical connections between closely related instances without meaning assumptions.

5.9.5 Inferencing Using FOPC

Inference is an important process in FOPC which has the capability to validate or

prove whether a proposition is true or false from a knowledge base. Modus Ponens
is a fundamental inferencing method used in FOPC.
Modus Ponens (MP) is a mode of reasoning from a hypothetical proposition. If
the antecedent is true, then the consequent should be also true. In other words, MP
is a kind of deductive reasoning in the form of: P implies Q, i.e. If P is true, then Q
must also be true. Its rule may be written in sequent notation as:

P → Q or P Q (5.14)
where P, Q, and P → Q are statements or propositions in a formal language and ├
is a metalogical symbol, meaning that Q is a syntactic consequence of P and P → Q
in a logical system. MP rule justification in a classical two-valued logic is given by
a truth table as shown in Fig. 5.5.
The following example uses a Tesla car to demonstrate how FOPC works in
logic inference. It has three statements to process:

ElectricCar Tesla
x ElectricCar x Fuel x,Electricity (5.15)
Fuel Tesla,Electricity
112 5 Meaning Representation

Fig. 5.5 Truth table of

Modus Pones in 2-valued
logic

Note: First statement says Tesla is an electric car, second statement says for all
electric cars x, if a car is an electric car, the fuel being used must be electricity.
The above predicate Electric Car (Tesla) matches the antecedent of the rule, so
based on simple MP deduction to conclude that Fuel(Tesla, Electricity) is a True
statement.
In fact, MP can be applied in Forward and Backward Reasoning modes.
Forward Reasoning (FR), also called normal mode is used in normal situation by
adding all facts into a KB to invoke all applicable implication rules to examine
clause correctness or new knowledge addition.
Backward Reasoning (BR) is MP operates in reverse mode to prove specific
proposition or called query in computer science. That is, to examine if the query
formula is true by its presence in KB, or without negative implication or facts on
return query results.

Exercises
5.1 What is meaning representation? Explain why meaning representation is
important in NLP. Give one or two live examples to support your answer.
5.2 State and explain FIVE major categories of meaning representation. For each,
give one live example to support your answer.
5.3 State and explain FOUR common types of meaning representation in NLP. For
each type, use the following sample sentence/utterance [5.70] [5.69] Jack buys
a new flat in London to illustrate how they work for meaning representation.
5.4 What are the THREE basic requirements for meaning representation. For each
requirement, give two live examples to support your answer.
5.5 What is Canonical Form? How canonical form is applied to meaning repre-
sentation. For sample sentence/utterance [5.70] [5.69] Jack buys a new flat in
London, give five variations of this sentence and work out the canonical form
in the forms of FOPC and Semantic Net.
5.6 What is Inference? Explain why inference is vital to NLP and the implementa-
tion of NLP applications such as Q&A chatbot.
5.7 What is Fillmore’s Theory of universal cases? State and explain SIX major
case roles of Fillmore’s Theory in meaning representation. Use a live example
for illustration.
5.8 What is the complication of Fillmore’s Theory in meaning representation by
using several live examples, explain how it can be solved.
References 113

5.9 What are FOUR basic components of First-Order Predicate Calculus (FOPC)?
State and explain their roles and function in FOPC formulation.
5.10 What is Modus Ponens (MP) in inferencing? In addition to MP, state and
explain other possible inferencing methods that can be applied to FOPC in
meaning representation.

References

Bender, E. M. (2013) Linguistic Fundamentals for Natural Language Processing: 100 Essentials
from Morphology and Syntax (Synthesis Lectures on Human Language Technologies). Morgan
& Claypool Publishers
Bender, E. M. and Lascarides, A. (2019) Linguistic Fundamentals for Natural Language Processing
II: 100 Essentials from Semantics and Pragmatics (Synthesis Lectures on Human Language
Technologies). Springer.
Best, W., Bryan, K. and Maxim, J. (2000) Semantic Processing: Theory and Practice. Wiley.
Blackburn, P and Bos, J. (2005) Representation and Inference for Natural Language: A First
Course in Computational Semantics (Studies in Computational Linguistics). Center for the
Study of Language and Information.
Bunt, H. (2013) Computing Meaning: Volume 4 (Text, Speech and Language Technology Book
47). Springer.
Butler, A. (2015) Linguistic Expressions and Semantic Processing: A Practical Approach. Springer.
Cui, Y., Huang, C., Lee, Raymond (2020). AI Tutor: A Computer Science Domain Knowledge
Graph-Based QA System on JADE platform. World Academy of Science, Engineering and
Technology, Open Science Index 168, International Journal of Industrial and Manufacturing
Engineering, 14(12), 543 - 553.
Dijkstra, E. W. and Scholten, C. S. (1989) Predicate Calculus and Program Semantics (Monographs
in Computer Science). Springer. Advanced Reasoning Forum.
Doyle, A. C. (2019) The Adventures of Sherlock Holmes (AmazonClassics Edition).
AmazonClassics.
Epstein, R. (2012) Predicate Logic. Advanced Reasoning Forum.
Fillmore, C. J. (1968) The Case for Case. In Bach and Harms (Ed.): Universals in Linguistic
Theory. New York: Holt, Rinehart, and Winston, 1-88.
Fillmore, C. J. (2020) Form and Meaning in Language, Volume III: Papers on Linguistic Theory
and Constructions (Volume 3). Center for the Study of Language and Information.
Goddard, C. (1998) Semantic Analysis: A Practical Introduction (Oxford Textbooks in Linguistics).
Oxford University Press.
Goldrei, D. (2005) Propositional and Predicate Calculus: A Model of Argument. Springer.
Jackson, P. C. (2019) Toward Human-Level Artificial Intelligence: Representation and Computation
of Meaning in Natural Language (Dover Books on Mathematics). Dover Publications.
Mazarweh, S. (2010) Fillmore Case Grammar: Introduction to the Theory. GRIN Verlag.
Minsky, M. (1975). A framework for representing knowledge. In P. Winston, Ed., The Psychology
of Computer Vision. New York: McGraw-Hill, pp. 211-277.
Parry, W. T. and Hacker, E. A. (1991) Aristotelian logic. Suny Press.
Potts, T. C. (1994) Structures and Categories for the Representation of Meaning. Cambridge
University Press.
Schank, R. C. (1972). Conceptual dependency: A theory of natural language processing. Cognitive
Psychology, 3, 552–631.
Sowa, J. (1991) Principles of Semantic Networks: Explorations in the Representation of Knowledge
(Morgan Kaufmann Series in Representation and Reasoning). Morgan Kaufmann Publication.
Chapter 6
Semantic Analysis

6.1 Introduction

6.1.1 What Is Semantic Analysis?

Semantic analysis (Cruse 2011; Goddard 1998; Kroeger 2019) can be considered as
the process of identifying meanings from texts and utterances by analyzing gram-
matic structures relationships between words, tokens of written texts, or verbal
communications in NLP.
Semantic analysis tools can assist organizations to extract meaningful informa-
tion from unstructured data automatically such as emails, conversations, and cus-
tomers feedbacks. There are many ways ranging from complete ad-hoc
domain-oriented techniques to some theoretical but impractical methods. It is a
sophisticated task for a machine to perform interpretation due to complexity and
subjectivity involved in human languages. Semantic analysis on natural language
captures text meaning with contexts, sentences, and grammar logical structures
(Bender and Lascarides 2019; Butler 2015).
Semantic analysis is a process to transform linguistic inputs to meaning repre-
sentation and stamina for machine learning tools like text analysis, search engines,
and chatbots. From computer science perspective, semantics can be considered as
group of words, phrases, or clauses that provide concern specific context to lan-
guage, or clues to word meanings and relationships. For instance, a successful
semantic analysis will base on quantity methods such as word frequency and con-
text on location to generate cognitive connection between the clause giant panda is
a portly folivore found in China and its semantic meaning instead of just the name
(panda) it stands for.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 115
Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_6
116 6 Semantic Analysis

6.1.2 The Importance of Semantic Analysis in NLP

Semantic analysis (Goddard 1998; Sowa 1991) is important to conscious of knowl-

edge relevance and information about meaning, e.g., giant panda characteristics,
comparisons with other panda species, evolution history, related news, and
information.
It ensures that the contents are relevant to the understanding of (1) user, (2) con-
tents, and (3) context presence in NLP. The problem with establishing relationships
between contents and context is that most data-driven technology cannot compre-
hend contextual message of the sentence (phrase or clause) it conveys. If the under-
standings on context and user’s behavior have deep semantic level, it can produce
contents relevance and resonant experience.
There are many automatic classification systems today with purely bag of words
approach to identify relevant features and determine documents meanings. Few
uses correlation and collocation to account for words that have several meanings
based on context. Nevertheless, none uses full semantic analysis for words mean-
ings. But this is very much needed to interpret a document correctness because
language, especially English language is ambiguous. English nouns have an average
of five to eight synonyms, e.g. run has more than 100 common meanings like run-
ning towards the finish line, run to a meeting, run a company, the machine is run-
ning, tears ran down her face, ran for president, run him a couple thousand dollars,
etc. If a bag of words is used as features, the software will never be able to distinct
between important facts and irrelevant information leading to imprecise classifica-
tion results and ambiguities.

6.1.3 How Human Is Good in Semantic Analysis?

Humans extract abstract ideas and notions like breathing without awareness. Use
the meaning of apple as example, when discussing about the concept of apple is
referred to fruit consume regularly. But now, a majority is referred to brand name
Apple that dominates mobile phones and computers industry. In other words,
humans are competent to extract context surrounding words, phrases, objects, sce-
narios and compare information with prior experience, common sense, and world
knowledge to construct overall meanings in a text or conversation. These analyses
outputs will be used to predict outcome with incredible accuracy, but algorithms and
computers capacity upgrades had modified habitual practices to fit in with machine
learning and NLP allowing machine-driven semantic analysis becomes reality. Such
machine learning based semantic analysis schemes can help to reveal the meanings
in online messages and conversations, determine answers to questions without man-
ually extracting relevant information from large volumes of unstructured data. The
truth is semantic analysis aims to make sense of everything from words to languages
in daily life.
6.2 Lexical Vs Compositional Semantic Analysis 117

6.2 Lexical Vs Compositional Semantic Analysis

6.2.1 What Is Lexical Semantic Analysis?

Lexical semantic analysis (Cruse 1986) is a subfield of linguistic semantics to study

word’s compositionality, grammar, structure mechanisms, and the relationships
between word senses and their usages.
The analytical unit in lexical semantics is called lexical unit, which includes not
only words, but also partial words, affixes (subunits), compound words, and phrases,
collectively referred to lexical terms. Lexical units are catalogue of words called
lexicon of a language. Lexical semantics can be interpreted as the relationship
between lexical terms, sentence/utterance syntax and its meaning.
Lexical semantics analyzes the meaning of lexical items that correlate with lan-
guage or syntax structure. They classify and decompose lexical terms and tokens;
examine the similarities and differences in lexical semantic structure across lan-
guages; review correlation between sentence’s lexical and syntactic meaning with
its semantic meaning.
Lexical relation in lexical semantic involves the analysis of meaning or words
relevance in lexical level that include homonymy, polysemy, metonymy, synonyms,
antonyms, hyponymy, and hypernymy to be studied in word sense and relation
section.

6.2.2 What Is Compositional Semantic Analysis?

Compositionality is a notion in philosophy of language: If the meaning of each

complex expression E in the sentence/utterance depends not only on the meaning of
a single word, but also on syntactic structure and the arrangement of different words,
then sentence/utterance can be considered as compositional in linguistic
perspective.
The meaning of a sentence/utterance in a compositional language relies only on
the meaning of words that construct sentence/utterance and how those group of
words are syntactically linkup to each other. Thus, composition semantics is to
investigate the meaning of sentence/utterance with its syntactic structure instead of
individual word. The logic behind is words conduct the overall sentence/utterance
meaning but not the combination of their own meaning. For example:
[6.1] Andrew likes Jane => likes (Andrew, Jane) vs
[6.2] Jane likes Andrew => likes (Jane, Andrew)
Although individual meaning of every single word in these sentences/utterances
is the same but due to different words arrangement, their meanings and predicate
logics can be different.
118 6 Semantic Analysis

Compositional semantics is to study the meaning of complex language units such

as sentences, paragraphs, or documents. It is vital to transform the information rep-
resented by language units into a formal representation which consists of (1) sym-
bolic and (2) vectorial representations.
Symbolic representations are meanings expressed as a logical formula by infer-
ential mechanisms, or graph-based representations expressed by graphical
transformation.
Vectorial representation are methods based on distributional semantics such as
word embeddings to represent meaning as word vectors in multi-dimensional space.
At present, only vectorial representations are applied in large-scale as it is diffi-
cult to guarantee large sets of logical propositions consistency based on textual
input to problematic inferential mechanisms. There is neither a consensus on suit-
able graph-based representations such as semantic nets to express linguistic entities
meaning, nor proper operations apply to these representations.

6.3 Word Senses and Relations

6.3.1 What Is Word Sense?

Word sense is a crucial concept to interpret meaning of words in linguistics. For

instance, there are over 20 different word senses of the word bank in a dictionary,
each has a distinct notion based on context and syntactic structure of the word being
used, such as:
1. Financial organizations that accept deposits and use funds for lending opera-
tions (Noun).
[6.3] Jack goes to the bank and withdraws some money.
2. Inventory or stock that keeps for emergencies (Noun).
[6.4] Jack goes to the food bank to acquire some food.
3. A container with an opening on top to store money (Noun).
[6.5] His coin bank was empty.
4. A sloping land besides a slope or body of water (Noun).
[6.6] Jack stands beside the bank of a river (Noun).
5. A long plie or ridge (Noun).
[6.7] Jack digs a bank of earth.
6. Enclose with a bank (Verb).
[6.8] bank roads
7. Cover with ashes to control the flames (Verb).
[6.9] Bank a fire
8. Tip laterally (Verb).
[6.10] The pilot had to bank the aircraft.
6.3 Word Senses and Relations 119

9. A fighter maneuvers the aircraft to tip laterally (Noun).

[6.11] F19 fighter went into a steep bank.
10. Similar Objects Are Arranged in a Row (Noun).
[6.12] He operated a bank of switches.

6.3.2 Types of Lexical Semantics

There are six types of commonly used lexical semantics: (1) homonymy, (2) poly-
semy, (3) metonymy, (4) synonyms, (5) antonyms, (6) hyponymy and hypernymy.

6.3.2.1 Homonymy

Homophones are words that are spelled and pronounced the same but have different
meanings. The word homonym comes from prefix homo- stands for same and
suffix -nym stands for name.
Example 1: bank1: financial institution vs bank2: slopping land:
[6.13] He went to the bank and withdrew some cash.
[6.14] He was standing at the bank of the lake in the forest.
Example 2: bat1: a sporting club for ball hitting vs bat2: a kind of flying mammal:
[6.15] He handles his bat skillfully during the game.
[6.16] Bats live the longest as compared with other species of similar size.
Example 3: play1: light-hearted recreational activity for amusement vs
play2: the activity of doing something in an agreed succession.
[6.17] This Shakespeare play is excellent.
[6.18] It is still my play.
There are two related concepts with homonymy: (1) homographs are usually
defined as words that have the same spelling with different pronunciations and (2)
homophones are words that share same pronunciation regardless of spellings as
examples above. Further, homographs are words with same spellings and hetero-
graphs are words that share same pronunciation but different spellings, e.g. chart vs
chat, peace vs piece, right vs write, etc.
Homonymy often causes problems in the following NLP applications:
1. information retrieval confusion e.g., cat scan.
2. machine translation confuses foreign languages’ meanings:
e.g. bank1—Financial institution, bank (English) → la banque (French)
[6.19] He goes to the bank and withdraws some cash. (English)
[6.20] Il va à la banque et retire de l’argent. (French)
e.g. bank2—sloping land, bank (English) → la rive (French)
120 6 Semantic Analysis

[6.21] He lived by the bank of the lake. (English)

[6.22] Il habitait au bord du lac. (French)
3. text-to-speech confusion
e.g., bass (string instrument) vs bass (fish)

6.3.2.2 Polysemy

Polysemy are words with same spellings but different in meanings and context. The
difference between homonymy and polysemy is delicate and subjective.
e.g., bank
[6.23] The bank was built in 1866. (financial building)
[6.24] He withdrew some money from the bank early this morning. (financial
organization)
In fact, many commonly words are polysemy with multiple contexts and mean-
ings in different sentence situations.
e.g., get is a commonly used word that has at least three distinct meanings.
[6.25] I get an apple from the basket. (have something)
[6.26] I get it. (understand)
[6.27] She gets thinner. (reach or cause to a specified state or condition)

6.3.2.3 Metonymy

Metonymy is a kind of figure-of-speech in which one word or phrase is replaced by

another association.
It is also a rhetorical strategy to describe the periphery of nucleus indirectly, as
in describing someone’s outfit to individual’s characteristics. It is regarded as a sys-
tematic relationship between senses, or systematic polysemy, e.g. college, hospital,
and museum can all stand for building with semantic relationship between that
building and an institution.
Metonymy and metaphor have fundamental differences in functions. Metonymy
is about referring a method of designation or component identification or symbolic
linkage with association, e.g., crown for monarchy or royalty. Metaphor is about
understanding and interpretation in contract. It is a means to understand or explain
a phenomenon by another description. For instance:
[6.28] Her business rises like phoenix.

6.3.2.4 Zeugma Test

Zeugma is the usage of a word(s) that makes sense in one way but not the other.
Examples of zeugma that caused conflicts in semantics:
[6.29] Wage neither war nor peace.
–– There is a term to wage war but literally incorrect to say to wage peace.
6.3 Word Senses and Relations 121

[6.30] He watched the brightness of lightning and the pounding of

thunderstorm.
–– He can only watch lightning but not thunder.
The zeugma test in semantic analysis consists of using a putatively ambiguous
expression in a sentence in which several of its putative meanings are crowded
together, whether it make sense or not. Let us use the word serve as example:
[6.31] Which United Airlines flights serve dinner?
[6.32] Does Jack serve the Army?
[6.33] Do United Airlines flights serve dinner and the Army?
It showed that there are two difference senses of serve though [6.33] may
sound odd.

6.3.2.5 Synonyms

Synonyms are words with same meaning in some or all contexts. They usually
appear in language in different contexts, such as formal and informal language,
daily conversations, and business correspondence. Synonyms have modest meaning
when used, although they have the same meaning, e.g., create/make, start/begin,
big/huge, attempt/try, house/mansion, pretty/beauty, etc. Synonyms have two lex-
emes if they are interchangeable in all cases and retain the same meaning.
However, there are very few truths synonymy in the real-world situation as to
whether two words are truly synonyms. The logic behind is if they are different
words then they must mean something else or have some context differences in
usage and cannot be the same in all situations. In many cases, two words are not
exactly interchangeable when they appear, even though many aspects of the mean-
ing are the same. These words are used and mean differently due to concepts of
politeness, slang, register, and genre etc.
e.g., large vs big (are they exactly the same?)
[6.34] This building is very big vs
[6.35] This building is very large.
[6.36] Janet is her big sister vs
[6.37] Janet is her large sister.
Although both words have same meanings in the description of size, the word big
has an additional notion of older in terms of seniority description.

6.3.2.6 Antonyms

Antonyms is the word sense between words with opposite context meanings. It has
the notion in which other sense relations do not occupy synonym regardless of human
tendency to categorize experience in dichotomous contrast is not easily judged.
However, the notion of antonyms is immeasurable. Humans understand the con-
cept of opposite from childhood, encounter them in daily life, and even use
122 6 Semantic Analysis

antonyms as a kind of cognitive method to organize notions, concepts, and experi-

ences, e.g. big vs small, dark vs bright, hot vs cold, in vs out, etc. Antonyms can also
use to interpret binary, scale, or position opposition such as long vs short, fast vs
slow and up vs down, etc.

6.3.2.7 Hyponymy and Hypernymy

Hyponym is a word sense of another word if the first word sense is specific, denoting
a subclass of the other sense in linguistics, e.g. truck is a hyponym of vehicle, mango
is a hyponym of fruit, and chair is a hyponym of furniture; or conversely hypernym/
superordinate (hyper is super), e.g. vehicle is a hypernym of truck, fruit is a hyper-
nym of mango, furniture is hypernym of chair.
It is interesting to know that hyponymy is not only limited to nouns but it can
also be found in verbs, e.g. gaze, glimpse, or stare are all regarded to specific
moment of seeing.
Hyponymy and hypernymy relationship between word sense and relation are
regarded as the relationship between class and subclass concepts in object-oriented
programming from computer science perspective, e.g. the class of vehicle have
three subclasses: car, lorry, and bus, while the class of fruit can have numerous
subclasses such as apple, orange, and mango; or in reverse manner, the concept
vehicle is the superclass of car, and the concept fruit is the superclass of mango.
Furthermore, words have hyponyms of same broader term are hypernym known
as co-hyponyms, e.g. daisy and rose and broader term of flower called hyponymy or
inclusion, which has the same situation for word sense relation of co-hypernymy.
Hyponymy has (1) extensional, (2), entailment, (3) transitive, and (4) IS-A hier-
archy characteristics:
1. Extensional is the class represented by the parent extension, including the class
represented by hyponym, e.g. the relations between vehicle and truck.
2. Entailment is a hyponym sense A of sense B if A entails B.
3. Transitive means if A entails B and B entails C, then A entails C, e.g. truck,
vehicle, transport where truck is a hyponymy of vehicle, vehicle is a hyponymy
of transport, so truck is a hyponymy of transport.
4. IS-A hierarchy where A IS-A B (or A IsA B), and B subsumes A in object-oriented
programming (OOP).

6.3.2.8 Hyponyms and Instances

Hyponyms has notions of instance and class. In linguistics, an instance can be con-
sidered as proper noun with unique entity, e.g. New York is an instance of city; USA
is an instance of country. It is regarded as the relationship between class vs object
in object programming.
6.4 Word Sense Disambiguation 123

In short, class is the notion of things and objects, whereas object is the instance
of class, e.g. person is a class concept to describe an individual person while Jack is
an object, which is an instance of that class concept.
A simple test: the relationship between car and Tesla, are they class-object rela-
tionship or class-subclass relationship?

6.4 Word Sense Disambiguation

6.4.1 What Is Word Sense Disambiguation (WSD)?

Word sense disambiguation (WSD) (Agirre and Edmonds 2007) is a well-known

challenge in computational linguistics that involves the identification for correct
semantic meaning of words used in sentences/utterances. WSD is the ability to
determine which meaning of a word is activated when a word is used in a specific
context of NLP.
Lexical ambiguity is one of the initial problems that any NLP system may
encounter. In summary, POS tagging is applied to resolve syntactic ambiguity, while
WSD is applied to resolve semantic ambiguity. However, it is always difficult to
resolve semantic than syntactic ambiguity. Consider distinct sense for the word bass
examples:
[6.38] Jane hates to hear the bass sound.
[6.39] Jack is eating fried bass.
It has completely different word sense in which [6.38] represents a musical
instrument and [6.39] represents a type of fish. So, by using WSD the two sentences
can be interpreted like this:
[6.40] Jane hates to hear the bass/instrument sound.
[6.41] Jack is eating fried bass/fish.

6.4.2 Difficulties in Word Sense Disambiguation

There are five major concerns in WSD: (1) difference meaning across dictionaries,
(2) POS tagging, (3) inter-judge variance, (4) pragmatic (discourse), and (5) senses
discreteness.
1. Meaning across dictionaries
A problem with WSD is senses decision as dictionaries and thesauri offer
several words divisions into senses. Many WSD research are commonly used
WordNet (WordNet 2022a) as the reference word sense corpus for English. It
124 6 Semantic Analysis

can be considered as a comprehensive lexicon that composes of word concepts

and their semantic relations with other concepts (e.g., synonyms). For example,
the concept of car is interpreted as {car, auto, automobile, machine, motorcar}.
BabelNet (2022) is a recent multilingual encyclopedic dictionary with multilin-
gual WSD.
2. POS Tagging
WSD and POS tagging involve disambiguation or tagging with words.
However, algorithms used for one tend not to work well for the other, mainly
because a word's POS is primarily determined by adjacent one to three words
versus word sense determined by more distant words in many cases. For exam-
ple, the success rate of POS tagging algorithms is around 96% versus 75% in
WSD with supervised learning current research and findings (Agirre and
Edmonds 2007).
3. Inter-judge variance
WSD system test results on a task are usually compared to ones by humans.
While it is easy to attribute POS to texts, it is difficult in training to mark word
senses. Since human performance serves as the standard, it is an upper limit for
computer performance. However, humans fared much better at coarse-grained
discrimination than at fine-grained discrimination and is the reason for research
of the former put to the test in recent WSD evaluation exercises.
4. Pragmatic (Discourse)
Pragmatic and discourse are complex problems in NLP. Many AI researchers
believe that one cannot analyze meanings of words without some form of sen-
sible ontology analysis and world knowledge at a pragmatic level. Also, common
sense is sometimes required to distinguish words such as pronouns in anaphors
or cataphors of the text.
5. Senses discreteness
The notion of word sense is sometimes unpredictable and controversial. Most
can agree on semantic interpretation at the level of coarse-grained homographs
but going down to fine-grained polysemy can lead to disagreement. For example,
Senseval-2 (Preiss 2006) uses fine-grained sensory distinctions, with only 85%
of the annotated words that can agree with. Word meanings are infinitely vari-
able, in principle dependent on context, and cannot be easily broken down into
distinct or separate sub-meanings.

6.4.3 Method for Word Sense Disambiguation

Commonly used WSD methods include: (1) knowledge base, (2) supervised learn-
ing, (3) semi-supervised, and (4) unsupervised learning methods for WSD (Agirre
and Edmonds 2007; Preiss 2006).
6.4 Word Sense Disambiguation 125

1. Knowledge-based (KB) is a method mainly based on dictionaries, thesauri, and

lexical knowledge databases. They do not need corpus evidence for disambigu-
ation. The Lesk method (Lesk 1986) is a pioneering dictionary-based method
introduced by Prof. Michael Lesk in 1986. The Lesk definition and its algorithm
aim to measure the overlap between the meaning definitions of all words in a
context. Kilgarriff and Rosenzweig (2000) simplified the Lesk definition to mea-
sure the overlap between meaning definition of a word and current context,
meaning the correctness of identifying one word at a time, the current context
being the set of words in surrounding sentence/utterance or paragraph (Ayetiran
and Agbele 2016).
2. Supervised learning (SL) methods are standard machine learning techniques that
use semantically annotated corpora to train disambiguation. These methods
assume that context alone can provide sufficient evidence to clarify meaning, so
verbal knowledge and reasoning are considered unnecessary. A context is inter-
preted as a set of word features that contains information about surrounding
words. Support vector machines (SVM) and memory-based learning are com-
monly used SL methods for WSD. However, they are usually computationally
intensive and require large manually labeled corpora to produce satisfactory
results.
3. Since many WSD problems lack training corpora, semi-supervised methods are
applied to both labeled and unlabeled data, which require only amount of anno-
tated text and a large amount of plain unannotated text as well as bootstrapping
from starting data. The bootstrapping method starts with a small amount of start-
ing data for each word, either with manually labeled training examples, or with
a small set of triggering decision rules. The seed value is intended to train an
initial classifier with some supervised method. This classifier is then used on the
unlabeled portion of corpus to extract a larger training set with the safest classi-
fication. This process is repeated to train each new classifier until the entire cor-
pus is exhausted or the maximum number of iterations is reached. Other
semi-supervised techniques use large unlabeled corpora to provide co-occur-
rence information to complement labeled corpus perspectives to help supervised
models adapt to different domains.
4. Unsupervised Learning (UL) methods assume that similar meanings appear in
similar contexts, that is why perceptions can be induced from texts by clustering
word occurrences using a similarity measure of context. This task is called word
sense induction or discrimination. UL methods can overcome knowledge acqui-
sition bottlenecks due to their independence from manual work. Although the
performance is lower than other methods abovementioned, fair comparison is
hard as the induced senses should link up to a known word senses dictionary.
126 6 Semantic Analysis

Fig. 6.1 WordNet basic

statistical information

6.5 WordNet and Online Thesauri

6.5.1 What Is WordNet?

WordNet (WordNet 2022a) is a lexical corpus of words with over 200 languages
with adjectives, adverbs, nouns, and verbs grouped into a set of synonyms where
each word in WordNet has a distinct concept. It is organized by concepts and mean-
ings against a dictionary in alphabets. Since traditional dictionaries were created by
humans, a lexical resource is required for computers effecting WordNet is applica-
ble in NLP. It is available for public access and free download with statistical infor-
mation as shown in Fig. 6.1.
WordNet’s structure is vital tool for computational linguistics and NLP imple-
mentations. It resembles a thesaurus and group words by meanings. However, they
have basic differences: (a) WordNet indicates word senses in addition to word forms.
As a result, words that are found near one another in network are semantically
related or even synonym with each other, (b) WordNet encodes semantic relations
among words, whereas words in a thesaurus does not follow a distinct pattern other
than the similarity in surface meaning.

6.5.2 What Is Synsets?

WordNet can be considered as a network of words connected by lexical and seman-

tic relations. Nouns, verbs, adjectives, and adverbs are combined into a group of
cognitive synonyms called synsets with each expresses a specific concept. Synsets
are associated with conceptual semantics and lexical relationships such as hypo-
nyms and antonyms. WordNet contains over 117,000 synsets. Each of these synset
is associated with other in a small number of conceptual relationships.
A synset contains a short definition called a gloss, and one or more short sen-
tences describing how members of synset are used in most contexts. Word forms
with many different meanings are represented in different synsets. This is the form
of each form-meaning pair in WordNet. Each synonym group is a synset within a
6.5 WordNet and Online Thesauri 127

WordNet term, and synonyms that are part of a synset are lexical variants of that
concept. Figure 6.2 shows a synset tree for synset concept book and all concept
relationships with all other related synsets. Meaningful related words and concepts
in the generated network can be browsed from WordNet browser (WordNet 2022b).

6.5.3 Knowledge Structure of WordNet

A WordNet structure is concepts of words relationship in a WordNet network to

arrange same concepts in similar interchange contexts in Fig. 6.3. These words are
unordered sets grouped into synsets and linked with small conceptual relations. An
example of synsets structure benefit arrayed synonyms profit with definitions and
examples as shown in Fig. 6.4. Benefit(profit) is defined as an advantage or profit
gain from something, e.g. He receives benefits of computers trade.

Fig. 6.2 The synsets concept of book in WordNet

128 6 Semantic Analysis

Fig. 6.3 Basic knowledge

structure of WordNet

Fig. 6.4 An example of knowledge structure of synset benefit

6.5 WordNet and Online Thesauri 129

6.5.4 What Are Major Lexical Relations Captured in WordNet?

Super-subordinate relation, also called hypernymy, hyponymy, or IS-A relation is a

frequently used relation among synsets. It links generic synsets such as {furniture,
piece_of_furniture} to sub-concepts like {chair} and {armchair}. Thus, WordNet
indicates that synset furniture consists of synset chair, which in turn includes synset
armchair; conversely, synsets like chair or armchair make up the synset furniture.
In fact, the synset tree goes up to root node {entity}.
As said, such hyponym relation is transitive in nature, e.g. if an armchair is a
kind of chair and if a chair is a kind of furniture, then an armchair is a kind of fur-
niture. WordNet distinguishes between types (general nouns) and instances (spe-
cific people, countries, and geographic entities), e.g. book is a type of publication,
Abraham Lincoln is an instance of President. Instances are always denoted as leaves
(terminal nodes) in synset tree hierarchies.
Major lexical relations include:
• Synonymy: words with similar meaning
• Polysemy: words with more than single sense
• Hyponymy/Hypernymy: IS-A relation between words
• Meronymy/Holonymy: part-whole relation between words
• Antonymy: opposite meanings between words
• Troponymy: applicable for verbs, e.g. whisper is toponym of speak.
Figure 6.5 shows the major lexical relations capture in WordNet with examples.

6.5.5 Applications of WordNet and Thesauri?

WordNet and Thesauri applications include information extraction, information

retrieval, question answering, medical informatics, and machine translation.
WordNet has another common usage to determine words similarity with algo-
rithms proposed, including to measure the distance(s) among words in WordNet
synset graphs (trees), e.g. counting the number of edges among synsets. Intuitive
words or synonyms are close to meaning. Many WordNet-based world similarity
algorithms are implemented in a Perl package called WordNet::Similarity and a
Python package using NLTK and SpaCy will be explored in the second part of
NLP workshops.
130 6 Semantic Analysis

Fig. 6.5 Major lexical relations captured in WordNet with examples

6.6 Other Online Thesauri: MeSH

6.6.1 What Is MeSH?

Medical Subject Thesaurus, aka. MeSH (MeSH 2022) is a hierarchically organized

vocabulary for indexing, cataloging, and searching biomedical and health-related
information created by U.S. National Library of Medicine (NLM). MeSH contains
subject headings that appear in MEDLINE/PubMed, NLM catalog, and other NLM
databases. It consists of 177,000 entries, 26,142 biomedical titles and continues to
soar as the literature expands. 2020 edition contains more than 25,000 subject head-
ings, 4,400 approximately more since its launch in 1960. These headings are orga-
nized into an 11-level hierarchy with 83 subheadings. MeSH can be freely used via
US NLM’s online MeSH browser (MeSH 2022). MeSH headings are organized in
a knowledge tree with 16 major branches:
6.8 Introduction 131

A. Anatomy, B. Organisms, C. Diseases, D. Chemicals and Drugs,

E. Analytical Diagnostics and Therapeutic Techniques and Equipment,
F. Psychiatry and Psychology, G. Phenomena and Processes, H. Disciplines
and Occupations, I. Anthropology, Education, Sociology and Social
Phenomena, J. Technology, Industry, Agriculture, K. Humanities,
L. Information Science, M. Named Groups, N. Health Care, V. Publication
Characteristics and Z. Geographicals.

MeSH glossary contains several entry terms intended to be synonyms for canoni-
cal title terms in addition to hierarchical set of canonical terms.

6.6.2 Uses of the MeSH Ontology

MeSH ontology usage includes:

1. synonyms as entry terms, e.g. sucrose and saccharose
2. hypernyms from hierarchy, e.g. sucrose is a glycosyl glycoside
3. index in MEDLINE/PubMED databases such as bibliographic database NLM
contains 20 million journal articles with 10–20 MeSH terms manually assigned
to each article.

6.7 Word Similarity and Thesaurus Methods

6.8 Introduction

A synonym can be considered as a binary relationship between two synonyms or

non-synonyms. Similarity or distance is a looser measure when two words share
more semantic features with each other. Similarity is a relationship between sensa-
tions, e.g. bank is usually not similar to slope, but in some cases, they may have the
same meaning, e.g. bank1 is similar to fund3, and bank2 is similar to slope5, in which
the similarity can be calculated by word sense relationship in a sentence.
Word similarity is important because a good measure can be used in information
retrieval, question answering, machine translation, natural language generation, lan-
guage modeling, automatic paper scoring, and even plagiarism detection.
132 6 Semantic Analysis

The difference between word similarity and word relation is that similar words
are almost synonyms, e.g. car, bicycle are similar in concept but not a kind of Is-A
relation, whereas related words can be related in any way, e.g. car, gasoline are
highly related but not similar in semantic meaning.
There are two types of similarity algorithms: (1) Thesaurus-based algorithms
and (2) Distributional algorithms. Thesaurus-based algorithms are designed to
examine adjacent words in a hypernym hierarchy with similar annotations or defini-
tions. Distribution algorithms are designed to examine words with similar distribu-
tional contexts.

6.8.1 Path-based Similarity

Path-based similarity aims to examine two concepts in general. Two concepts are
similar if they are in the vicinity of thesaurus hierarchy. Synset tree (graph), the
distance (path) between two synset nodes can provide a good indication of semantic
similarity between two concepts. This evaluation method is known as path-based
similarity measurement. Figure 6.6 depicts an example of path-based similarity for
concept car. Note that all concepts have path value 1 point to themselves.

12
object

artifact
10
instrumentality article
7
transport ware

5 vehicle tableware

5
wheeled vehicle cutlery

2
automotive motorcyle fork

3
car truck minibike

Fig. 6.6 Path-based similarity for concept related to car

6.8 Introduction 133

For example:
pathlen(car, car) = 1
pathlen(car, automotive) = 2
pathlen(car, truck) = 3
pathlen(car, minibike) = 5
pathlen(car, transport) = 5
pathlen(car, artifact) = 7
pathlen(care, tableware) = 10
pathlen(car, fork) = 12
In general:

pathlen ( c1 ,c2 ) = 1 + nos.of edges in the shortest path at hypernym graph

between sense nodes c1 and c 2. (6.1)

where pathlen(c1, c2) ranges from 0 to 1.

The path-based similarity simpath(c1, c2) of two nodes (concepts) is given by

1
simpath ( c1 , c2 ) = (6.2)
pathlen ( c1 , c2 )

wordsim ( w1 , w2 ) = max ( simpath ( c1 , c2 ) )

∀c1 ∈ senses ( w1 ) , c2 ∈ senses ( w2 ) (6.3)

Using car concept as example:

simpath(car, car) = 1/1 = 1.0
simpath(car, automotive) = 1/2 = 0.50
simpath(car, truck) = 1/3 = 0.33
simpath(car, minibike) = 1/5 = 0.20
simpath(car, transport) = 1/5 = 0.20
simpath(car, artifact) = 1/7 = 0.14
simpath(car, tableware) = 1/10 = 0.10
pathlen(car, fork) = 1/12 = 0.08

6.8.2 Problems with Path-based Similarity

Let us assume every link denotes a uniform distance. It seems that car to minibike
is closer than car to transport because higher synsets are more abstract in synset
tree, e.g. object is abstract than artifact, transport is abstract than vehicle, etc.
Despite simpath(car, minibike) and simpath(car, transport) have identical values,
their semantic relationship between each other is different, naturally synsets in
other branch of the synset tree are less related in concept, e.g. car vs tableware or
even fork.
134 6 Semantic Analysis

Thus, it is suggested to have a metric that can represent the cost of each edge
independently, so that words associated with abstract nodes should have less simi-
larity scores.

6.8.3 Information Content Similarity

The Information Content Similarity metric uses information content to assess

semantic similarity in taxonomy which was first proposed by Prof. Philip Resnik,
whose distinguished work Using information content to evaluate sematic similarity
in taxonomy published in 1995 (Resnik 1995).
Let us define P(c) as the probability of a random word in corpus for an instance
of concept c. There is a unique random variable ranging from words formally asso-
ciated with each concept in the hierarchy. For a given concept, each observed noun
is either a member of the concept with probability P(c) or is not a member of the
concept with probability 1-P(c). All words are members of the root node entity, i.e.
P(root)=1, lower nodes in the hierarchy have lower probability.
Information content similarity is generally determined by counting against the
corpus. When applying to car concept example, each instance of car counts toward
frequency of automotive, wheeled vehicle, vehicle, etc. So given word(c) is the col-
lection of all words that are children of node c, the probability of information con-
tent similarity P(c) in a corpus is given by Eq. (6.4):

∑ w∈words ( c )
count ( w )
P (c) = (6.4)
N

Thus, (1) words(transport) = {transport, wheeled vehicle, automotive, car, truck,

motorcycle, minibike} and (2) words(automotive) = {car, truck}.
A synset tree of car associated with P(c) up to transport level in each corpus is
shown in Fig. 6.7.
Information content (IC) is given by

IC ( c ) = − log P ( c ) (6.5)

where the lowest common subsume (LCS) is given by

LCS ( c1 , c2 ) = the lowest common subsumer (6.6)

i.e. the lower node in hierarchy that subsumes (is a hypernym of) both c1 and c2 is
ready to apply information content as a similarity metric.
6.8 Introduction 135

transport 0.415

vehicle 0.225

wheeled vehicle 0.102

0.0172 automotive motorcyle 0.00249

car truck minibike

0.00872 0.00117 0.000537

Fig. 6.7 Synset tree of “car” with associated P(c) (up to transport level in the corpus)

6.8.4 The Resnik Method

Resnik Method (Resnik 1995, 1999) refers to the similarity between two words that
are in the vicinity of their common information. It is defined to measure the most
informative common information contents, i.e. (lowest) subsumer (MIS/LCS) of
two nodes, given by

sim resnik ( c1 , c2 ) = − log P ( LCS ( C1 , C2 ) ) (6.7)

6.8.5 The Dekang Lin Method

Dekang Lin method was proposed by Prof. Dekang Lin with his work Information-
Theoretic Definition of Similarity at ICML in 1998 (Lin 1998). It determines the
similarity between concepts A and B is not only what they have in common but also
the differences between them. It concerns with (1) commonality and (2) difference.
Commonality, denoted by IC(common(A,B)) means A and B are more in common
that has more similarity. Difference, denoted by IC(description(A,B) −
IC(common(A,B)) means more differences between A and B that has less similarity.
Similarity theorem is similarity between A and B measured by the ratio between
amount of information required to state commonality of A and B, the information
required to describe what A and B are given by

simLin ( A,B ) – log P ( common ( A, B ) ) / log P ( description ( A, B ) ) (6.8)

136 6 Semantic Analysis

He further modified Resnik method demonstrating that information in common

is twice the LCS information content given by:

2 x log P ( LCS ( c1 , c2 ) )
SimLin ( c1 , c2 ) = (6.9)
log P ( c1 ) + log P ( c2 )

Using car concept as example:

2x log P ( wheeled vehicle )

SimLin ( car, minibike ) =
log P ( car ) + log P ( minibike )
2 x log P ( 0.102 )
= = 0.372
log P ( 0.00872 ) + log P ( 0.000537 )

2x log P ( automotive )
SimLin ( car, truck ) =
log P ( car ) + log P ( truck )

2 x log P ( 0.0172 )
= = 0.707
log P ( 0.00872 ) + log P ( 0.00117 )

This calculation showed that car is related to truck than minibike at hierarchy
tree in Fig. 6.10.

6.8.6 The (Extended) Lesk Algorithm

The (extended) Lesk Algorithm uses a thesaurus-based to measure glosses which

contain similar words for concept similarity. For instance, drawing paper is a type
of paper for drafting, including the art of transferring designs from specially pre-
pared paper to a glass, wood, or even metal surface.
For all n-word phrases which appear in two glosses:
1. Add a score of n2
2. Paper and specially prepared for 1 + 22 = 5
3. Evaluate the overlaps for other relations which define glosses of hypernyms and
hyponyms
The extended Lesk for similarity (simeLesk) is given by

sim eLesk ( c1 , c2 ) = ∑ (
overlap gloss ( r ( c1 ) ) , gloss ( q ( c2 ) ) ) (6.10)
r , q∈RELS
6.9 Distributed Similarity 137

6.9 Distributed Similarity

6.9.1 Distributional Models of Meaning

Distributional models of meaning can be considered as a kind of vector space mod-

els of meaning. Prof. Zellig Harris (1909–1992) claimed that oculist and eye-doc-
tor… occur in almost the same environments… (Harris 1954) which means A & B
is synonym if A and B have almost identical environments. Sir John R Firth
(1890–1960) also stated that you shall know a word by the company it keeps!
(Firth 1957):
[6.41] A bottle of Baileys is on the table.
[6.42] Many coffee drinkers like Baileys.
[6.43] Baileys will make you drunk.
[6.44] We make Baileys out of Irish whiskey and cream.
Humans can guess Baileys from context words is an alcoholic coffee beverage
flavored with cream and Irish whiskey. This means that two words are semantically
similar if they are similar in context of the word being used for algorithm
interpretation.

6.9.2 Word Vectors

Word vector is a vector of weights. In a simple 1-of-N encoding every element in the
vector is associated with a word in vocabulary. Word encoding is vector where the
corresponding element is set to one, and other elements are zero.
Given a target word w, assume there is a binary feature fi for each N words in
lexicon vi, the word vector is given by:

W = ( f1 , f 2 , f 3 , … f N ) (6.11)

Apply to above Baileys’ example, if w = Baileys, f1 = coffee, f2 = whiskey, f3 =

beer, f4 = cream,…

w = (1, 1, 0, 1, …) (6.12)

6.9.3 Term-Document Matrix

Text data is denoted as a matrix in this method. The rows represent sentences from
the data to be analyzed and columns represent words of the matrix. Each cell is the
counting of term t in a document d:tft,d, and each document is a counter vector in ℕͮ .
138 6 Semantic Analysis

Figure 6.8 shows a term-document matrix to investigate the relationships of four

important words: battle, soldier, fool, and trick from six famous literatures: As you
like it, Henry V, Julius Caesar, and Twelfth Night extracted from The Complete
Works of Shakespeare by William Shakespeare (1564–1616) (Shakespeare 2021),
The Adventures of Sherlock Holmes (Doyle 2019), and Moby Dick by Herman
Melville (1819–1891) (Melville 2012).
It showed that:
1. two documents Julius Caesar and Henry V are similar if their term-document
vectors are similar as in Fig. 6.9.
2. each word is a count vector in ℕD as a row, Fig. 6.10 shows row vector for the
word fool across these six documents.
3. two words are semantically similar if their word vectors are similar, e.g. fool and
trick. It makes sense because they are related to each other semantically as com-
pared with battle and soldier, as shown in Fig. 6.11.

Fig. 6.8 Term-document matrix of 6 famous English literature

Fig. 6.9 Term-document matrix comparison by document vectors

Fig. 6.10 Illustration of count vector for six documents domain

6.9 Distributed Similarity 139

Fig. 6.11 Sample of two similar word by vector comparison across six documents

A term-context matrix can be formed using smaller context, e.g. a set of 10 suc-
cessive words from a paragraph or search engine. A word is now defined by a vec-
tor over the number of context words, which can be an entire document, literature,
or a list of words in a search engine, etc.
There is an argument as to whether raw counts can be used. tf-idf (term frequency
and inverse document frequency) are commonly used in place of raw term counts
for term-document matrix whereas Positive Point-wise Mutual Information (PPMI)
method for term-context matrix, respectively.

6.9.4 Point-wise Mutual Information (PMI)

Point-wise Mutual Information (PMI) is to evaluate whether events x and y co-

occur more than if they are independent given by

P ( x,y )
PMI ( X ,Y ) = log 2 (6.13)
P ( x )( y )

For word similarity measurement application, Church and Hanks (1990) pro-
posed PMI between two words given by

P ( word1 , word 2 )
PMI ( word1 , word 2 ) = log 2 (6.14)
P ( word1 ) P ( word 2 )

Niwa and Nitta (1994) proposed Positive PMI (or PPMI) by replacing all PMI
values less than zero into zero values, which is now commonly used in PMI calcula-
tions for document similarity comparison.
140 6 Semantic Analysis

6.9.5 Example of Computing PPMI on a Term-Context Matrix

Given matrix F with C columns (contexts), W rows (words) and fij is the number of
times wi occurs in context cj, Positive PMI(PPMI) between word1 and word2 is
given by:

 P ( word1 , word 2 ) 
PPMI ( word1 , word 2 ) = max  log 2 , 0  (6.15)
 P ( word1 ) P ( word 2 ) 

where:

 PMI (W , C ) , if PMI (W , C ) > 0

 (6.16)
 0, if PMI (W , C ) < 0

∑
C
∑ f
W
f ij fij
p (W , C ) = , p (Wi ) p (C ) =
j =1 ij
= , j
i =1
(6.17)
∑ ∑
W C
f N N
i =1 j =1 ij

in which: p(W, C) is the probability of considering target word W and context word
C together. p(W) and p(C) is the probability of occurring target word W and context
word C, if they are independent fij is the number of times Wi occurs in context Cj.
Let us use the previous document term matrix of six English literatures as exam-
ple to calculate word and context of total counts and probabilities as shown in
Figs. 6.12 and 6.13.

P (W = fool, C = As You Like It ) = 37 / 225 = 0.164

P (W = fool ) = 111 / 225 = 0.493

Fig. 6.12 Term-context matrix of six contexts with word and context total counts
6.9 Distributed Similarity 141

Fig. 6.13 Term-context matrix of six contexts with word and context total probabilities

Fig. 6.14 Term-context matrix of six contexts with PPMI values

P ( C = As You Like It ) = 41 / 225 = 0.182

Let us calculate PMI score for the word fool co-occurred with context from C1 =
As You Like It based on the above information from Fig. 6.13.
p (W , C )
Using PMI (W , C ) = log
p (W ) p ( C )
0.164
PMI ( fool, C1) = log = 0.604 (6.18)
0.493 ∗ 0.182

Similarly, the rest of PMI values for this term-context matrix are calculated as
follows in Fig. 6.14:
PMI (W , C ) , if PMI (W , C ) > 0 
Note that: PPMI (W , C ) =   from (6.16)
 0, if PMI (W , C ) < 0 

6.9.6 Weighing PMI Techniques

It is noted that PMI is biased toward infrequent events from above matrix such as
rare words have high PMI values. There are two possible methods to improve PMI
values: (1) apply add-k smoothing, e.g. add-1 Smoothing and (2) assign rare words
with higher probabilities.
142 6 Semantic Analysis

6.9.7 K-Smoothing in PMI Computation

Since PMI is usually biased with infrequent events, K-smoothing method can be
solution. For example, apply add 2 Smoothing (i.e., set k = 2) in every cell of co-
occurrence matrix as in Fig. 6.15 and see how it works.
The corresponding probabilities matrix after Add-2 Smoothing is shown in
Fig. 6.16.
The Term-context matrix with PPMI values after applying Add-2 Smoothing is
shown in Fig. 6.17.

Fig. 6.15 Term-context matrix of six contexts with word and context total count with Add-2
Smoothing

Fig. 6.16 Term-context matrix of six contexts with word and context total prob. with Add-2
Smoothing

Fig. 6.17 Term-context matrix of six contexts with PPMI values with Add-2 smoothing
6.9 Distributed Similarity 143

It may have certain improvement in PPMI values giving the rate context words
theoretically.
However, there were not many improvements in this case.
Another method to achieve is by raising context probabilities to a certain factor
α, say 0.8.

 P ( w, c ) 
PPMIα ( w, c ) = max  log , 0  (6.19)
 P ( w ) Pα ( c ) 
count ( c )
α

where: Pα ( c ) =
∑ count ( c )
α
c

For example: say P(a) = 0.95 and P(b) = 0.05:

0.950.8 0.050.8
Pα ( a ) = = 0 . 913, Pα ( b ) = = 0.083. (6.20)
0.950.8 + 0.050.8 0.950.8 + 0.050.8

Results using α = 0.8 and 0.9 are shown in Figs. 6.18 and 6.19, respectively.

Fig. 6.18 Term-context matrix of six contexts with PPMI values with α = 0.80

Fig. 6.19 Term-context matrix of six contexts with PPMI values with α = 0.90
144 6 Semantic Analysis

6.9.8 Context and Word Similarity Measurement

When applying context and world similarity measurement against context and word
vector, remember that cosine for computing similarity is given by

 
∑
N
  v ·w v wi
cos ( v , w ) =   = i =1 i
(6.21)
v w
∑ ∑
N N
v2
i =1 i
w2
i =1 i

where vi is PPMI value for word v in context i; wi is PPMI value for word w in con-
text I, cos(v,w) is the cosine similarity of v and w.
Context and word similarity measurement of six literatures is shown in Fig. 6.20.
For context comparison, cosine similarity measurement is performed between
C1 As You Like It and other five literatures, in which cosine (C1, C2) have the high-
est 0.453 as compared with others ranging from 0.044 (C3:Julius Caesar) and 0.157
(C6:Moby Dick). It showed that it makes sense as the context of As You Like It has
theme similarity with Twelfth Night than other literatures.
For word comparison, comparison is performed at W4: trick with three other
words across six literatures, in which cosine W4:trick, W3:fool have the highest
similarities among other two words W1:battle and W2:Solder which in fact they are
related in meanings and English usage.
It also showed other possible similarity measurements include Jaccard, Dice,
and JSs methods given by

∑ min ( vi , wi )
N
 
sim Jaccard ( v , w ) = i =1
(6.22)
∑ max ( vi , wi )
N
i =1

2 x ∑ i =1 min ( vi , wi )
N
 
sim Dice ( v , w ) = (6.23)
∑ i =1 ( vi + wi )
N

v + w

 v + w   
sim JS ( v w ) = D  v |  + Dw | 
 2   2 

Fig. 6.20 Context and Word Similarity from six sample literature
6.9 Distributed Similarity 145

6.9.9 Evaluating Similarity

Like N-grams, similarity methods have (1) intrinsic and (2) extrinsic evaluation
schemes. Intrinsic evaluation refers to the correlation between similarity scores of
algorithms and human words. Extrinsic evaluation, also called task-based or end-to-
end evaluation, refers to detect misspellings, word sense disambiguation (WSD),
and use in grading essays or TOEFL multiple choice vocabulary tests.

Exercises
6.1 What is semantic analysis? State and explain the importance of semantic anal-
ysis in NLP. Give 2 live examples for illustration.
6.2 State and explain how humans are good in semantic analysis. Give 2 daily life
examples to support your answers.
6.3 What is the difference between lexical vs compositional semantic analysis?
Each of them gives 2 examples to support your answers.
6.4 What is word sense in linguistic? State and explain any 5 basic types of lexical
semantics and their word senses. For each of them, give 2 examples for
illustration.
6.5 What is Zeugma is linguistic and why is important in NLP? Give 2 live exam-
ple to illustrate how Zeugma Test for testing semantic correctness of sen-
tences/utterances.
6.6 What are the major concerns and difficulties encountered in word sense dis-
ambiguation (WSD). Give one example for each concern to support your
answers.
6.7 State and explain FOUR major methods to tackle word sense disambiguation
(WSD). Which one(s) is(are) commonly used in NLP application nowadays to
tackle WSD? Why?
6.8 What is Synsets in WordNet framework? Give 2 examples on how it works to
support your answers.
6.9 What is Path-based Similarity in Semantic Analysis? Use book as the basic
synset to construct a synset tree like Fig. 6.9 and calculate all the related Path-
based Similarity between different concepts related to book.
6.10 Based on the synset tree created in question 6.9, calculate the similarity values
by using: (1) Resnik Method and (2) Dekang Lin Method and compare them
with the ones calculated in 6.9.
6.11 What is distributed similarity? State and explain methods used for distributed
similarity measurement.
6.12 Use four famous literatures: (1) Moby Dick (Melville 2012), (2) Little Women
by Louisa Mary Alcott (1832–1888) (Alcott 2017), (3) The Adventures of
Sherlock Holmes (Doyle 2019), and (4) War and Peace by Leo Tolstoy
(1828–1910) (Tolstoy 2019) as context documents, and select ANY 4 words
(wisely) to illustrate how term-context matrix, PMI and PPMI for document
and word similarity measurement in semantic analysis.
146 6 Semantic Analysis

6.13 Repeat question 6.12 by using K-smoothing method for PMI/PPMI calcula-
tions (with k = 1 and 2) and different values of α and compare them with
results found in 6.12. Explain why it can/cannot be improved.

References

Agirre, E. and Edmonds, P. (Eds) (2007) Word Sense Disambiguation: Algorithms and Applications
(Text, Speech and Language Technology Book 33). Springer.
Alcott, L. M. (2017) Little Women. AmazonClassics.
Ayetiran, E. F., & Agbele, K. (2016). An optimized Lesk-based algorithm for word sense disam-
biguation. Open Computer Science, 8(1), 165-172.
BabelNet. 2022. BabelNet official site. https://babelnet.org/. Accessed 25 July 2022.
Bender, E. M. and Lascarides, A. (2019) Linguistic Fundamentals for Natural Language Processing
II: 100 Essentials from Semantics and Pragmatics (Synthesis Lectures on Human Language
Technologies). Springer.
Butler, A. (2015) Linguistic Expressions and Semantic Processing: A Practical Approach. Springer.
Church, K. W., & Hanks, P. (1990). Word association norms, mutual information, and lexicog-
raphy. Computational Linguistics - Association for Computational Linguistics, 16(1), 22-29.
Cruse, A. (2011) Meaning in Language: An Introduction to Semantics and Pragmatics (Oxford
Textbooks in Linguistics). Oxford University Press
Cruse, A. (1986) Lexical Semantics (Cambridge Textbooks in Linguistics). Cambridge
University Press.
Doyle, A. C. (2019) The Adventures of Sherlock Holmes (AmazonClassics Edition).
AmazonClassics.
Firth, J. R. (1957). Papers in Linguistics 1934-1951. Oxford University Press.
Goddard, C. (1998) Semantic Analysis: A Practical Introduction (Oxford Textbooks in Linguistics).
Oxford University Press.
Harris, Z. S. (1954). Distributional structure. Word (Worcester), 10(2-3), 146-162. doi: https://doi.
org/10.1080/00437956.1954.11659520.
Kilgarriff, A. and Rosenzweig, J. (2000). Framework and results for English
SENSEVAL. Computers and the Humanities, 34(1/2), 15-48.
Kroeger, P. (2019) Analyzing Meaning: An Introduction to Semantics and Pragmatics (Textbooks
in Language Sciences). Freie Universität
Lesk, M. 1986. Automatic Sense Disambiguation Using Machine Readable Dictionaries: How
to Tell a Pine Cone from an Ice Cream Cone. ACM Special Interest Group for Design of
Communication: Proceedings of the 5th Annual International Conference on Systems
Documentation. ACM; 24–26. https://doi.org/10.1145/318723.318728.
Lin, D. K. (1998) An Information-Theoretic Definition of Similarity. In Proceedings of the
Fifteenth International Conference on Machine Learning (ICML '98). Morgan Kaufmann
Publishers Inc., 296–304.
Melville, H. 2012. Moby-dick. Penguin English Library.
MESH. 2022. MeSH browser official site. https://www.nim.nih.gov/mesh/meshome.html.
Accessed 25 July 2022.
Niwa, Y. and Nitta. Y. 1994. Co-occurrence Vectors from Corpora vs. Distance Vectors from
Dictionaries. In COLING 1994 Volume 1: The 15th International Conference on Computational
Linguistics, Kyoto, Japan. https://aclanthology.org/C94-1049.pdf.
Preiss, J. (2006). A detailed comparison of WSD systems: An analysis of the system answers for
the SENSEVAL-2 English all words task. Natural Language Engineering, 12(3), 209-228.
References 147

Resnik, P. (1995). Using Information Content to Evaluate Semantic Similarity in a Taxonomy.

Cornell University Library. https://arxiv.org/abs/cmp-lg/9511007.
Resnik, P. (1999). Semantic similarity in a taxonomy: An information-based measure and its appli-
cation to problems of ambiguity in natural language. JAIR 11, 95-130.
Shakespeare, W. (2021) The Complete Works of Shakespeare (AmazonClassics Edition).
AmazonClassics.
Sowa, J. (1991) Principles of Semantic Networks: Explorations in the Representation of Knowledge
(Morgan Kaufmann Series in Representation and Reasoning). Morgan Kaufmann Publication.
Tolstoy, L. (2019) War and Peace. AmazonClassics.
WordNet. 2022a. WordNet official site. https://wordnet.princeton.edu/. Accessed 25 July 2022.
WordNet. 2022b. WordNet browser official site: http://wordnetweb.princeton.edu/parl/webwn.
Accessed 25 July 2022.
Chapter 7
Pragmatic Analysis and Discourse

7.1 Introduction

Pragmatics and discourse analysis (Bender and Lascarides 2019; Cruse 2011;
Goddard 1998; Kroeger 2019) refer to the study of language in context meaning of
sentences/utterances unlike word layers, syntax, grammatic relationship, semantic
and meaning presentations learnt in previous chapters.
Pragmatics analysis focuses on context meaning. Discourse analysis studies
social context in written and spoken language. They consist of structured, coherent,
and cohesive sets of sentences/utterances to reflect what constitutes an utterance
versus a set of unrelated sentences and how the text is related.
There are two types of discourse in daily life: (1) monologue and (2) dialogue. A
monologue is a one way communication between a speaker (writer) and an audi-
ence (reader), e.g. read or write a book, watch a TV show or a play, listen to a
speech, attend a presentation, or a lecture depends on the deposition of dialogue.
Dialogue refers to participation in turn to speaker and hearer. It has a two-way or
multiple ways of communications.
There are also two types of dialogue: (1) human-to-human, e.g. daily conversa-
tions, group discussions, and (2) (a) human-to-computer interaction (HCI), e.g.
conversational agent, chatbot in NLP, and (b) computer-to-computer interaction
(CCI), e.g. cross machines verbal communication in smart city and intelligent trans-
portation system, multi-agent based bargain and negotiation systems, etc.

7.2 Discourse Phenomena

There are many discourse phenomena solved by humans naturally, but some like
conference resolution required a lot of effort by machine to solve.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 149
Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_7
150 7 Pragmatic Analysis and Discourse

7.2.1 Coreference Resolution

Coreference resolution (Bender and Lascarides 2019; Goddard 1998) is a task to

identify all linguistics expressions, also known as mentions which correspond to
real-word entity described in the text. These mentions are assembled and replaced
by correct pronouns and noun phrases. It is simple by humans but always mistaken
by machines. For example:
[7.1] Jack saw Andrew in the examination hall. He looked nervous.
[7.2] Jack saw the student in the examination hall. He looked nervous.
Human and machine will likely consider the first subject mentioned in foregoing
sentence/utterance as reference to pronoun of the following sentence. For instance,
He in [7.1] will refer to Jack. However, coreference resolution by human perspec-
tive in [7.2] will consider He may not refer to Jack but the student as it is natural and
logical to relate student with examination.
Example below is more obvious:
[7.3] Jane talked to Amy about her examination result. She looked worry.
[7.4] Jane talked to Amy about her examination result. She felt sorry about it.
She in [7.3] should refer probably to Amy is worried as she participated the exam-
ination instead of Jane.
She in [7.4] should probably refer to Jane instead of Amy participated the exami-
nation but Jane is more likely to feel sorry as empathy to Amy.
Humans can discern the above naturally by context and common sense or world
knowledge but confound computers to develop judgement.

7.2.2 Why Is it Important?

Let us look at some standard situations prior to complex coreference resolu-

tion cases:
[7.5] Jack gives Ian 1,000 dollars. He is generous. (original sentence)
[7.6] Jack gives Ian 1,000 dollars. Jack is generous. (with coreference resolution)
or compact cases handled by computer satisfactorily:
[7.7] “I voted for Jack as he is more aligned to my values”, Ian said (original
sentence)
[7.8] “Ian voted for Jack as Jack is more aligned to Ian’s values”, Ian said (with
coreference resolution)
From The Adventures of Sherlock Holmes (Doyle 2019):
[7.9] I was seized with a keen desire to see Holmes again, and to know how was
employing his extraordinary powers. (original sentence)
7.2 Discourse Phenomena 151

[7.10] Watson was seized with a keen desire to see Holmes again, and to know how
Holmes was employing Holmes’ extraordinary powers (with coreference
resolution)
or more challenging sentences of famous discourse from A Scandal in Bohemia:
[7.11] To Sherlock Holmes, she is always “ the woman”. I have seldom heard him
mention her under any other name. (original sentence)
[7.12] To Sherlock Holmes, Irene Adler is always “the woman”. Watson has seldom
heard Holmes mention Irene Adler under any other name. (with coreference
resolution)
[7.11] is more challenging as the reference name Irene Adler for she did not
occur, but after two sentences, there is no emotion akin to affection for Irene Adler.
This phenomenon is called cataphor to acquire meaning from a subsequent word
or phrase in linguistics.
The subsequent phrase (or word group) is called antecedent or a referent against
anaphora, a rhetorical term for a phrase (or word group) repetition at the start of
consecutive sentences/utterances used in many English sentences’ construction, i.e.
[7.5], [7.7], [7.9] are reference terms mentioned repetitively prior pronouns
replacement.
Coreference resolution is a versatile tool applied in many NLP applications
including text understanding and analysis, information retrieval and extraction, text
summarization, machine translation, and even sentiment analysis. It is a great way
to obtain unambiguous sentences comprehensible by computers.

7.2.3 Coherence and Coreference

7.2.3.1 What Is Coherence?

In linguistics, coherence (Bender and Lascarides 2019; Goddard 1998) refers to

meaning relationships between individual units, which can be sentences (discourses)
or textual statements. Texts appear to have logical and semantical consistency for
reader or hearer due to these relations.
Coherence-oriented text analysis is primarily concerned with the construction
and configuration of meaning in a text, that is, how various components are con-
nected to make the text meaningful to recipient as a random sequence of disjointed
phrases and clauses.
In other words, if a text has coherence, its parts are well-connected and head for
the same direction. Without coherence, a discussion or utterance may neither make
sense nor follow by audience. It has both verbal and written languages
significance.
152 7 Pragmatic Analysis and Discourse

Here are some coherence examples:

[7.13] History reveals that humans have come a long way from birth. They have
invented many new technologies that improve the standard of living. However,
technologies that are supposed to provide us a better world are sometimes end
up to disaster, such as the invention of nuclear weapons, environmental pollu-
tion, and the extinction of some animal species.
In [7.13], coherence terms History→ humans→ They→ technologies→ nuclear
weapons with repetitive terms and concepts provide a stream of ideas flow and
knowledge for hearer or reader to understand the message conveyed in this passage.

7.2.3.2 What Is Coreference?

Coreference (co-reference) appears when two (or group of) terms refer to the same
person or thing with a unified reference to achieve linguistic coherence. For example:
[7.14] Jack said Helen would arrive soon, and she did.
– Helen and she refer to the same person.
Conference is not always trivial to determine, e.g.
[7.15] Jack said he would join the term vs
[7.16] Jack told Ian to come, he smiled.
When comparing [7.15] vs [7.16], [7.15] is trivial as there is only a subject
(noun) that he can refer to (i.e. Jack), while he in [7.16] can refer to either Jack or Ian.
Determining coreference expressions are important in many NLP applications
such as information retrieval and extraction, text summarization and dialogues
understanding in Q&A chatbot systems.

7.2.4 Importance of Coreference Relations

To understand the meaning of a coreference relationship, let us look at how to

extract key information or summarize the following text:
[7.17] XYZ bank is continuing to struggle with severe financial problems. According
to finance news report, their CEO Charles Smith will announce to step-down at
the press conference tomorrow morning.
The texts in [7.17] are coherent with well-structured coreference in a typical
news article. Coherence concept terms are also used to extract information:
[XYZ bank] → [financial problem] → [CEO] → [Charles Smith] → [step down] →
[press conference] → [tomorrow morning]
7.2 Discourse Phenomena 153

A reasonable text summary may be:

[7.18] The CEO of XYZ bank Charles Smith will announce his step-down at tomor-
row morning’s press conference.
This example shows the coherent relationships between text segments, where the
first sentence provides context weights of the second sentence.
Remarks: A well-structured text summarization/information extraction case will
and should match with Fillmore’s Case Roles Theory with well-defined: agent,
patient, location, time, purpose, beneficiary, possessor, instrument etc., in other
words, a well coherence text message and utterance regards the first sentence as the
opening of a speech followed by elaboration of an open statement in coreference
relation with a thematic relation like watching a movie or a TV show.
Further to elaboration and thematic relation, coreference relation has another
type called inference-type. It regards first sentence/utterance as claims followed by
explanation of claims sentence. For inference argument, the first sentence is the
effect followed by cause(s) of the following sentences:
[7.19] Jack keeps Ian’s car key. He was drunk last night. (coherence) vs
[7.20] Jack keeps Ian’s car key. He wants to see movie tonight. (without coherence)
Coherence occurred in [7.19] as the first statement has relevance to the second
statement with pragmatic meaning, whereas the second statement is probably an
explanation, or a cause of the event where Jack keeps Ian’s car key because Ian was
drunk by common sense/world knowledge. Thus, He should be Ian instead of Jack
by inference.
While two statements in [7.20] have neither coherence nor logic cause–effect
relationship between them, it is difficult to judge whether He in the second state-
ment should refer to Jack or Ian. Thus, Jack regards as the subject and the referent
He in usage of English although it may be incorrect.

7.2.5 Entity-Based Coherence

Let’s look at the following examples:

[7.21] Helen went to the superstore to buy a cello.
[7.22] She had frequented the store for a long time.
[7.23] She was delighted to buy the cello finally.
[7.24] She just discovered that the store is closed.
[7.25] It was the store Helen had frequented for a long time.
[7.26] She was delighted to buy that cello.
[7.27] The music generated by it is beautiful.
[7.28] It was closed when Helen arrived.
154 7 Pragmatic Analysis and Discourse

Entity-based coherence models measure coherence to track salient central enti-

ties across utterances. Centralization theory (Grosz et al. 1995) is a remarkable
entity-based coherence theory for tracking whether entities (the so-called Central
Entity, CE) are prominent at each point in a discourse model. For examples from
[7.21] to [7.23] are Helen who will be the reference for she in these statements natu-
rally. While CE in [7.25] is shifted from the superstore to cello in [7.26] and [7.27],
CE is shifted back to the store in [7.28] to make it more complex.

7.3 Discourse Segmentation

7.3.1 What Is Discourse Segmentation?

Discourse segmentation is the task of determining the smallest non-overlapping

discourse units, known as elementary discourse units (EDUs), which can be further
categorized into (1) sentence segmentation and (2) sentence-level discourse
segmentation.
The main purpose of discourse segmentation is to divide a text document (set of
utterances) into a list of subtopics. This is often a higher-level simplification struc-
ture of a discourse. For example, an academic article is usually segmented into
abstract, introduction, methodology, implementation, results, discussion, conclu-
sion, etc. to comprehend.
There are (1) unsupervised and (2) supervised discourse segmentation methods.
The applications of automatic discourse segmentation include (1) information
extraction or retrieval and (2) text summarization on each segment separately.

7.3.2 Unsupervised Discourse Segmentation

Unsupervised discourse segmentation is a class usually presented as a linear seg-

mentation of raw data and segmentation into multiple paragraph subtopics.
Unsupervised means that the task is not given training data as examples to under-
stand linear segmentation task. These examples involve splitting the text into multi-
paragraph units to represent paragraphs of the original text. These algorithms rely
on cohesion, which can be defined as the linguistic means of linking units of text
together.
Cohesion-based approach involves dividing text into subtopics, where sentences
or paragraphs cohere to each other and reveal the relationship between two or more
words in two units like synonyms.
Cohesion is the linking of text units based on linguistic means. Lexical cohesion
is the use of similar words to link units of text with the same word, synonym or
hypernym. For instance:
7.3 Discourse Segmentation 155

[7.29] Yesterday was Jane’s birthday. Betty and Mary went to buy a present from the
gift shop. Mary intended to buy a purse. “Don’t do that.”, mentioned Betty. “Jane
already got one. She will ask you to return it.”
Non-lexical cohesion approach using anaphora.
[7.30] Peel, core and slice peaches and pineapples, then place these fruits in the
skillet.
Unsupervised discourse segmentation was proposed by Prof. Marti Hearst in his
classical works on TextTiling in early 1990.

7.3.3 Hearst’s TextTiling Method

Hearst’s TextTiling (Hearst 1997) is a typical discourse segmentation algorithm to

subdivide explanatory text into multiple paragraphs or automatically grouped sub-
topic segments representing in the original text.
Hearst’s TextTiling method is a typical unsupervised method that no training
dataset and prior knowledge base are required. Hearst’s original work used articles
from Stargazers, a science magazine with TextTiling method to characterize article
text messages into subtopics.
For example, consider a 21-paragraph science news article extracted from the
magazine with a topic focused on reports of life on Earth and other plants, its con-
tents are characterized into the following subtopic discussions (Hearst 1997):
[Para 1–3] Introduction – the search of life in space
[Para 4–5] The moon’s chemical composition
[Para 6–8] How early earth-moon proximity shaped the moon
[Para 9–12] How the moon helped life evolve on earth
[Para 13] Improbability of the earth-moon system
[Para 14–16] Binary/trinary star systems make life unlikely
[Para 17–18] The low probability of nonbinary/trinary systems
[Para 19–20] Properties of earth’s sun that facilitate life
[Para 21] Summary
TextTiling is a technique to divide a full-length text document into coherent
multi-paragraph units that correspond to a series of subtopic paragraphs as shown in
example above. The algorithm assumes that during a subtopic discussion, a set of
words is used, and subtopics changed significant parts of vocabulary accordingly.
The distribution of terms extracted from Stargazer text is assigned with a single-
digit frequency for each sentence number, with spaces for zero frequencies (Hearst
1997) as shown in Fig. 7.1. It revealed that terms:
1. occurred frequently throughout the text such as moon and planet are often indic-
ative of main topic(s) of the text,
156

Fig. 7.1 Distribution of selected terms in Stargazer text (blanks mean zero frequency)
7 Pragmatic Analysis and Discourse
7.3 Discourse Segmentation 157

2. less common but evenly distributed, such as scientists and form are both generic
to create a subtopic title,
3. like space and star occurred more frequent from sentences 5 to 20 and 60 to 90,
while term life to planet occurred more frequent from sentences 58 to 78 which
may create two distinct clusters of subtopic discussion, and
4. like life to species have similar phenomena occurred to create a natural cluster
between sentences 35 and 55 and conform with human judgement as subtopic
discussion of How the moon helped life evolve on earth.
These results suggested that the logic behind sentences or paragraphs in subtop-
ics are consistent with each other but not with paragraphs in adjacent topics.

7.3.4 TextTiling Algorithm

TextTiling algorithm (Hearst 1997) for discourse segmentation and subtopic struc-
ture characterization using term repetition consists of three processes: (1) tokeniza-
tion, (2) lexical score determination, and (3) boundary identification.
Tokenization includes converting words to lowercase, removing stop-words and
root-words, and converting words into pseudo-sentences with the same length such
as 15 words.
Lexical score determination includes calculating lexical cohesion scores for each
gap between pseudo-sentences. This lexical cohesion score represents words simi-
larity. For instance, take 10 pseudo-sentences each before and after gap, followed
by the computation of cosine similarity between word vectors which is given by
 
∑ b ×a
N
 b ⋅a
sim cosine ( )
b ,a =   =
b a
i =1 i i
(7.1)
∑ b ∑
N N
i =1 i
2
a2
i =1 i

Boundary identification involves assigning a boundary distance to identify a new

segment. Similarity is first created, and the depth value of similarity valley (a-b) +
(c-b) is calculated as shown in Fig. 7.2, then performed segmentation if the depth
score value is greater than the threshold as shown in Fig. 7.3.

Fig. 7.2 Lexical score determination with similarity valleys

158 7 Pragmatic Analysis and Discourse

Fig. 7.3 Boundary identification with discourse segments

7.3.5 Supervised Discourse Segmentation

It is relatively easy to collect bounded training data using supervised discourse seg-
mentation such as news reports from TV shows, paragraph segmentation in text or
dialogue to find paragraphs in speech recognition output.
Several classifiers can be used to achieve supervised segmentation, one is called
feature set which is a superset for unsupervised segmentation with often domain-
specific utterance tokens and keywords.
Supervised discourse segmentation is also a model. It is (1) a classification task
that uses one of the supervised classifier methods, such as SVM, Naïve Bayer, maxi-
mum entropy, etc. to distinguish whether sentence boundaries have paragraph
boundaries, or (2) a sequence labeling task to label sentences with or without para-
graph borders. It uses cohesive features including word overlap, word cosine simi-
larity, anaphora, and additional features such as discourse markers or keywords.
Discourse tokens or keywords/phrases indicate discourse structure, e.g. good
evening, join our broadcast news now, or join the company at the beginning/end of
the segment, etc. They can be manual codes or automatically determined by feature
selection.
However, measuring precision, recall, and F-measure are not always good evalu-
ation ideas as they are insensitive to near misses. Pevzner and Hearst (2002) pro-
posed a good and effective evaluation metric for text segmentation called
WindowDiff method.

7.4 Discourse Coherence

7.4.1 What Makes a Text Coherent?

A text coherent refers to the application of:

1. A coherent relationship between a subfield of discourse called rhetorical struc-
ture and a whole theory called Rhetorical Structure Theory (RST). It is a text
organization theory that describes the relationships exist between parts of a text.
7.4 Discourse Coherence 159

It was proposed by Mann and Thompson (1988) in their remarkable paper

Rhetorical structure theory: toward a functional theory of text organization,
published in 1988. The theory was developed as part of research on computer-
aided text generation in text summarization and applications used by NLP
researchers.
2. The ordering of subsections of discourse called discourse topic structure. It is
the key to discourse cohesion and embodies the essence of discourse analysis. It
has been extensively adopted by past decades and become a key component in
text analysis. Linearly segmenting text into appropriate topic structures can
reveal valuable information such as the overall topic structure of the text, which
can be used for text analysis tasks such as text summarization, information
retrieval, and discourse analysis.
3. A Referring Expression (RE) is any noun phrase or a substitute for a noun phrase
whose function in spoken, signed, or written text is to single out a single person,
place, object or group of people, places, objects, etc.

7.4.2 What Is Coherence Relation?

Coherence relation refers to discourse properties that make each discourse mean-
ingful (or have appropriate meaning) in the context. It refers to common denomina-
tor to identify possible connections between utterances in a series of statements or
discourses about the same topic.
These sense relations in discourse analysis named Coherence Relations by Prof.
Jerry R. Hobbs in his works Coherence and Coreference published by Cognitive
Science in 1979 (Hobbs 1979) had further developed by other linguistics including
Sanders et al. (1992) and Kehler (2002) into a well-defined theory.
These meaning relationships, called propositional relations defined by Mann
and Thompson (1986), are encoded in text recognized by the reader trying to under-
stand the text and its components, and to see why the speaker or author added the
sentence. Coherent relationships are sometimes referred to as types of thematic
development such as the narrative of a movie or TV show involving cause-and-
effect story type in sense relations development.

7.4.3 Types of Coherence Relations

There are five major types of coherence relations (1) parallel, (2) elaboration, (3)
cause-and-effect, (4) contrast, and (5) occasion.
1. Parallel infers p(a1, a2, …) from the assertion of S0 and p(b1, b2…) from the
assertion of S1, where ai and bi are similar for all i.
[7.31] Rich man wants more power. Poor man wants more food.
160 7 Pragmatic Analysis and Discourse

They are frequently used in describing two sense relations with similar situa-
tion (meaning) but different in object, reference, and scenario.
2. Elaboration infers the same proposition P from the assertions of S0 and S1.
[7.32] Dorothy was from Kansas. She lived in the great Kansas prairies.
[7.33] Nicolas Telsa was a genius. He invented over hundreds of things in
his life.
They are frequently used in discourse construction, the successive sentences/
utterances are further elaboration of the previous one.
3. Cause-and-effect are S0 and S1 if S1 infers S0, i.e. S1 → S0
[7.34] Jack cannot afford to buy the car. He lost his job.
[7.35] Nicolas Tesla invented over hundreds of things in his life. He was
a genius.
Cause-and-effect discourse relation can refer to animate or inanimate subjects
in [7.35] which is the reverse of elaboration statement [7.33] but do not always occur.
4. Contrast in S0 and S1 if P0 and P1 infer from S0 and S1 with one pair of elements
that are contrast with each other, where other elements are similar in context.
[7.36] Hope for the best. Prepare for the worst.
[7.37] Jack is meticulous while Bob is sloppy.
Contrast coherence relations can exist within a sentence, or in successive sen-
tences/utterances. It often refers to two subjects, or events with contrast sense
relations.
5. Occasion is the alteration of state that can infer from the assertion of S0, where
final state can infer from S1, or the alteration of state can infer from the assertion
of S1, whose initial state can infer from S0.
[7.38] Jane put the books into a schoolbag, she left the classroom with Helen.
[7.39] Jack failed in the exam. He started to work hard.
State change invokes new action.

7.4.4 Hierarchical Structure of Discourse Coherence

Discourse coherence can also be revealed by the hierarchy between coherent rela-
tions. For example:
[7.40] Jack went to town to buy a toy.
[7.41] He took a bus to the shopping mall.
[7.42] He needed to buy a toy for his child.
[7.43] It is Jane’s birthday.
[7.44] He also wanted to buy some books for weekend reading.
A hierarchical structure of discourse coherence is shown in Fig. 7.4. [7.40]–
[7.44] can be organized in a hierarchy tree structure, e.g. Occasion consists of two
expressions, one is expression e1 (statement [7.40]) and the other is an explanatory
clause which in turn consists of expression e2 (statement [7.41]) and a parallel clause
7.4 Discourse Coherence 161

Occasion (e1; e2)

[7.40] (e1) Explanation (e2)

[7.41] (e2) Parallel (e3; e5)

Explanation (e3) [7.44] (e5)

[7.42] (e3) [7.43] (e4)

Fig. 7.4 Hierarchical structures in discourse coherence

which consists of two entities, one is explanatory expression e3 and the other is
expression e5 (statement [7.44]), e3 is further divided into statements [7.42] and
[7.43], respectively.

7.4.5 Types of Referring Expressions

Referring expression (RE) is a surrogate for any noun phrase or noun phrase whose
function in utterance is to identify some discrete objects. There are five frequently
used REs in discourse coherence: (1) indefinite noun phrases, (2) definite noun
phrases, (3) pronouns, (4) demonstratives, and (5) names.
1. Indefinite noun phrases introduce entities into context that are new to listener,
e.g., a policeman, some apples, a new iPad, etc.
[7.45] I go to the electronic store to buy a new notebook computer.
2. Definite noun phrases refer to entities recognizable by listener such as above-
mentioned, combination of beliefs about the world, e.g., a furry white cat, the
cat, etc.
[7.46] Don’t look at the sun directly with bare eyes, it will hurt yourself.
3. Pronouns are another form of definite designation, usually with stronger restric-
tions than standard designation, e.g., s/he, it, they, etc.
[7.47] I go to the electronic store to buy a new notebook computer. This com-
puter is rather light and fast.
4. Demonstratives are pronouns that can act alone or as determiners, e.g. this, that.
[7.48] That book seems to be very interesting and worth buying it.
5. Names are common methods to refer people, organizations, and locations.
[7.49] I bought lunch at KFC today.
162 7 Pragmatic Analysis and Discourse

7.4.6 Features for Filtering Potential Referents

There are four common features to filter potential references in discourse coher-
ence: (1) number agreement, (2) person agreement, (3) gender agreement, and (4)
binding theory constraints.
1. Number agreement are pronouns and references must agree in number, e.g., sin-
gle or plural.
[7.50] The children are playing in the park. They look happy.
2. Person agreement refers to the first, second, or third person.
[7.51] Jane and Helen got up early. They needed to take an exam this morning.
3. Gender agreement refers to male, female, or non-person, e.g. he, she, or it.
[7.52] Jack looked tired. He didn’t sleep last night.
4. Binding theory constraints refer to constraints imposed by syntactic relations
between denotative expressions and possible preceding noun phrases in the same
sentence.
[7.53] Jane purchased herself an iPad. (herself should be Jane)
[7.54] Jane purchased her an iPad. (her may not be Jane)
[7.55] She claimed that she purchased Mary a iPad. (She and she may not
be Mary)

7.4.7 Preferences in Pronoun Interpretation

There are six types of preferences in pronoun interpretation: (1) recency, (2) gram-
matical role, (3) repeated mention, (4) parallelism, (5) verb semantics, and (6)
selected restrictions.
Recency refers to entities from recent utterances:
[7.56] Tim went to see a doctor at the clinic. He felt sick. It might be influenza.
Grammatical role is to emphasize the hierarchy of entities according to gram-
matical position of the terms that represent them, e.g. subject, object, etc.
[7.57] Jane went to Starbucks to meet Jackie. She ordered a hot mocha. (She should
be Jane)
[7.58] Jane discussed with Jackie about her exam results. She felt so nervous about
it. (She should be Jackie instead of Jane)
[7.59] Jane discussed with Jackie about her exam results. She felt so sorry about it.
(She should be Jane instead of Jackie)
Repeated mention refers to mentioning about the same thing.
[7.60] Jane went to supermarket to buy some food. It turned out it was closed.
Parallelism refers to subject-to-subject or object-to-object kind of expression:
7.5 Algorithms for Coreference Resolution 163

[7.61] Mary went with Jane to Starbucks. Ian went with her to the bookstore after-
wards. (her should probably be Jane instead of Mary)
Verb semantics are verbs that seem to emphasize one of their argument positions:
[7.62] Jane warned Mary. She might fail the test.
[7.63] Jane blamed Mary. She lost the watch.
In [7.62] She should be Mary as Mary is the one being warned about failing the
test. For [7.63] She should be Jane who suffered. It is a pragmatic phenomenon
because it involves common sense by word meaning blamed to understand correct
coreference in the second statement.
Selectional restrictions refer to another semantic knowledge playing a role:
[7.64] Mary lost her iPhone in the shopping mall after carrying it the whole
afternoon.
Note that [7.64] involves high-level semantic or common sense understanding of
it can mean iPhone or shopping mall but it has been carried for the whole afternoon,
so it cannot be an unmovable object except iPhone.

7.5 Algorithms for Coreference Resolution

7.5.1 Introduction

Coreference resolution (CR) is the task of finding all linguistic expressions (called
mentions) in any text involving real-world entities. After finding these mentions and
grouping them, they can be resolved by replacing pronouns with noun phrases.
There are three fundamental algorithms for conference resolution: (1) Hobbs
algorithm, (2) Centering algorithm, and (3) Log-linear model.

7.5.2 Hobbs Algorithm

7.5.2.1 What Is Hobbs Algorithm?

Hobbs algorithm was one of the early approaches to pronoun resolution proposed
by Prof. Jerry R. Hobbs in 1978 (Hobbs 1978) and further consolidated as well-
known algorithm for coreference resolution in his remarkable work Coherence and
Coreferences published in Cognitive Science 1979 (Hobbs 1979).
He original work proposed two CR algorithms, a simple algorithm based purely
on grammar, and a complex algorithm that incorporated semantics into parsing
methods (Hobbs 1978, 1979).
164 7 Pragmatic Analysis and Discourse

Unlike other algorithms, Hobbs' algorithm does not turn to a discourse model for
parsing because its parse tree and grammar rules are the only information used in
pronoun parsing. Let us look at how it works.

7.5.2.2 Hobbs’ Algorithm

Hobbs’ algorithm assumes a parse tree where each NP node has an N type node
below it as the parent of a lexical object. It operates as follows:
1. Start with the node of noun phrase (NP) that directly dominates the pronoun.
2. Go up tree to the first NP or sentence (S) node visited, denote this node as X, and
name the path being applied to reach it as p.
3. Visit all branches under node X to the left of path p, breadth-first, from left to
right, taking any NP node found as an antecedent, there is an NP or S-node
between it and X.
4. If node X is the highest S-node in sentence, visit the surface parse trees of previ-
ous sentences in the text with the most recent first, each tree is then visited in a
left-to-right and breadth-first manner. When an NP node is encountered, it is
recommended as an antecedent. If X is not the first S-node in the set, go to step 5.
5. Climb up from node X to the first NP or S-node encountered, denote this new
node as X and name the path as p.
6. If X is an NP vertex, and if the path p to X does not pass through a nominal ver-
tex immediately dominated by X, then denote X as an antecedent.
7. Visit all branches under node X to the left of path p, breadth-first manner, from
left to right, denoting each NP node encountered as an antecedent.
8. If X is an S-node, visit all branches of node X to the right of path p from left to
right and breadth-first manner, but do not visit below any NP or S being encoun-
tered as the antecedent.
9. Return to Step 4.

7.5.2.3 Example of Using Hobbs’ Algorithm

Statement [7.65] is a classic example stated in Hobbs’ original paper (Hobbs 1978)
to demonstrate how Hobbs’ algorithm works as shown in Fig. 7.5.
[7.65] The castle in Camelot remained the residence of the king until 536 when he
moved it to London.
Example—What does it stand for?
1. Start with node NP1, step 2 climbs up to node S1.
2. Step 3 searches the left part of S1’s tree but fails to locate any eligible NP node.
3. Step 4 fails to apply.
7.5 Algorithms for Coreference Resolution 165

NP3 VP

Det N PP remained NP4 PP

Det N until NP2

the castle in NP5

the residence PP N Rel

N
536 when S1
Camelot of NP6

N VP
Det N

he moved NP1 PP
the king
it to NP

London

Fig 7.5 Parse Tree for statement [7.65]

4. Step 5 climbs up to NP2 which step 6 proposes 536 as antecedent of it.

5. The algorithm can be further improved by applying simple selectional con-
straints, such as
Date can’t move;
Places can’t move;
Large or fixed objects can’t move.
6. After NP2 is rejected, steps 7 and 8 turn up nothing, and control is returned to
step 4 which fails to apply.
7. Step 5 climbs up to S2 which step 6 fails to apply.
8. In step 7, the breadth-first search recommends that NP3 the castle is rejected by
the constraint number 3.
9. The algorithm continues to visit NP4 where it correctly recommends the resi-
dence as antecedent.
Exercise: How to check coreference resolution of he as the king?

7.5.2.4 Performance of Hobbs’ Algorithm

In the original work, Hobbs manually analyzed 100 consecutive examples from
three different texts, assuming correct parsing was available, and the algorithm was
72.7% correct (Hobbs 1978); which is quite impressive for such simple algorithm.
If the algorithm is integrated with syntactic constraints when resolving pronouns as
shown in Fig. 7.5, the performance can be even higher.
166 7 Pragmatic Analysis and Discourse

However, Hobbs’ algorithm experiences two major problems.

1. When looking for the antecedent of a pronoun within a sentence, it goes sequen-
tially further up the tree to the left of pronoun, such an error is looked for in the
previous sentence.
2. This algorithm does not assume a discourse segmentation structure and may
revert to arbitrarily far of the text to find an antecedent.
Nevertheless, as he concluded in his original paper, naïve-based approach on co-
reference resolution did provide a high baseline and works in many usual situations
in discourse analysis, and still being used as a benchmark in related CR research
nowadays (Cornish 2009; Kehler et al. 2008; Lata et al. 2022; Wolna et al. 2022).

7.5.3 Centering Algorithm

Centering Theory (CT) was proposed by Profs Barbara J. Grosz and Candace
L. Sidner in their distinguished work Attention, Intentions, and the Structure of Dis-
Course, as part of its main theory of discourse analysis (Grosz and Sidner 1986). It
is a theory of discourse structure that models the interrelationships between foci or
centers as the choice of reference terms and the perceived coherence of discourse.
The basic idea is:
1. a discourse has a focus, or center,
2. the center typically remains the same for a few sentences, then shifts to a
new object,
3. the center of a sentence is typically pronominalized,
4. once a center is determined, there is a strong inclination for subsequent pronouns
to continue referring to it.
In centering algorithm, utterances from a discourse have a backward-looking
center (Cb) and a set of forward-looking centers (Cf). The Cf set of an utterance U0
is the set of utterance units elicited by that utterance. Cf set is ranked by discourse
emphasis, the most accepted ranking is by grammatical role. The highest-ranked
element in this list is called the preferred center (Cp), which represents the highest-
ranked element among previous utterances found in the current utterance and serves
as a link between these utterances. Any sudden shifts in the topic of utterances are
reflected in changes in Cb between utterances.

7.5.3.1 What Is Centering Algorithm?

Centering algorithm (Grosz and Sidner 1986; Tetreault 2001) consists of three parts:
(1) initial settings, (2) constraints, (3) rules and algorithm.
7.5 Algorithms for Coreference Resolution 167

7.5.3.2 Part I: Initial Setting

–– Let Un, Un+1 be 2 successive utterances.

–– Backward-looking center of Un, written as Cb(Un), denotes focus after Un is
interpreted.
–– Forward-looking centers of Un, written as Cf(Un), forms ordered list of entities in
Un that can serve as Cb(Un+1).
–– Cb(Un+1) is the highest-ranking element of Cf(Un) mentioned in Un+1.
–– Order of entities in Cf(Un): in which subject > existential predicate nominal >
object > indirect object > demarcated adverbial PP.
–– Let Cp(Un+1) be the highest-ranked forward-looking center.

7.5.3.3 Part II: Constraints

For each utterance Ui (i = …m) in a discourse segment D:

–– There is precisely one Cb.
–– Every element of Cf-list for Ui must be realized in Ui.
–– The center, Cb (Ui, D) is the highest-ranked element of Cf (Ui-1, D) realized by Ui.

7.5.3.4 Part III: Rules and Algorithm

For each utterance Ui (i = …m) in a discourse segment D:

Rule 1: If some elements of Cf (Ui-1, D) is realized as a pronoun in Ui, then so is Cb
(Ui, D).
Rule 2: Transition states, defined as follows, are ordered such that the sequence of
Continue is preferred over the sequence of Retains, which are preferred over
Smooth-Shift and then Rough-Shift.
The relationship between Cb and Cp of two utterances determines coherence
between words. Centering theory ranks the coherence of adjacent utterances with
transitions determined by:
1. Cb is the same from Un-1 to Un or not
2. this entity coincides with Cp of Un or not
Figure 7.6 shows the criteria for each transition in centering algorithm.
The algorithm based on these rules and conditions is defined as follows:
1. Create all possible Cb – Cf combinations.
2. Filter these combinations by constraints and centering rules.
3. Rank remaining combinations by transitions.
168 7 Pragmatic Analysis and Discourse

Fig. 7.6 The criteria for each transition in Centering Algorithm

7.5.3.5 Example of Centering Algorithm

U1: Jane heard some beautiful music at the CD store.

U2: Jane played it to Mary.
U3: She bought it.
By applying grammatical role hierarchy to construct Cf. So, for U1 will have:
Cf(U1): {Jane, music, CD store}
Cp(U1): Jane
Cb(U1): Undefined
U2 has two pronouns: She and it. She is compatible (in syntax) with Jane, while
it is compatible with either music or CD store.
Since Jane is the highest Cf(U1) ranked member, Cb(U2) should be referred to
Jane by comparing result transitions for every possible referent of it.
If it is assumed to music, the result will be:
Cf(U2): {Jane, music, Mary}
Cp(U2): Jane
Cb(U2): Jane
Result: Continue (since Cp(U2) = Cb(U2) and Cb(U1) is undefined).
Conversely, if it is assumed to CD store, the result will be:
Cf(U2): {Jane, CD store, Mary}
Cp(U2): Jane
Cb(U2): Jane
Result: Continue (since Cp(U2) = Cb(U2) and Cb(U1) is undefined)
As both are Continue, it will be set referring to music instead of CD store.
Next, let us look at U3.
For U3, She is compatible with either Jane or Mary, while it is compatible with
music. So, if she refers to Jane, i.e., Cb(U3) = Jane, the result will be:
Cf(U3): {Mary, music}
Cp(U3): Mary
Cb(U3): Mary
Result: Smooth-Shift (since Cp(U3) = Cb(U3) but Cb(U3)≠ Cb(U2)).
7.5 Algorithms for Coreference Resolution 169

Since Continue is preferred to Smooth-shift using Rule 2, Jane should be assigned

as the referent, so Centering algorithm works in this situation.

7.5.3.6 Performance of Centering Algorithm

Clearly, centering algorithm implicitly accounts for grammatical roles, recency, and
repeated-mention preference in pronoun interpretation.
However, the grammatical role hierarchy affects emphasis indirectly because the
final conversion type specifically determines the final reference assignment.
Confusion can arise if the former lead to a high-level transformation in this case,
where a referent in a low-level grammatical role prefers a referent in a high-level
role. For instance:
U1: Jane opened a new music store in the city.
U2: Mary entered the store and looked at some CDs.
U3: She finally bought some.
In this example, common sense indicates that She in U3 should refer to Mary
instead of Jane. However, by applying Centering algorithm in this case, it will
assign she to Jane incorrectly because Cb(U2) = Jane becomes Continue while Mary
becomes a Smooth-shift. While if applying Hobbs’ algorithm, Mary will still be
assigned as the referent.
Obviously, such situation occurs usually depended on situation and thematic sce-
nario. As Prof. Marilyn A. Walker in her study A corpus-based evaluation of center-
ing and pronoun resolution (Walker 1989) compared a version of Centering to
Hobbs on 281 examples from three genres of text in 1989 with 77.6% and 81.8%
accuracy, respectively.

7.5.4 Machine Learning Method

7.5.4.1 What is Machine Learning Method?

Machine learning (ML) method is a simple supervised machine learning by either

using stochastic or AI approach. It trains classifier by using manual labelled corpus
markers: (1) positive samples are antecedents marked with each pronoun and (2)
negative (derived) samples are pairing pronouns with non-antecedent NPs.
In a typical supervised ML scenario, ML system trains on a set of features and
produces a pro-antecedent pair to predict 1 if they co-refer and 0 otherwise. A typi-
cal example by applying Log-Linear model for pronominal anaphora resolution is
introduced with the following features:
Strict number [true or false]
Compatible number [true or false]
Strict gender [true or false]
170 7 Pragmatic Analysis and Discourse

Compatible gender [true or false]

Sentence distance [0, 1, 2, 3, …] from pronoun
Hobbs’ distance [0, 1, 2, 3, …] (non-groups)
Grammatical role [subject, object, PP] (taken by potential antecedent)
Linguistic form [definite, indefinite and proper pronouns]
Example for Pronominal Anaphora Resolution:
U1: Jack saw a beautiful Mercedes GLB300 at a used car dealership.
U2: He showed it to Jim.
U3: He bought it.
A table of feature vector values for sentence U2 is shown in Fig. 7.7.

7.5.4.2 Performance of Log-Linear Model

A Log-Linear model trains on vectors and filters out pleonastic it as in it is raining.

It results in weights for each and the combination of features. Most of the time it is
rigid, harder and must decide if any two noun phrases co-refer.
New features can be added to improve model performance such as:
Anaphor edits distance
Antecedent edits distance
Alias [true or false] (based on the named entity tagger)
Appositive [true or false]
Linguistic form [proper, definite, indefinite, pronoun]

7.5.4.3 Other Advanced Machine Learning Models

Big data and AI offer advancement for current machine learning models CR research
focus on Convolutional Neural Networks (CNN) (Auliarachman and Purwarianti
2019), Recurrent Neural Networks (RNN) (Afsharizadeh et al. 2021), Long-short
Term Memory Networks (LSTM) (Li et al. 2021), Transformers and BERT Models
(Joshi et al. 2019).

Fig 7.7 Table of feature vector values for sentence U2: He showed it to Jim
7.6 Evaluation 171

7.6 Evaluation

From performance perspective, commonly used methods emphasis on coreference

chains evaluation as forming a set of facts A, B, and C are assigned with A, B, and C
classes. They consist of two data types: (1) reference/true chain is correct or true
coreference chain occurred in an entity and (2) hypothesis chain/class is assigned
with the entity by a coreference algorithm.
For instance, Precision of the system can be evaluated according to:

weighted sum of correct elments in hypothesis chain

(7.2)
Number of elements in hypothesis chain

and Recall can be evaluated according to:

Number of correct elements in hypothesis chain

(7.3)
Number of eleements in reference chain

Like previous chapters on N-gram and Semantic Analysis, CR model evaluation

can be achieved by using: (1) intrinsic (using prototype and model itself) vs (2)
extrinsic (task-based, end-to-end) evaluation schemes.

Exercises
7.1 What is pragmatic analysis and discourse in linguistics? Discuss their roles
and importance in NLP.
7.2 What is the difference between pragmatic analysis and semantic analysis in
terms of their functions and roles in NLU (Natural Language Understanding)?
7.3 What is coreference resolution in linguistics? Why it is important in NLP?
Use two live examples as illustration to support your answer.
7.4 State and explain the differences between the concept of coherence vs corefer-
ence in pragmatic analysis. Give two live examples to support your answer.
7.5 What is discourse segmentation? State and explain why it is vital to pragmatic
analysis and the implementation of NLP application such Q&A chatbot. Give
two examples to support your answer.
7.6 State and explain Hearst’s TextTiling technique on discourse segmentation.
How can it be further improved by using nowadays’ AI and machine learning
technology?
7.7 What is coherence relation? State and explain five basic types of coherence
relations. For each type, give a live example for illustration.
7.8 What is referencing expression in pragmatic analysis? State and explain five
basic types of referencing expressions. For each type, please provide a live
example for illustration.
7.9 State and explain Hobbs’ algorithm for coreference resolution. Use a sample
sentence/utterance (other than the one given in the book) to illustrate how
it works.
172 7 Pragmatic Analysis and Discourse

7.10 State and explain the pros and cons of Hobbs’ algorithms for coreference reso-
lution. Use live example(s) to support your answer.
7.11 State and explain Centering algorithm for coreference resolution. Use a sam-
ple sentence/utterance (other than the one given in the book) to illustrate how
it works.
7.12 Compare pros and cons between Hobbs’ algorithm vs Centering algorithm.
Use live example(s) to support your answer.
7.13 What is machine learning? State and explain how machine learning can be
used for coreference resolution. Use live example(s) to support your answer.
7.14 Name any three types of machine learning models for coreference resolution.
State and explain how they work.
7.15 Name any two types of evaluation method/metrics for coreference resolution
model in pragmatic analysis. State and explain how they work.

References

Afsharizadeh, M., Ebrahimpour-Komleh, H., and Bagheri, A. (2021). Automatic text summariza-
tion of COVID-19 research articles using recurrent neural networks and coreference resolution.
Frontiers in Biomedical Technologies. https://doi.org/10.18502/fbt.v7i4.5321
Auliarachman, T., & Purwarianti, A. (2019). Coreference resolution system for Indonesian text
with mention pair method and singleton exclusion using convolutional neural network. Paper
presented at the 1-5. https://doi.org/10.1109/ICAICTA.2019.8904261
Bender, E. M. and Lascarides, A. (2019) Linguistic Fundamentals for Natural Language Processing
II: 100 Essentials from Semantics and Pragmatics (Synthesis Lectures on Human Language
Technologies). Springer.
Cornish, F. (2009). Inter-sentential anaphora and coherence relations in discourse: A perfect
match. Language Sciences (Oxford), 31(5), 572-592.
Cruse, A. (2011) Meaning in Language: An Introduction to Semantics and Pragmatics (Oxford
Textbooks in Linguistics). Oxford University Press
Doyle, A. C. (2019) The Adventures of Sherlock Holmes (AmazonClassics Edition).
AmazonClassics.
Goddard, C. (1998) Semantic Analysis: A Practical Introduction (Oxford Textbooks in Linguistics).
Oxford University Press.
Grosz, B. J., Joshi, A. K., and Weinstein, S. (1995). Centering: A framework for modeling the
local coherence of discourse. Computational Linguistics - Association for Computational
Linguistics, 21(2), 203-225.
Grosz, B. J., and Sidner, C. L. (1986). Attention, intentions, and the structure of discourse.
Computational Linguistics - Association for Computational Linguistics, 12(3), 175-204.
Hearst, M. A. (1997). TextTiling: Segmenting text into multi-paragraph subtopic passages.
Computational Linguistics - Association for Computational Linguistics, 23(1), 33-64.
Hobbs, J. R. (1979) Coherence and Coreference. Cognitive Science 3, 67-90.
Hobbs, J. R. (1978) Resolving pronoun references. Lingua, 44:311–338.
Joshi, M., Levy, O., Weld, D.S., and Zettlemoyer, L. (2019) BERT for Coreference Resolution:
Baselines and Analysis. In Proc. of Empirical Methods in Natural Language Processing
(EMNLP) 2019. https://doi.org/10.48550/arXiv.1908.09091
Kehler, A. (2002) Coherence, Reference, and the Theory of Grammar. Stanford, Calif.: CSLI
Publishers.
References 173

Kehler, A., Kertz, L., Rohde, H., and Elman, J. L. (2008). Coherence and coreference revisited.
Journal of Semantics (Nijmegen), 25(1), 1-44.
Kroeger, P. (2019) Analyzing meaning: An introduction to semantics and pragmatics (Textbooks in
Language Sciences). Freie Universität Berlin.
Lata, K., Singh, P., & Dutta, K. (2022). Mention detection in coreference resolution: Survey.
Applied Intelligence (Dordrecht, Netherlands), 52(9), 9816-9860.
Li, Y., Ma, X., Zhou, X., Cheng, P., He, K. and Li, C. (2021). Knowledge enhanced LSTM for
coreference resolution on biomedical texts. Bioinformatics, 37(17), 2699-2705. https://doi.
org/10.1093/bioinformatics/btab153
Mann, W. C. and Thompson, S. A. (1988) Rhetorical Structure Theory: Toward a functional theory
of text organization. Text & Talk, 8, 243 - 281.
Mann, W. C. and Thompson S. A. (1986) Relational Propositions in Discourse. Discourse
Processes 9: 57-90.
Pevzner, L., and Hearst, M. A. (2002). A critique and improvement of an evaluation metric for
text segmentation. Computational Linguistics - Association for Computational Linguistics,
28(1), 19-36.
Sanders, T., Spooren, W. and Noordman, L.G. (1992). Toward a taxonomy of coherence relations.
Discourse Processes, 15, 1-35.
Tetreault, J. R. (2001). A corpus-based evaluation of centering and pronoun resolution.
Computational Linguistics. Association for Computational Linguistics, 27(4), 507-520.
Walker, Marilyn A. (1989). Evaluating discourse processing algorithms. In Proceedings of the 27th
Annual Meeting of the Association for Computational Linguistics, pp. 251-261.
Wolna, A., Durlik, J., and Wodniecka, Z. (2022). Pronominal anaphora resolution in pol-
ish: Investigating online sentence interpretation using eye-tracking. PloS One, 17(1),
e0262459-e0262459.
Chapter 8
Transfer Learning and Transformer
Technology

8.1 What Is Transfer Learning?

Transfer learning focuses in solving a problem from acquired knowledge and apply-
ing such knowledge to solve another related problem(s) (Pan and Yang 2009; Weiss
et al. 2016; Zhuang et al. 2020). It is like two students learn to play guitar. One has
musical knowledge and the other has not. It is natural for the one to transfer back-
ground knowledge to the learning process. Every task has its isolated datasets and
trained model in traditional ML, whereas learning a new task in TL relies on previ-
ous learned tasks to acquire knowledge with larger datasets as shown in Fig. 8.1.

Fig. 8.1 Traditional machine learning vs transfer learning

© The Author(s), under exclusive license to Springer Nature Singapore Pte 175
Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_8
176 8 Transfer Learning and Transformer Technology

8.2 Motivation of Transfer Learning

Traditional ML datasets and trained model parameters cannot reuse. They involve
enormous, rare, inaccessible, time-consuming, and costly training process in NLP
tasks and computer vision. For example, if a task is text sentiment reviews predic-
tions on laptops, there are large amounts of labeled data, target data, and training
data from these reviews.
Traditional ML can work well on correlated domains, but when there are large
amounts of target data like food reviews, the inference results will be unsatisfactory
due to domains differences. Nevertheless, these domains are correlated in some
sense to bear same domain reviews as language characteristics and terminology
expressions, which makes TL possible to apply in a high-level approach on predic-
tion task. This approach enables source domains to become a target domain and
determine its sub-domains correlations as shown in Fig. 8.2.
TL has been implemented to several machine learning applications such as image
and text sentiment classifications.

8.2.1 Categories of Transfer Learning

Domain is to be assigned with a definition by feature space X and marginal proba-

bility distribution P(X) where X = {x1, x2, x3, …, xn} ∈ X.
If a feature space X and distribution P(X) between two domains are different,
they are different domains.

Source Labels Target label

Additional
Transfer Learned Layers
knowledge
Transfer
Large amount
Classification Old Model Learning
of data/labels
Model without output
Layers

(Large Dataset) Target Data

E.g. ImageNet (E.g. Grouper Sound data)

Fig. 8.2 Transfer learning

8.2 Motivation of Transfer Learning 177

If a task is defined by a label space Y with a predictive function f(⋅), f(⋅) is repre-
sented by a conditional probability distribution given by (8.1)

f ( xi ) = P ( yi |xi ) (8.1)

If a function f(⋅) and label space Y between two tasks are different, they are dif-
ferent tasks.
Now TL can give a new representation by above definitions that have Ds as
source domain and Ts as source learning task. Dt represents target domain and Tt
represents target learning task. Given two domains are unidentical or has two differ-
ent tasks, TL aim is to improve the results P(Yt| Xt) of Dt when Ts and Ds knowledge
can be obtained.
There are two types of TL (1) heterogeneous and (2) homogeneous as shown in
Fig. 8.3.
Heterogeneous Transfer Learning When source feature space and feature space
are different which means that Yt ≠ Ys and/or Xt ≠ Xs. Under the condition of same
domain distributions, the strategy of resolution is to adjust feature space smaller and
transform it to homogeneous so that the differences between marginal or condi-
tional of source and target domains will be reduced.

Homogeneous Transfer Learning When there are conditions Xt = Xs and Yt = Ys,

the difference of two domains lies on data distributions. Three strategies are com-
monly used to tackle homogenous TL problems: (1) reduction the differences of
P(Xt) ≠ P(Xs), (2) reduction the differences of P(Yt| Xt) ≠ P(Ys| Xs), and (3) the com-
bination of strategies (1) and (2).

Fig. 8.3 Two categories of transfer learning

178 8 Transfer Learning and Transformer Technology

8.3 Solutions of Transfer Learning

There are four methods to solve problems produced by homogeneous and heteroge-
neous TL: (1) instance-based, (2) feature-based, (3) parameter-based, and (4)
relational-based.
1. Instance-based
This method reweights samples from source domains and uses them as target
domain data to bridge the gap of marginal distribution differences which works
best when conditional distributions of two tasks are equal.
2. Feature-based
This method works for both heterogeneous and homogeneous TL problems.
For homogeneous types is to bridge the gap between conditional and marginal
distributions of target and source domains. For heterogeneous types is to reduce
the differences between source and target features spaces. It has two approaches
(a) asymmetric and (b) symmetric.
(a) Asymmetric feature transformation aims to modify the source domain and
reduce the gap between source and target instances by transforming one of
the source and target domains to the other as shown in Fig. 8.4. It can be
applied when Ys and Yt are identical.
(b) Symmetric feature transformation aims to transform source and target
domains into their shared feature space, starting from the idea of discovering
meaningful structures between domains. The feature space they share is usu-
ally low-dimensional. The purpose of this approach is to reduce the marginal
distribution distance between destination and source. The difference between
symmetric and symmetric feature transformation is shown in Fig. 8.5.
3. Parameter-based
This method transfers learnt knowledge by sharing parameters common to
the models of source and target learners. It applies to the idea that two related
tasks have similarity in model structure. The trained model is transferred from
source domain to target domain with parameters. This approach has a huge
advantage because the parameters are usually trained from randomly initialized
parameters as training process can be time-consuming for models trained from
the beginning. This approach can train more than one model on the source data
and combine parameters learnt from all models to improve results of the target
learner. It is often used in deep learning applications as shown in Fig. 8.6.

Fig. 8.4 Asymmetric feature transformation

8.3 Solutions of Transfer Learning 179

Fig. 8.5 Symmetric feature transformation (left) and asymmetric feature transformation (right)

Fig. 8.6 Parameter-based methods

Fig. 8.7 Relational-based approaches: an example of learning sentence structure of food reviews
to help with movie reviews’ sentiment analysis

4. Relational-based
This method transfers learnt knowledge by sharing its learnt relations between
different samples parts of source and target domains as shown in Fig. 8.7. Food
and movie domains are a related domain example. Although the reviews texts are
different, sentence structures are similar. It aims to transfer learnt relations of
different review sentences parts from these domains to improve text sentiment
analysis results.
180 8 Transfer Learning and Transformer Technology

8.4 Recurrent Neural Network (RNN)

8.4.1 What Is RNN?

Recurrent neural network (RNN) is a class of artificial neural networks (ANNs) to

consider time series or sequential data as input and use them prior inputs to produce
current input and output (Cho et al. 2014; Sherstinsky 2020; Yin et al. 2017). RNN
has memory which means its output is influenced by prior elements of the sequence
against traditional Feedforward Neural Network (FNN) with independent inputs
and outputs as shown in Fig. 8.8.

8.4.2 Motivation of RNN

There are many learning tasks required: sequential data processing which includes
speech recognition, image captioning, and synced sequence in video classification.
Sentiment analysis and machine translation model outputs are sequences, but tasks
inputs are time or space related that cannot be modeled by traditional neural net-
works to assume that test and training data are independent.
For example, a language translation task aims to translate a phrase feel under the
weather means unwell. This phrase makes sense only when it is expressed in that
specific order. Thus, the positions of each word in sentence must be considered
when model predicts the next word.
There are five major categories of RNN architecture corresponding to different
tasks: (1) simple one to one model for image classification task, (2) one to many for
image captioning tasks, (3) many to one model for sentiment analysis tasks, (4)
many to many models for machine translation, and (5) complex many to many mod-
els for video classification tasks as shown in Fig. 8.9.

Fig. 8.8 Recurrent neural network(left) vs. feedforward neural network(right)

8.4 Recurrent Neural Network (RNN) 181

Fig. 8.9 5 major types of RNNs

Fig. 8.10 Basic architecture of RNN

8.4.3 RNN Architecture

RNN is like standard neural networks consists of input, hidden, and output layers as
shown in Fig. 8.10.
An unfolded RNN architecture is narrated by xt as the input at time step t, st
stores the values of hidden units/states at time t and ot is the output of network at
time-step t. U are weights of inputs, Ws are weights of hidden units, V is bias as
shown in Fig. 8.11.
with the activation function f, the hidden states st is calculated by equation:

st = f (Uxt + Wst −1 ) (8.2)

the output of each recurrent layer ot is calculated by equation:

ot = softmax (Vst ) (8.3)

182 8 Transfer Learning and Transformer Technology

Fig. 8.11 Unfolded RNN architecture

Fig. 8.12 A simple recurrent neural network right (left) and fully connected recurrent neural net-
work (right)

The hidden states st are considered as network memory units which consists of
hidden states from several former layers. Each layer’s output is only related to hid-
den states of the current layer. A significant difference between RNN and traditional
neural networks is that weights and bias U, W, and V are shared among layers.
There will be an output at each step of the network but unnecessary. For instance,
if inference is applied for sentiment expressed by a sentence, only an output is
required when the last word is input, and none after each word for input. The key to
RNNs is the hidden layer to capture sequence information.
For RNN feedforward process, if the number of time steps is k, then hidden unit
values and output will be computed after k + 1 time steps. For backward process,
RNN applies an algorithm called backpropagation through time (BPTT).
RNN topologies range from partly to fully recurrent. Partly recurrent is a layered
network with distinct output and input layers where recurrence is limited to the hid-
den layer. Fully recurrent neural network (FRNN) connects all neurons’ outputs to
inputs as shown in Fig. 8.12.
8.4 Recurrent Neural Network (RNN) 183

8.4.4 Long Short-Term Memory (LSTM) Network

8.4.4.1 What Is LSTM?

Long short-term memory (LSTM) network (Staudemeyer and Morris 2019; Yu

et al. 2019) is a type of RNN with special hidden layers to deal with gradient explo-
sion and disappearance problems during long sequence training process proposed
by Hochreiter and Schmidhuber (1997). LSTM has better performance with train-
ing longer sequences against Naïve RNNs.
Structure frameworks of LSTM and naïve RNN are shown in Fig. 8.13.
LSTM has two hidden layers as RNN where a memory cell in the layer is to
replace hidden node. RNN has only one transfer state ht as compared with
RNN. There are two transfer states, ct (cell state) and ht (hidden state) in
LSTM. RNN’s ht corresponds to LSTM’s ct. ct passed down information among
them, output ct is produced by adding ct-1 passed from state and values of previous
step. RNN’s ht has larger difference among nodes usually.

8.4.4.2 LSTM Architecture

xtand ht − 1 are concatenated inputs from the state of previous step to train with acti-
vations for four states as shown in Fig. 8.14.
z is input calculated by multiplying the concatenate vector with weights w and
converted into values 0-1 through activation function tanh. zf,zi,zo are calculated by
multiplying the concatenate vector with corresponding weights and converting to
values 0-1 by a sigmoid function σ to generate gate states. zf represents forget gate,
zi represents input gate, zo represents output gate. A memory cell of LSTM calcula-
tion is shown in Fig. 8.15.

Fig. 8.13 Standard RNNs (left) and LSTM (right)

184 8 Transfer Learning and Transformer Technology

Fig. 8.14 Four states of LSTM

Fig. 8.15 Calculations in memory cell of LSTM

Memory cells ct,ht,yt are calculated by gate states as equations below: (⨁ is matrix
addition, ⊙ is Hadamard Product)

c t = z f  c t -1 + z 1  z
ht = z o  tanh ( c t ) (8.4)
y = σ (W' h
t t
)
LSTM has: (1) forget, (2) memory select, and (3) output stages.
1. Forget stage
This stage retains important information passed in by previous node ct‐1 (the
previous cell state) and discards unimportant ones. The calculated zf is used as a
forget gate to control what type of ct‐1 information should be retained or discarded.
2. Memory select stage
This stage remembers input xt selectively to record important information. z
refers to present input. zi is the input gate to control gating signals.
3. Output stage
8.4 Recurrent Neural Network (RNN) 185

This stage determines what is considered as ht (the current state) to be passed

down to the next layer. zo is output gate to control this process prior ct is scaled
from memory select stage (convert through a tanh function).

Each layer output yt is calculated by multiplying weights with ht and converted

the product through an activation function like RNN, the cell state ct is passed to
next layer at the end of each layer.

8.4.5 Gate Recurrent Unit (GRU)

8.4.5.1 What Is GRU?

Gate Recurrent Unit (GRU) can be considered as a kind of RNN like LSTM but to
manage backpropagation gradients problems (Chung et al. 2014; Dey and Salem
2017). GRU proposed in 2014 and LSTM proposed in 1997 had similar perfor-
mances in many cases but the former is often exercised due to simple calculation
with comparable results than the latter.
GRU’s input and output structures are like RNN. There are inputs xt and ht‐1 to
contain relevant information of the prior node. Current outputs yt and ht are calcu-
lated by combining xt and ht‐1. A GRU architecture is shown in Fig. 8.16.

8.4.5.2 GRU Inner Architecture

r is reset gate and z is update gate. They are concatenated with input xt and hidden
state ht‐1 from the prior node and multiply results with weights as shown in Fig. 8.17.
When a gate control signal is available, apply r reset gate to obtain data ht‐1 = ht‐
1
er after reset, ht‐1 are concatenated with xt and apply a tanh function to generate data
that lies within range (−1, 1) as shown in Fig. 8.18.
At this point, h′ contains current input xt, its selection memory stage is like LSTM.

Fig. 8.16 General yt

architecture of GRU

ht-1 GRU ht

xt
186 8 Transfer Learning and Transformer Technology

Fig. 8.17 Reset and

update gates of GRU xt
r = s( Wr )
ht-1

xt
z = s( Wz )
ht-1

Fig. 8.18 Computation of h

Finally, update memory stage is the most critical step where forget and remem-
ber steps are performed simultaneously. The gate z obtained earlier is applied as:

ht = (1 − z )  ht −1 + z  h′ (8.5)

where z (gate signal) is within range 0–1. If it is close to 1 or 0, it signifies more data
is remained or forgotten, respectively.
(1‐z) ⊙ ht‐1 represents to forget the original hidden state selectively. (1−z) is con-
sidered as a forget gate to forget ht‐1 unimportant information.
z ⊙ h′ represents h′ memory selective information of present node. Like (1‐z), it
will forget h′ unimportant information or is considered as selective h′ information.
ht = (1‐z) ⊙ ht‐1 + z ⊙ h′ is the calculation to forget ht‐1 information from passed
down and add information from the current node.
It is noted that forget z and select (1-z) factors are linked, which means it will
forget the passed in information selectively. When weights (z) are forgotten, it will
apply weights in h′ to configurate (1-z) at a constant state.
GRU’s input and output structures are like RNN, its internal concept is like
LSTM. GRU has one less internal gate as compared with LSTM and fewer param-
eters but can achieve comparable satisfactory results with reduced time and compu-
tational resources. A GRU computation module is shown in Fig. 8.19.

8.4.6 Bidirectional Recurrent Neural Networks (BRNNs)

8.4.6.1 What Is BRNN?

Bidirectional Recurrent Neural Network (BRNN) is a type with RNN layers in two
directions (Singh et al. 2016). It links with previous and subsequent information
outputs to perform inference against both RNN and LSTM to possess information
from previous one. For example, in text summarization, it is insufficient to consider
8.4 Recurrent Neural Network (RNN) 187

Fig. 8.19 Computation

module of GRU

Fig. 8.20 Structure of BRNN

the information from previous content, sometimes it also requires subsequent text
information for words prediction of a sentence. BRNN is proposed to deal with
these circumstances.
BRNN consists of two RNNs superimposed on top of each other. The output is
mutually generated by two RNNs states. A BRNN structure is shown in Fig. 8.20.
BRNN training process is as follows:
1. begin forward propagation from time step 1 to time step T to calculate hidden
layer’s output and save at each time step,
2. proceed from time step T to time step t to calculate backward hidden layer output
and save at each time step,
188 8 Transfer Learning and Transformer Technology

3. obtain each moment final output according to forward and backward hidden lay-
ers after calculating all input moments from both forward and backward
directions.

8.5 Transformer Technology

8.5.1 What Is Transformer?

Transformer is a network based on attention mechanism without recurrent and con-

volution units (Vaswani et al. 2017). Transformer and LSTM have different training
processes. LSTM is serial and iterative; it cannot proceed until the word before is
processed against transformer is parallel which means that all words are trained
simultaneously to improve computational efficiency. A transformer system struc-
ture is shown in Fig. 8.21.

8.5.2 Transformer Architecture

A transformer model has two parts (1) encoder and (2) decoder. Language sequence
extracts as input, encoder maps it into a hidden layer, and decoder maps the hidden
layer inversely to a sequence as output.

8.5.2.1 Encoder

There are six identical encoder layers in the transformer with two sublayers: (1)
self-attention and (2) feedforward in each encoder layer. Self-attention layer is the
first sublayer to exercise attention mechanism, and a simple fully connected feed-
forward network is the second sublayer. There follows a residual connection and
layer normalization from each of the sublayers. An encoder layer architecture is
shown in Fig. 8.22.

8.5.2.2 Decoder

There are 6 identical encoder layers in the transformer. In addition to identical two
sublayers as each encoding layer, a third sublayer is added to the decoder to perform
multi-head attention, taking the output of last encoder layer as input. Residual con-
nections and layer normalization are used sequentially for all sublayers, which is the
same as the encoder. The decoder's self-awareness is modified by the mask to ensure
that inference of the position can only use information from a known position, or in
other words, its previous position.
8.5 Transformer Technology 189

Output
Probabilities

Softmax

Linear

Add & Norm

Feed
Forward

Add & Norm

Add & Norm
Multi-Head
Feed Attention
Forward Nx

Add & Norm

Nx Add & Norm
Masked
Multi-Head Multi-Head
Attention Attention

Positional Positional
Encoding Encoding
Input Output
Embedding Embedding

Inputs Outputs
(Shifted right)

Fig. 8.21 Transformer architecture

8.5.3 Deep Into Encoder

8.5.3.1 Positional Encoding

Since transformer has no iterative process, each word’s position information must
be provided to ensure that it can recognize the position relationship in language.
Linear transformation of sin and cos functions is applied to provide model position
information as equation:
190 8 Transfer Learning and Transformer Technology

Fig. 8.22 Architecture of an encoder layer

(
PE ( pos, 2i ) = sin pos / 10,0002i / dmodel )
(
PE ( pos, 2i + 1) = cos pos / 10,0002i / dmodel ) (8.6)

where pos represents to a word’s position in a sentence, i represents to word vector’s

dimension number, dmodel represents to embedded dimension’s value. There is a set
of formulas such as sets of 0, 1, or 2, 3 processed with the above sum function,
respectively. As the dimension number increases, the periodic changes moderately
to generate a texture containing position information.

8.5.3.2 Self-Attention Mechanism

For input sentence, the word vector of each word is obtained through word embed-
ding, and the position vector of all words is obtained in same dimensions through
positional encoding that can be added directly to obtain the true vector representa-
tion. ith word’s vector is written as xi, X is input matrix combined by all word vec-
tors. ith row refers to ith word vector.
WQ, WK, WV are matrices defined to perform three linear transformations with X
to generate three matrices Q (queries), K (keys), and V (values), respectively.
8.5 Transformer Technology 191

Q = X ⋅ WQ

K = X ⋅ WK

V = X ⋅ WV (8.7)

Attention mechanism computation can be described as:

 QK T 
Attention ( Q, K , V ) = softmax  V (8.8)
 d 
 k 

The dot products are calculated by multiplying query Q by keys K, dividing the
result by d k , and applying a softmax function to obtain value scores V.

8.5.3.3 Multi-Head Attention

The previously defined set of Q, K, V allows a word that uses the information of
related words. Multiple Q, K, V defined groups can enable a word to represent sub-
spaces at different positions with identical calculation process, except that the
matrix of linear transformation has changed from one group (WQ, WK, WV) to multi-
ple groups ( WQ0 ,WK0 ,WV0 ), ( WQ1 ,WK1 ,WV1 )... as equation:

MultiHead ( Q, K , V ) = Concat ( head1, …, head h ) ⋅ WO

where

head i = Attention ( XWQi , XWKi , XWVi ) (8.9)

where WO is the weights of concatenated results.

Adding input with a sublayer (self-attention layer, for example) to generate
residual connections as equation:

X attention = X embedding + Attention ( Q, K , V ) (8.10)

8.5.3.4 Layer Normalization of Attention Sublayer

Layer normalization is to standardize the distribution of hidden layers indepen-

dently to improve convergence and training processes effectively.

X attention = LayerNorm ( X attention ) (8.11)

192 8 Transfer Learning and Transformer Technology

8.5.3.5 Feedforward Layer

It is a two-layer linear map with an activation function, i.e. ReLU.

(
X hidden = Linear ReLU ( Linear ( X attention ) ) )
followed by residual connection and layer normalization scheme:

X hidden = X attention + X hidden

X hidden = LayerNorm ( X hidden ) (8.12)

8.6 BERT

8.6.1 What Is BERT?

BERT is a pre-trained model of language representation called Bidirectional

Encoder Representation from Transformers (Devlin et al. 2018). It uses MLM
(masked language model) to generate deep bidirectional linguistic representation
instead of traditional one-direction model or concatenate two one-direction models
to pre-train language.

8.6.2 Architecture of BERT

BERT models are pre-trained either by left-to-right or right-to-left language models

previously, this unidirectional property restricts model structure to obtain unidirec-
tional context information only and propensity for representation. BERT adopted
MLM in pre-training stage and a bidirectional transformer with deep layers to build
the entire model, the representation generated integrates both left and right content
information. A BERT system architecture is shown in Fig. 8.23.

8.6.3 Training of BERT

BERT has two training process steps: (1) pre-training and (2) fine-tuning.
8.6 BERT 193

Fig. 8.23 System architecture of BERT

8.6.3.1 Pre-training BERT

BERT is not constrained by a one-way language model because it randomly replaces

tokens in each training sequence with mask tokens ([MASK]) with 15% probability
to predict the original word at position [MASK]. [MASK] does not appear in fine-
tuning of downstream tasks, leading to differences in pre-training and fine-tuning
stages, because the pre-training objective improves language representation, being
sensitive to [MASK] and to other insensitive tokens. BERT applies the following
strategies:
First, in each training sequence, a token position is randomly selected for predic-
tion with a probability of 15%. If ith token is selected, it will be replaced by one of
the following tokens:
1. 80% is [MASK]. For instance, the cat is adorable → the cat is [MASK].
2. 10% is a random token. For instance, the cat is adorable → the cat is ginger.
3. 10% is the original token (no change). For instance, his cat is adorable → his cat
is adorable.
Second, apply Ti corresponding to the position, predict the original token through
full connection, then apply softmax to output the probability of each token, and
finally apply cross-entropy to evaluate loss.
This method causes BERT sensitive to [MASK] and all tokens to extract repre-
sentative information.

8.6.3.2 Next Sentence Prediction (NSP)

There are tasks such as question answering and natural language reasoning to
understand the relationship between two sentences. Sentence-level representations
cannot be captured directly, as MLM tasks tend to extract token-level representa-
tions. BERT applies NSP pre-training task to let the model understands the relation-
ships between sentences and predict whether they are connected.
194 8 Transfer Learning and Transformer Technology

For every training sample, select Set A and B from corpus to create a sample,
where Set A is 50% of Set B (labeled “IsNext”), and Set B is 50% random. Next,
training examples are put into BERT model to generate binary classification
predictions.

8.6.3.3 Fine-tuning BERT

It is necessary to add an additional output layer to fine-tune downstream tasks for

satisfactory performance. It does not require task-specific structural modification in
this process.

8.7 Other Related Transformer Technology

8.7.1 Transformer-XL

8.7.1.1 Motivation

Transformers are widely used as a feature extractor in NLP but required to set a
fixed length input sequence, i.e. the default length for BERT is 512. If text sequence
length is shorter than fixed length, it must be solved by padding. If text sequence
length exceeds fixed length, it can be divided into multiple segments. Each segment
is processed at training separately as shown in Fig. 8.24.
Nevertheless, there are two problems: (1) segments are trained independently,
the largest dependency between different tokens depends on the segment length; (2)
segments are separated according to a fixed length without sentences’ natural
boundaries consideration to produce semantically incomplete segments. Thus,
transformer-XL (Dai et al. 2019) is proposed.

Fig. 8.24 Segment training of standard transformer

8.7 Other Related Transformer Technology 195

8.7.1.2 Transformer-XL technology

1. Segment-level recurrence: When processing the current segment,

Transformer-XL caches and applies hidden vector sequence to all layers from
previous segment. These sequences only participate in forward calculation with-
out backpropagation called segment-level recurrence. Figure 8.25 shows the seg-
ment training of Transformer-XL.
2. Relative position encodings: Each token has an embedding position to repre-
sent positions relationship in standard transformer. This embedding position
encoding is either generated by sin/cos function or learning, but it is impractical
in Transformer-XL because positional relationship of different segments is
unidentified if the same positional code is added to each segment. Transformer-XL
applies relative position encoding instead of absolute position encoding, so when
calculating the hidden vector of current position, it considers tokens’ relative
position relationships to calculate attention score.

8.7.2 ALBERT

BERT model has many parameters, but it is limited by GPU/TPU memory size as
model size increases. Google proposed A Lite BERT (ALBERT) to solve this prob-
lem (Lan et al. 2019). ALBERT applies two techniques to reduce parameters and
improve NSP pre-training task, which include:
1. parameter sharing—apply same weights to all 12-layers,
2. factorize embeddings—shorten initial embeddings to 128 features,
3. pretrain by LAMB optimizer—replace ADAM Optimizer,
4. Sentence Order Prediction (SOP)—replace BERT’s Next Sentence Prediction
(NSP) task,
5. N-gram masking—modify Masked Language Model (MLM) task to mask out
words’ N-grams instead of single words.

Fig. 8.25 Segment training of transformer-XL

196 8 Transfer Learning and Transformer Technology

Exercises
8.1. What is Transfer Learning (TL)? Compare the major differences between
Transfer Learning (TL) and traditional Machine Learning (ML) in AI.
8.2. Describe and explain how Transfer Learning (TL) can be applied to NLP. Give
two NLP applications as examples to support your answer.
8.3. Compare the major differences between Heterogeneous vs. Homogeneous
Transfer Learning. Give two NLP applications/systems as examples for
illustration.
8.4. What is Recurrent Neural Network (RNN)? State and explain why RNN is
important for the building of NLP applications. Give 2 NLP applications as
example to support your answer.
8.5. State and explain FIVE major categories of Recurrent Neural Networks (RNN).
For each type, give a live example for illustration.
8.6. What is LSTM network? State and explain how it works by using NLP applica-
tion such as Text Summarization.
8.7. What is Gate Recurrent Unit (GRU)? Use an NLP application as example, state
and explain the major differences between GRU and standard RNN.
8.8. State and explain the key functions and architecture of Transformer technol-
ogy. Use NLP application as example, state briefly how it works.
8.9. What is BERT model? Use NLP application such as Q&A chatbot as example,
state and explain briefly how it works.

References

Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio,
Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine
translation. arXiv preprint arXiv:1406.1078.
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014). Empirical evaluation of gated recurrent
neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019). Transformer-xl:
Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860.
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional
transformers for language understanding. arXiv preprint arXiv:1810.04805.
Dey, R., & Salem, F. M. (2017, August). Gate-variants of gated recurrent unit (GRU) neu-
ral networks. In 2017 IEEE 60th international midwest symposium on circuits and systems
(MWSCAS) (pp. 1597-1600). IEEE.
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8),
1735-1780.
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., & Soricut, R. (2019). Albert: A lite bert
for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Pan, S. J., & Yang, Q. (2009). A survey on transfer learning. IEEE Transactions on knowledge and
data engineering, 22(10), 1345-1359.
Sherstinsky, A. (2020). Fundamentals of recurrent neural network (RNN) and long short-term
memory (LSTM) network. Physica D: Nonlinear Phenomena, 404, 132306.
References 197

Singh, B., Marks, T. K., Jones, M., Tuzel, O., & Shao, M. (2016). A multi-stream bi-directional
recurrent neural network for fine-grained action detection. In Proceedings of the IEEE confer-
ence on computer vision and pattern recognition (pp. 1961-1970).
Staudemeyer, R. C., & Morris, E. R. (2019). Understanding LSTM--a tutorial into long short-term
memory recurrent neural networks. arXiv preprint arXiv:1909.09586.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., & Polosukhin,
I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
Weiss, K., Khoshgoftaar, T. M., & Wang, D. (2016). A survey of transfer learning. Journal of Big
data, 3(1), 1-40.
Yin, W., Kann, K., Yu, M., & Schutze, H. (2017). Comparative study of CNN and RNN for natural
language processing. arXiv preprint arXiv:1702.01923.
Yu, Y., Si, X., Hu, C., & Zhang, J. (2019). A review of recurrent neural networks: LSTM cells and
network architectures. Neural computation, 31(7), 1235-1270.
Zhuang, F., Qi, Z., Duan, K., Xi, D., Zhu, Y., Zhu, H., & He, Q. (2020). A comprehensive survey
on transfer learning. Proceedings of the IEEE, 109(1), 43-76.
Chapter 9
Major NLP Applications

9.1 Introduction

This chapter will study three major NLP applications: (1) Information Retrieval
Systems (IR), (2) Text Summarization Systems (TS), and (3) Question-&-Answering
Chatbot System (QA Chatbot).
Information retrieval is the process of obtaining the required information from
large-scale unstructured data relative to traditional structured database records from
texts, images, audios, and videos. Information retrieval systems are not only com-
mon search engines but also recommendation systems like e-commerce sites, ques-
tion and answer, or interactive systems.
Text Summarization is the process of diminishing a set of data computationally,
creating a subset or summary to represent relevant information for NLP tasks such
as text classification, question answering, legal texts, news summarization, and
headlines generation.
Question-Answer (QA) system represents human–machine interaction system
with human natural language is the communication medium. It is a task-oriented
system to deal with objectives or answer specific questions through dialogues with
sentiment analysis.

9.2 Information Retrieval Systems

9.2.1 Introduction to IR Systems

NLP used AI techniques like N-gram, rule-based approaches, Word2vec to retrieve

information but encountered computational limitations to process large amount of
corpus information, define text and model frameworks for domain specifics, GPU
clusters, and induce high costs to maintain rule sets due to standard modifications.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 199
Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_9
200 9 Major NLP Applications

Corpora cater for IR in open machine-readable standard format had grown expo-
nentially due to pre-trained models’ technological advancements. IR models for
generic language that combines generic terms with domain-specific terms, e.g.
lease can be a place or a leasehold, its objectives can be organized by abstract, for-
mal, or colloquial language in a large narrative component based on document type
to improve retrieval results.
Text or document classification and clustering in IR research focuses on two
aspects: 1) text representation and 2) clustering algorithms. Text representation is to
convert unstructured text into a computer-processable data format. During text rep-
resentation process, it is necessary to extract and mining textual information.
Semantic similarity computation is the link between text modelling and representa-
tion with application on potential information text layer. Clustering algorithms are
to extract semantic information to facilitate similarity calculation for text classifica-
tion and clustering effectiveness.

9.2.2 Vector Space Model in IR

Vector Space Model (Salton et al. 1975) was a leading IR method from 1960 to
1970. Queries and retrieved documents are represented as vectors with dimension-
ality related to word list size in this model. A retrieved document D can be repre-
sented as a vector of lexical items: Di = (d1, d2, , , , dn ), where di is the weight of a ith
lexical item in Di. Query Q is expressed as a lexical item vector: Q = (q1, q2, , , , qn ),
where qi is the weight of ith lexical item in query term. The relevance is determined
by computing the distance between lexical item vectors of the retrieved document
and query based on this representation. Although it cannot prove cosine relevance is
superior to other similarity methods, but it achieved satisfactory performance
according to search engines evaluation results. Cosine similarity for angle between
retrieved document and query calculation is expressed as:

∑ d ⋅q
n
di ⋅ q j =1 ij j
sim ( Di , Q ) =
= (9.1)
di × q
∑ d ⋅∑
n n
2
j =1 ij
q2
j =1 j

Equation (9.1) is the weights for dot or inner product of all word terms in query
matching documents. There are many words item weights for vector space models.
Most of the weighting methods are based on TF (Term-Frequency) variation.
Inverted document frequency (IDF) (Aizawa 2003) represents the number of term
occurrences in retrieved document and reveals lexical term significance in the entire
document dataset. A lexical item is insignificant with high occurrence frequency in
multiple retrieved documents.
9.2 Information Retrieval Systems 201

There are other text representations methods in addition to vector space model,
e.g. phrase or concept representations. Although phrase representation can improve
semantic contents, the reduced statistical quality of feature vector becomes sparse
and difficult to extract statistical properties applying machine learning algorithms.
Figures 9.1 and 9.2 show a text is encoded by Sentence Transformers (Reimers and
Gurevych 2019) to demonstrate and compute cosine similarity between embed-
dings. It uses a pre-trained model to encode two sentences and outperform other
pre-train model like BERT (Vaswani et al. 2017).
It is natural to identify the combination with the highest cosine similarity score.
By doing so, an intense ranking scheme is used as shown in Fig. 9.3 to identify the
highest scoring pair with a secondary complexity. However, it may not work for
long lists of sentences.

Fig. 9.1 Sentence transformers frame

Fig. 9.2 BERT frame

202 9 Major NLP Applications

Fig. 9.3 The singer example of vector space model

A chunking concept to divide corpus into smaller parts is shown in Figs. 9.4 and
9.5. For example, parse 1000 sentences at a time to search the rest (all other sen-
tences) of corpus or search a list of 20k sentences to divide into 20 × 1000 sen-
tences. Each query is compared with 0–10k sentences first and 10k–20k sentences
to reduce memory storage. The increases of these two values intensified speed and
memory storage, then identified pair with the highest similarity to extract top K
scores for each query as opposed to extract and sort scores for all n2 pairs.
Such method is faster than brute force methods due to fewer samples. In practical
industrial scenarios, more attention is paid to the speed of pre-trained models,
encoding methods, and data retrieval. For example, two-tower model (Yang et al.
2020) and Wide&Deep model (Cheng et al. 2016) are shown in Figs. 9.6 and 9.7.

9.2.3 Term Distribution Models in IR

Probabilistic Ranking Principle (PRP) models were firstly proposed by Croft and
Harper in 1979 (Croft and Harper 1979) to compute query relevance degrees and
retrieval. PRP regards IR as a process of statistical inference, where an IR system
predicts query relevance from retrieved documents and sorts in descending order
based on predicted relevance scores. This approach is like Bayesian model machine
learning. A PRP model combines relevant feedback information with IDF and esti-
mate each item’s probabilities to optimize search engine retrieval performance.
However, it is a difficult task to estimate each probability accurately in practical
applications. Okapi BM25 (Whissell and Clarke 2011) retrieval model had solved
9.2 Information Retrieval Systems 203

Fig. 9.4 Multiple examples of vector space model

the difficulties encountered by PRP model with satisfactory performance in TREC

retrieval experiments and commercial search engines. Many IR researchers had
modifications based on BM25 model resulting in many variations, the most com-
mon form is as follows:

( ri + 0.5) / ( R − ri + 0.5)
sim ( Q, D ) = ∑ log
q∈Q ( ni − ri + 0.5) / ( N − ni − R + ri + 0.5)
( k1 + 1) fi ( k2 + 1) qfi
⋅ ⋅ (9.2)
K + fi k2 + qf i
204 9 Major NLP Applications

Fig. 9.5 Chunk multiple examples of vector space model

There are two approaches to consider which is the best BM25 method:
1. BM25 + Word2Vec embedding across all documents.
2. BM25 + BERT + Word2Vec embedding for each top-k documents, select the
most similar sentence embedding across top-k paragraphs.
Word2vec (Church 2017) is about word occurrences proportions in relations
holding in general over large text corpora and combine vectors of similar words into
a vector space called distributional hypothesis. Word2vec embeddings are to com-
pare query with sentence embeddings to select the one with higher cosine similarity.
Transformer-based neural network models are popular NLP research areas on
enhanced parallelized processing capabilities. BERT is among one that uses
Transformer-based deep bidirectional encoders to learn contextual semantic rela-
tionships between lexical items and performed satisfactory in many NLP tasks.
It began to retrieve document with the most relevant documents followed by
paragraphs and extract sentences from selected paragraphs. BERT embeddings are
used to compare query with paragraphs and select the one with higher cosine simi-
larity. Once relevant paragraphs are available, select sentence with answer by com-
paring sentence embeddings based on Word2Vec embeddings trained on the whole
dataset, then average word embeddings in the paragraph with BM25 score calcula-
tion are shown in Fig. 9.8.
9.2 Information Retrieval Systems 205

Fig. 9.6 Two-tower model (Yang et al. 2020)

Output Units

Hidden Layers

Dense
Embeddings
Sparse Features
Wide Models Wide & Deep Models Deep Models

Fig. 9.7 Wide&Deep model (Cheng et al. 2016)

Common words queries occurred rarely in documents with a higher number of

occurrences produce sparse distribution. Contrarily, there will be similar scores at
many documents if common words with same frequency occurred across docu-
ments. Documents distribution with scores and codes are shown in Figs. 9.9
and 9.10.
Since word2vec relies heavily on each occurrence frequency, thus, it may pro-
duce satisfactory performance on specific queries while the same for BERT on gen-
eral queries.
The results of two selected queries showed that query (Sentence 1) achieved
satisfactory performance on specific/rare terminology, while second query (Sentence
2) achieved satisfactory performance on normal terminology. They depend on words
specification level in the query. For queries which have specific/rare terminology per-
formed satisfactorily with the most similar sentences across all documents. For
206 9 Major NLP Applications

Fig. 9.8 Sample code for Word2vec embeddings with BM25 score calculation

Incubation Period Prevalence of asymptomatic shedding and transmission

8000
Number of documents

Number of documents

6000

4000

2000

0
0 5 10 15 20 25 0 10 20 30 40 50 60
Scores Scores

Fig. 9.9 Documents distribution with scores and codes

queries which have general terms, e.g. age, human, climate performed satisfactorily
with the most relevant documents instead of embeddings comparison across all of
them. Thus, it is reasonable to compare each time the results of two approaches and
select the appropriate one based on words distribution for each query.
9.2 Information Retrieval Systems 207

Fig. 9.10 BM25 results

9.2.4 Latent Semantic Indexing in IR

Term Distribution Models in IR is a rapid and effective model. It uses topics to

express implicit semantics of a document as index to replace incomplete, unreliable
search terms by reliable indicants based on two assumptions:
1. Words have common topics in document.
2. Words not in document less likely to be related.
Topic is filtered out by keywords in the Doc. Thus, P = (ω/Doc) probability dis-
tribution table is introduced: the statistics of word frequency (frequency) in the
document, i.e. the law of large numbers.

P (ω Topic D ) ≈ P (ω D ) = tf (ω , D ) / len ( D ) (9.3)

Topic is regarded as a language model, and P = (ω/Doc) is the probability of

word generation in this language model so the word not only occur in topic, but has
probability generated.
There are two methods of sorting according to statistical language model when
query Q is given, which are (1) Query-likelihood and (2) Document-likelihood
methods.

9.2.4.1 Query-Likelihood

Determine MD, corresponding to each Doc, user’s Query is denoted as

Q = (q1, q2, , , , qn ). Query probability will be generated under the language model of
each document that can be calculated as follows (Zhuang and Zuccon 2021):
208 9 Major NLP Applications

k
P ( q1 … qk M D ) = ∏P ( qi M D ) (9.4)
i =1

Search results are obtained by sorting all computed results. However, this method
calculates the probability for each Doc independently from other Docs, and the
relevant documents are not utilized.

9.2.4.2 Document-Likelihood

Determine each Query corresponding MQ. Calculate the probability that any given
document will be generated under the query’s language model (Zhuang and
Zuccon 2021):

( )
P D M Q = ∏P ω M Q
ω ∈D
( ) (9.5)

The object of one-mode factor analysis traditionally is a matrix composed of

identical object-pair types of relationships. An example is a document–document
matrix. The matrix elements may be evaluated for similarity between documents
manually. This symmetric square matrix is decomposed into two matrices by eigen-
analysis. The decomposed matrix is composed of linearly independent factors.
Many of the factors are tiny that can be ignored usually producing an original matrix
approximation.
Two-mode factor analysis object is a matrix consisting of object–pair relation-
ships. This matrix can be decomposed into term–term, document–document, and
term–document matrices using singular value decomposition (SVD) (Aharon et al.
2006). SVD reconstructs spatial response to the main patterns association between
data by ignoring less significant effects. Thus, a term which does not occur in a
document may be immediately adjacent to that document in semantic space based
on identified association patterns. The information located in semantic space has a
role in semantic index. SVD model test and lean with results in Latent Semantic
Indexing in IR is shown in Fig. 9.11.
Singular-value decomposition and corresponding validation results are shown in
Figs. 9.12 and 9.13.

9.2.5 Discourse Segmentation in IR

Document contents combine with articulated parts such as paragraphs exalt auto-
matic documents segmentation according to meanings using machine learning
methods to compare two adjacent sentences similarity in turn and generate segmen-
tation point with the lowest similarity. This unsupervised method is called TextTiling
(Hearst 1997) as shown in Fig. 9.14. Further, supervised learning methods can also
9.2 Information Retrieval Systems 209

Fig. 9.11 SVD frame

Fig. 9.12 Example of singular-value decomposition

be used such as classifiers constructions (Florian 2002) or sequence models

(Keneshloo et al. 2019) to detect segmentation point.
Rhetorical Structure Theory (RST) framework (Taboada and Mann 2006) is a
commonly used framework for parsing discourse as shown in Fig. 9.15. RST com-
mon relations in English are conjunction, justify, concession, elaboration, etc. as
shown in Figs. 9.16 and 9.17.
210 9 Major NLP Applications

Fig. 9.13 Validation results

Fig. 9.14 Examples of discourse segmentation

There are two approaches to identify relationships: (1) rule-based on iconic

words such as but, so, for example, and (2) machine learning with commonly fea-
tures such as bag of words (Zhang et al. 2010), Discourse markers (Fraser 1999),
Starting/ending N-grams (Robertson and Willett 1998), Location in the text
9.2 Information Retrieval Systems 211

Fig. 9.15 Example of

rhetorical structure theory

Fig. 9.16 Examples of relations

(Rothkopf 1971), Syntax features (Sadler and Spencer 2001), Lexical and distribu-
tional similarities (Weeds et al. 2004).
Discourse segmentation task is a significant evaluation indicator for NLP devel-
opment directions. From the application perspective, discourse segmentation can
assist users rely on intelligence to improve productivity, its technology core value
can convert semi-structured and unstructured data to specific description structured
in turn to support substantial downstream applications.
212 9 Major NLP Applications

Fig. 9.17 Attention map

9.3 Text Summarization Systems

9.3.1 Introduction to Text Summarization Systems

9.3.1.1 Motivation

There is excess information from copious sources to obtain the latest information
daily. Although automatic and accurate summarization systems can assist users to
simplify, identify, and understand key information in the shortest possible time but
they remain challenging as new words and complex text structure documents are
available constantly.

9.3.1.2 Task Definition

Text summarization process generates text (document or document) summaries by

rewriting and summarizing long text into short form (Mahalakshmi and Fatima
2022). It refers to extract or refine text or text set key points through technologies to
display original text or text set main contents or general idea. Text generation task
is an information compression technique, whereas a summarization process is con-
sidered as a function where input is a document or documents, and output is an input
texts summary. Hence, input and output are quintessential types to classify sum-
mary tasks.

9.3.1.3 Basic Approach

Summarization approaches are mainly divided into extractive and abstractive (Chen
and Zhuge 2018).
Extractive methods select important phrases from input text, combine them to
form a summary like a copy and paste process. Many traditional text summarization
methods use Extractive Text Summary (ETS) because it is simple to generate sen-
tences without grammatical errors but cannot reflect exact sentences meanings.
They are inflexible to use novel expressions, words, or connectors outside text
descriptions.
9.3 Text Summarization Systems 213

Abstractive Text Summary (ATS) methods use language generation methods to

re-organize contents, generate new words, and conclude the implied information as
compared with ETS. They paraphrase text meanings composed of new words with
original words summary (Agrawal 2020) and mimic human understanding to
develop contents which may not contain in actual document text (Malki et al. 2020).

9.3.1.4 Task Goals

Summarization task objectives are to assist users to understand raw text within a
short period as shown in Fig. 9.18.

9.3.1.5 Task Sub-processes

Summarization tasks are divided into the following modules as shown in Fig. 9.19.

Fig. 9.18 Summarization tasks objectives

Fig. 9.19 Summarization tasks sub-processes

214 9 Major NLP Applications

Input document or documents are first combined and preprocessed from continu-
ous text form to split sentences. The sentences will be encoded into vectors form
data to fit into a matrix for similarity scores calculation to obtain sentence rankings,
followed by a summary with the highest possibility according to the ranking list.

9.3.2 Text Summarization Datasets

Text summarization datasets commonly used include DUC (DUC 2022), New York
Times (NYT 2022), CNN/Daily Mail (CNN-DailyMail 2022), Gigaword (Gigaword
2022), and LCSTS datasets (LCSTS 2022).
DUC datasets (DUC 2022) are the most fundamental text summarization datas-
ets developed and used for testing purposes only. They consist of 500 news articles,
each with four human-written summaries.
NYT datasets (NYT 2022) contain articles published in New York Times between
1996 and 2007 with abstracts compiled by experts. The abstract datasets are some-
times incomplete and sporadic short sentences with average of 40 words.
CNN/Daily Mail datasets (CNN-DailyMail 2022) are widely used multi-sentence
summary datasets often trained by generative summary system. They have (a) ano-
nymized version to include entity names and (b) non-anonymized version to replace
entities with specific indexes.
Gigaword datasets (Gigaword 2022) are abstracts comprising of the first sen-
tence and article title with heuristic rules of approximately 4-million articles.
LCSTS datasets (LCSTS 2022) are Chinese short texts abstract datasets con-
structed by Weibo (2022).

9.3.3 Types of Summarization Systems

Text summarization task for input documents can be divided into two types:
1. Single document summarization considers each input is one document.
2. Multiple document summarization considers input has several documents.
Text summarization task viewpoint can be divided into three classes:
1. Query-focused summarization adds viewpoint to query.
2. Generic summarization is generic.
3. Update summarization is a special type which sets difference (update) viewpoint.
Summarization systems based on contents can be divided into four types:
1. Indicative summarization describes contexts without revealing details especially
the endings, it contains partial information only.
2. Informative summarization contains all information in a document or documents.
9.3 Text Summarization Systems 215

3. Keyword Summarization reveals output generation is sporadic text which con-

tains phrases or words of input documents.
4. Headline Summarization is usually single line summary
These summarization systems can be divided according to summary languages
such as Arabic (Elsaid et al. 2022), Chinese (Yang et al. 2012), English and Spanish
summarization systems, etc.

9.3.4 Query-Focused Vs Generic Summarization Systems

Text summarization can be query-focused or generic. Summary associative with

query shows document contents is relative to initial search query. A query-related
summary generation is a process of retrieving query-related sentences/paragraphs
from a document that has a strong similarity to text retrieval process. Hence,
abstracts relevant searches are often undertaken by extending traditional IR tech-
niques with many text abstracts in the literature fall into this category. A general
summary, however, provides an overall sense of the document’s contents. A proper
general summary should cover main topics and minimize redundancy. Since there
are no queries or topics to feed into summarization process, it is difficult to develop
a high-quality general summarization method for evaluation (Gong and Liu 2001).

9.3.4.1 Query-Focused Summarization Systems

Query-Focused Summarization (QFS) is primarily addressed using extractive meth-

ods to produce text lacks coherence. QFS methods can overcome these limitations
and improve incoherent texts availability. A Relevance Sensitive Abstractive QFS
(RSA-QFS) framework (Baumel et al. 2018) is shown in Fig. 9.20.
This model assumes that a trained abstractive model includes reusable language
knowledge to accomplish QFS tasks. Methods of enhancing this pre-trained single
document abstraction model with explicit modelling of query dependencies are
studied to improve multiple input documents operating ability and adjust generated
abstractions lengths accordingly.
Further, a sequence-to-sequence (seq2seq) architecture is applied to obtain sum
via an iterative extraction or abstraction pairs process: identify relevant content
batches from multiple documents and abstract into a coherent text segments
sequence.
QFS task includes two stages as shown in Fig. 9.21:
1. a relevance model to determine passages relevance to input query from source
documents and
2. a generic summarization method to combine relevant passages into a coher-
ent summary
216 9 Major NLP Applications

Fig. 9.20 RSA-QFS framework

Fig. 9.21 Two stages of QFS

Query-related text summarization are practical for answering questions such as

whether a whole or partial document has relevance to a user’s query. Query-related
summaries do not provide an overall sense of the document’s content, they have
query bias and unsuitable for content summaries to answer questions such as docu-
ment category, key points, text summary, etc.
9.3 Text Summarization Systems 217

9.3.4.2 Generic Summarization Systems

A proper generic summarization should cover main topics as many as possible and
minimize redundancy leading to fractious system generation and evaluation. It often
lacks consensus on summary output and performance judgments without query pro-
visions and topics to summary task.
Typical generic summarization ranking models and selected sentences are based
on relevance similarity values and other semantic analysis (Gong and Liu 2001).

9.3.5 Single and Multiple Document Summarization

Single document extraction in journalism has developed to multi-document extrac-

tion since 1990. A variety of news articles, such as Google News (Google 2022),
Columbia News Blaster (Columbia 2022), and News Essence (NewsInEssence
2022) are inspired by multi-document summaries. The reason is that individual
documents always produce contradictory results through overlapping information
from multiple documents (Alami et al. 2015) which may affect the performance of
summarization results.
Single document summarization research method gradually faded in past decades
(Svore et al. 2007) as mainstream research focused on multi-document summariza-
tion which could reduce text size, gather ideas, compare documents, maintain syn-
tactic and semantic relationships (Pervin and Haque 2013).

9.3.5.1 Single Document Summarization

Single document summarization’s challenge is to identify or generate informative

sentences significance of the document because it often has inconsistent and inter-
mittent information.
Salient features like sentence placement are early research (Baxendale 1958)
where 200 paragraphs selected and identified paragraphs have topic sentences at the
beginning and end of paragraphs with 85% and 7%, respectively.
A corpus with around 400 technical documents research focusing on word fre-
quency and word position, and cue words and skeleton were proposed in 1969
(Edmundson 1969). Results showed that extracted summary to actual summary
accuracy rate was about 44%.
Further, lexical indicators (Rath et al. 1961), cohesion (Hasan 1984) and seman-
tic relationships (Halliday and Hasan 1976), algebraic method such as naïve-bayes
classifier processed features like uppercase words, lengths, words position (Kupiec
et al. 1995), symbolic word knowledge (Hovy and Lin 1999), human abstraction
concept (Jing 2000) are research areas applied in this field.
218 9 Major NLP Applications

9.3.5.2 Multiple Document Summarization

Multiple document summarization similarity measures and extractive techniques

are comparable to single document summarization.
It used clustering to identify common themes (Erkan and Radev 2004), compos-
ite sentences from clusters (Barzilay et al. 1999), maximal marginal relevance
(MMR) (Carbonell and Goldstein 1998), and concatenated to multilingual environ-
ment (Evans 2005).
Further, TFI X IDFI techniques (Salton 1989), TF/IDF (Fukumoto 2004), word
hierarchical technique for frequent terms (You et al. 2009), graph-based methods
(Mani and Bloedorn 1997; Wan 2008), sentence co-relation method (Hariharan
et al. 2013), logical closeness (Zhu and Zhao 2012), and query-oriented approach
(Agarwal et al. 2011) are well-developed.

9.3.6 Contemporary Text Summarization Systems

9.3.6.1 Contemporary Extractive Text Summarization (ETS) System

Text summarization research methods aim to (Dong 2018):

1. acquire important sentences.
2. predict sentence option according to ranking sentences.
The extractive summarization for proper sentences selection from original source
text are required to:
1. include logical and consistent summary information from original text.
2. reduce similar and unimportant sentences information redundancy.
Lead 3 is a commonly used and effective method to extract the first three sen-
tences as topic titles of an article. When dealing with important sentences, docu-
ment equivalence to document topic and relevant sentences position characteristics
are considered. Topic modelling, frequency-based models LSA and Bayesian are
methods applied (Farsi et al. 2021).
Extractive summarization produces incoherent summaries compared with man-
ual ones, its shortcomings include unresolved anaphora, unreadable sentence order,
lacks textual cohesion to extract salient information from long sentences. When the
system focuses on a sentence, it extracts the entire sentence (Nallapati et al. 2017).

9.3.6.2 Graph-Based Method

Graph-based ranking algorithms are successfully used in citation analysis, link

social networks’ structure analysis and the World Wide Web.
9.3 Text Summarization Systems 219

They generate graphs from input document and summary by considering the
relationships between nodes (units of text) (Chi and Hu 2021). TextRank (Mihalcea
and Tarau 2004) is a typical graph-based approach that has developed many models.
A summarization of TextRank system to extract keywords from a sample text and
graph is shown in Figs. 9.22 and 9.23.
This kind of system is based on PageRank algorithm (Langville and Meyer 2006)
applied by Google’s search engine, its algorithm principle is linked pages are good,
and even better if they come from multiple linked pages. Links between pages are
represented by matrices like circular tables. This matrix can be converted to a transi-
tion probability matrix divided by the sum of links per page, and the page will be
moved by page viewer following a feature matrix in Fig. 9.24.
TextRank processes words and sentences as pages in PageRank, its algorithm
defines text units and adds them as nodes in a graph with relations are defined
between text units and added as edges in the graph. Generally, the weights of edges
are set by similarity or score values.
Then, PageRank algorithm is used to solve the graph. There are other similar
systems such as LexRank (Erkan and Radev 2004) to consider sentences as nodes
and similarity as relations or weights, i.e. IDF-modified cosine similarity to calcu-
late similarity.

9.3.6.3 Feature-Based Method

Feature-based model extracts sentences features and evaluates their significances.

There are many representative studies that include Luhn’s Algorithm (Luhn 1958),
TextTeaser and SummaRuNNer (Nallapati et al. 2017).
Luhn’s Algorithm is used to evaluate input words significance calculated by fre-
quency. TextTeaser is an automatic feature-based summarization algorithm.
SummaRuNNer is implemented by deep neural networks (DNN) structure as shown
in Fig. 9.25.
SummaRuNNer generates sentence feature (vector) by two layers bidirectional
Gate Recurrent Unit—Recurrent Neural Network (GRU-RNN) from word embed-
ding vectors. The lowest level classifies each sentence word level, while the highest

Fig. 9.22 Sample text

220 9 Major NLP Applications

Fig. 9.23 Sample graph for key phrase extraction in TextRank

Fig. 9.24 PageRank algorithm process

9.3 Text Summarization Systems 221

Classification
Layer
1 0 1

DOC

Sentence
layer
REP

Word layer
Input Layer
Sentence 1 Sentence 2 Sentence 3

Fig. 9.25 Network structure of SummaRuNNer

level classifies sentence level. Double arrows indicate two-way RNN. The top layer
numbered with 1s and 0s is a classification layer based on sigmoid activation to
determine whether each sentence is a summary. Each sentence decision depends on
substantial sentence contents, sentences to document relevance, sentences to cumu-
lative summary representation originality, and other positional characteristics.

9.3.6.4 Topic Based Method

Topic-based model considers document’s topic features and input sentences scores
according to topic types contained as major topic would obtain a high rate when
scoring sentences.
Latent Semantic Analysis (LSA) is based on Singular Value Decomposition
(SVD) to detect topics (Ozsoy et al. 2011). An LSA based sentence selection pro-
cess is shown in Fig. 9.26 by topics represented by eigenvectors or principal axes
with corresponding scores.

9.3.6.5 Grammar-Based Method

Grammar-based model parses text and constructs a syntax structure, selects, or reor-
ders the substructure. A representation framework is shown in Fig. 9.27.
222 9 Major NLP Applications

Topic1 Sentence1 Topic1 Topic2

Sent1 0.9 0.25
Sent2 0.7 0.7
Sentence2 Sent3 0.45 0.4
Sentence3 Sent4 0.3 0.6
Sentence5 Sent5 0.4 0.9
Sentence4

Topic2

Fig. 9.26 LSA based sentence selection sample

NP NP

NNP/nn JJ/amod NNP/nn NNP/nn NNP/- NNP/nn NNP/-

ABC dissident Jack Norman Smith Norman Smith

PP/Prep_in PP/Prep_in

IN/- DT/det JJ/amod NN/- PP/prep_of IN/- NNP/nn NNP/-

in the disputed territory in GHI Island

IN/- NNP/nn NNP/-

of GHI Island

NP/dobj PP/Prep_in

DT/det NN/- PP/prep_of NP/poss NN/-

the extradition IN/- NNP/nn NN/nn NNP/- DFG’s extradition

of XYZ leader Jones

Fig. 9.27 Grammar-based method sample network (Ozsoy et al. 2011)

9.3 Text Summarization Systems 223

Grammar pattern can produce significant paraphrases based on grammatical

structures. The above example in Fig. 9.28 showed how paraphrase extraction and
replacement can be achieved by using such method. Analyzing grammatical struc-
ture feature is useful for semantic phrases reconstruction.

9.3.6.6 Contemporary Abstractive Text Summarization (ATS) System

Abstractive summarization often generates summary that maintains original intent

completed by humans.
This process can generate words that are not in original input representations but
to facilitate summaries characteristics and fluency. However, it is complex to gener-
ate coherent phrases and connectors.
Abstractive summarization systems applying deep learning methods,
Reinforcement Learning (RL), Transfer Learning (TL) and Pre-Trained Language
Models (PTLMs) had developed rapidly (Alomari et al. 2022) in recent years. These
models use rules-based frameworks to consider significant events and summaries.
Tree methods are ontology-related methods for abstractions (Jain et al. 2020).

9.3.6.7 Aided Summarization Method

This method combines automatic computer model or algorithm to provide signifi-

cant document information for human decision.
Machine translation model to text summarization was proposed (Banko et al.
2000) applying encoder–decoder framework as neural network model mainstream
and used in abstractive summarization systems (Chopra et al. 2016).

Fig. 9.28 Network framework of point generator baseline model

224 9 Major NLP Applications

9.3.6.8 Contemporary Combined Text Summarization System

Pointer-Generator Networks (See et al. 2017) is a frequently used baseline network.

It focuses on keywords and sentences with Attention technique (Vaswani et al.
2017), to lever generator and pointer network according to calculated probability.
Vocabulary and attention with different weights distribution are then combined. A
baseline pointer-generator network framework is depicted in Fig. 9.28.
It noted that article tokens are fed into an encoder layer, which is a single-layer
bidirectional LSTM with encoder hidden states provided. Decoder consists of a
single-layer unidirectional LSTM, processes word embedding of previous words on
each step and output decoder state with attention distribution.

9.4 Question-and-Answering Systems

9.4.1 QA System and AI

QA system is a remarkable way to mimic human-to-human interaction through

state-of-the-art technology development. Different from other classification or pre-
diction problems, a QA system is a cross discipline in traditional linguistic includ-
ing computer science for computational linguistic with statistics, pattern recognition,
data mining, machine learning, deep learning methods for a well-trained communi-
cation system. It has a critical role for autoresponder, personal assistant, sentiment
chatbot nowadays.
QA system is a popular research topic in NLP which contains one of the open-
domain common sense or special domain knowledge as a qualified conversation
partner. Dialogue realization relies on automatic speech recognition (ASR), natural
language understanding (NLU), dialogue management (DM), natural language gen-
eration (NLG), speech synthesis (SS). A QA system flowchart is shown in Fig. 9.29.
It is an integral part of system acumen. Dialogue Management is the communica-
tion policy or dialogue strategy applied to large corpus for content organization.
After transferring natural language to computer language in sequence-sequence
data with character, word, or sentence level in Natural Language Understanding
(NLU), machine intelligence selects suitable contents for language generation.
Backend technology with generated candidate answers is combined and reranked

Fig. 9.29 Flowchart of a typical QA system

9.4 Question-and-Answering Systems 225

for optimization response in Natural Language Generation (NLG). Apart from text
aspect, ASR and TTS are procedures that resemble machine by human voice recog-
nition and generation.
QA system research is divided into two categories: (1) pattern matching with
rule-based and (2) language generated-based on information retrieval and neural
network. However, the backend equipped more than one method to generate mean-
ingful communication and provide meaningful feedbacks. A QA system in a chatbot
includes an open-domain focus on (1) common sense/world knowledge and (2)
task-oriented for special domain knowledge databases that resemble expert system
involving in-depth knowledge base to support appropriate responses.
First rule-based human–computer interaction as in Fig. 9.30 pattern recognition
system challenged the Turing test in 1950s, reaching a milestone where humans
could not recognize whether the opposite was a machine or human. After a long
period of data collection, database used for dialogue pattern matching is large
enough to rank appropriate feedbacks and give the highest scoring answers, which
is a process of selection from a database of human answers regardless of the
machine. After decades of development, search engines and data crawlers have sup-
ported sources for building knowledge bases, including information retrieval,
enabling search engines to retrieve relevant and up-to-date data for structured pro-
cessing to form answers from QA systems. The advent of AI era enhanced QA
systems mainstream can focus on cognitive science than big data feeds of neural
networks on systems generations. Gradually, traditional QA system is replaced by
AI machine communication as rule-based matching recurrent neural network train-
ing to realize large knowledge base to support the AI brain to imitate human reason-
ing called Natural Language Understanding (NLU).
The main source of knowledge base in a typical QA system comes from: (1)
human–human dialogue collection with handcraft is the answer from human lan-
guage in linguistic and meaning where database consist of pairs dialogues. Without

Fig. 9.30 Human and

machine interaction via
QA system
226 9 Major NLP Applications

any imitation or learning ability, this first version rule-based QA system relies on
pattern matching to measure the distance between proposed question and Question-
Answer pattern stored pair in database. For example, Artificial Intelligence Markup
Language (AIML) can answer most of daily or even professional dialogues based
on large and classified handcraft database without intelligence; (2) building data-
base focus on search engine for Information Retrieval-based knowledge base. The
feature of IR-based QA system is the combination of knowledge building from up-
to-date knowledge bases. An IR-based QA system uses domain knowledge such as
expert system to extract and generate knowledge. The procedure of unstructured
data extraction and reorganization depends on Natural Language Understanding
(NLU) for reasoning. Natural Language generation (NLG) includes knowledge
engineering analysis for reasoning and rerank candidates’ answers optimization.
The latest database used big data for data-driven model to realize machine intel-
ligence. When neural network had provided with sufficient data, sequence-to-
sequence model like Recurrent Neural Network and its related Long-Short-Term
Memory naturally model as in Fig. 9.31 skilled in sequential data processing (Cho
et al. 2014). A neural network model is considered as the black box producing learn-
ing ability with accuracy but cannot comprehend by humans. Prior preprocessing
data was fed to neural model, they required to transform data format from natural
word to vector for data training (Mikolov et al. 2013). Tokenization has three levels:
(1) character, (2) word, and (3) sentence. The input format decides output outcomes
in encoder–decoder framework. Recurrent Neural Network generated words may
not be meaningful in English dictionary because the character level training lacked
enough corpus for a well-trained model. Further, transfer learning with enormous
data pre-trained Transformer model required to select the intended decoder for
training target. For example, Dialogue GPT from OpenAI focuses on formatted
dialogue training to generate responses.
Neural Network system transformed natural language to word vectors for math-
ematical computation to acquire response in NLP. Neural Network can generate
own natural language as compared with traditional techniques.

Fig. 9.31 LSTM structure

9.4 Question-and-Answering Systems 227

Traditional Recurrent Neural Network (RNN) of seq2seq language model

response generation performed lesser than big data-oriented transfer learning such
as Google’s BERT and Open AI’s GPT.
Pre-trained unsupervised learning language model achieved satisfactory perfor-
mance in fine-tuning with small dataset than traditional ones, their performances
attributed to self-attention mechanism (Vaswani et al. 2017) and identified relations
in sequences with fluent and syntactic response for task execution based on GPT
with fine-tuned model (Wolf et al. 2018).

9.4.1.1 Rule-based QA Systems

Rule-based QA systems were proposed at the same time as Turing test in 1950s.
However, original QA systems only followed rules set by humans without self-
improvement capabilities like machine learning, number of dialogue pairs is stored
in database prior the system provided a concrete answer. The simplest but most
efficient way to measure similarity of two groups is the cosine distance of two vec-
tors. It is undeniable that rule-based systems have collected huge dialogue corpora
over decades, giving system confidence when relying on new problems with high
vector similarity. To date, mature rule-based systems are quintessential for all com-
mercial QA systems, as the accumulation of corpora can avoid meaningless
responses that compensate for insufficient domain knowledge with appropriate and
specific human feedbacks.

9.4.1.2 Information Retrieval (IR)-based QA Systems

The knowledge base for IR is unstructured data source using data mining methods
obtained by websites, wordnet which are different from the paired dialogue.
Question-Answer System (KBQA system) is a significant branch of IR-based QA
system knowledge base, its usage depends on knowledge base size of unstructured
data for storage. That is related to knowledge base construction to extract purpose-
ful knowledge from mass data. There are two methods (1) property and (2) relations
to process natural language. Property refers to the definition or concept of one thing
in English–English dictionary to explain another concept. Relations refer to the
relationship between two entities, where a Name Entity Recognition (NER) and
idea from Ontology with Subject–Predicate–Object (SPO) triple must be used to
extract relation. KBQA extension is ontology or knowledge graph (KG) in research.
When entities are linked, the knowledge for one entity can be extracted according to
questions during Natural Language Understanding (NLU). A typical KBQA with
domain knowledge about ontology is shown in Fig. 9.32, its fundamental question
is about who and what correspond to name and relations entities (Cui et al. 2020).
228 9 Major NLP Applications

Fig. 9.32 KBQA system demo

9.4.1.3 Neural Network-Based QA Systems

Neural Network structure in a QA generated-based system is considered as machine

brain imitated by human. Encoder–Decoder framework is a sequence-to-sequence
model like RNN that has natural memory recalling priority and context with atten-
tion mechanism. Dialogue system has identical requirements to represent dialogue
history and avoid meaningless responses to improve users’ experiences.
Deep learning models such as TensorFlow and Pytorch, RNN is easy to imple-
ment for text generation as language model. Google proposed masked language
model to generate language representation called Bidirectional Encoder
Representation from Transformer (BERT), focusing on encoder part trained by
magnitude unlabeled data in 2017. Neural network feeds data for training according
to network advantages due to different NLP tasks in long sentences. BERT can
solve such problem because it deals with 11 common NLP tasks initially. Language
model pre-trained by magnitude data is used to understand common knowledge in
NLP. Fine-tuned should be applied to training specific NLP tasks based on funda-
mental ability (Vaswani et al. 2017).
Open AI released another Transformer framework with unsupervised learning
for pre-trained model directing decoder scheme based on GPT, Open AI GPT-2, and
GPT-3 (Brown et al. 2020). GPT with masked self-attention focuses on known text
so that the word preceding is predicated as different from BERT context self-
attention. GPT-3 can do inference and synonym replacement in addition to normal
function for bilingual translation, text generation, and Question-Answer. It seems
that BERT can handle more NLP tasks than GPT, but GPT text generation prowess
for pre-trained model is widely used in many commercial QA systems and text
summarization.

9.4.2 Overview of Industrial QA Systems

An industrial QA system contains automatic dialogue system assembling chatbot

internal technologies. They have several backend composited control system
responses to equip with necessary knowledge. Meanwhile, QA system evaluation is
9.4 Question-and-Answering Systems 229

proposed during the training period for language model performance (Chen et al.
2017) and on system design sufficient for both languages generations.
Since Encoder–Decoder framework proposed as an end-to-end system and a
sequential language model, RNN is a popular generated-based model in commer-
cial and academics. However, its applications are mainly focused on casual scenar-
ios at open domain without proposed question details. Thus, the response from a
generated-based QA system is appropriate in pairs but lack contents due to the data-
driven model considered basic linguistic and excluded facts from knowledge base
which are identical to traditional dialogue system with meaningless answers. A
knowledge grounded neural conversation model (Ghazvininejad et al. 2018) is pro-
posed based on sequence-to-sequence RNN model and combined dialogue history
with facts related to current contexts as shown in Fig. 9.33.
Microsoft extended its industrial conversation system to achieve useful conver-
sational applications on knowledge grounded with conversation history and external
facts in 2018. It has significant progress in real situations according to conversation
history in Dialog-Encoder, word, and contextually relevant facts in Facts Encoder to
responses as compared with baseline seq2seq model.
The data-driven model of QA system divided source data into conversation data
and non-conversational text which means the conversation pairs are used to training
system in linguistic; however, non-conversational text is the knowledge base to be
filled including real-world information related to system target usage.
The performance of versatility and scalability in open domain with external
information knowledge combined with textual and structured data of QA system is
shown in Fig. 9.34. Datasets like Wikipedia, IMDB, TripAdvisor are used to gener-
ate conversation with real-world information and included a recommendation sys-
tem function.

Fig. 9.33 Architecture of knowledge grounded model

230 9 Major NLP Applications

Fig. 9.34 Response from conversation model knowledge grounded (Ghazvininejad et al. 2018)

After fact-based encoder, the response from this system becomes more meaning-
ful with related information and logical content. Based on this model, 23 million
open domains Twitter conversations and 1.1 million Foursquare tips are used to
achieve a significant improvement over the previous seq2seq model, and different
from the traditional content filling which add the predefined content and fill the
space in sentences.
It is well known that industrial QA systems are not limited to one model, many
models have been assembled into a language model for end-to-end dialogue. In this
architecture, the dialogue encoder is independent of fact encoder in the system, but
it is complementary to fact encoder when applied because facts require information
from dialogue history, especially to match context-dependent information bands.
There is intentional information as part of the response. From implementation per-
spective, multi-task learning is used to handle factual, non-factual, and autoencoder
tasks depending on intended work of the system. Multi-task learning can separate
two encoders independently while training the model, and after training on dialogue
dataset, the factual encoder part uses information retrieval (IR) to expand knowl-
edge base for more meaningful answers. In a way, a fact encoder is like a memory
network, which uses a store of relevant facts relevant to a particular problem. Once
the query contains a specific entity in the sentence, the sentence has been assigned
a specific name entity, the name entity recognizes (NER) by matching keywords or
linked entities, or even named entities and calculates its weight on input and dia-
logue history to generate a response. The original storage network model uses a bag
of words, but in this model the encoder directly converts input set to a vector unlike
storage network model.
Since the system is a fully neural-based data-driven model, they created an end-
to-end RNN system using a traditional seq2seq model, including long short-term
memory (LSTM) and Gate Recurrent Unit (GRU) model. For ensemble structures
such as two-class RNNs, constructing a simple GRU is usually faster than LSTM
model. The implementation of GRU means that the system does not have
Transformer’s attention mechanism or other invariants for neural network
computation.
9.4 Question-and-Answering Systems 231

9.4.2.1 AliMe QA System

AliMe is a module of Taobao app commercial QA product. The answer consists of

information retrieval (IR) and sequence-to-sequence-based generation models (Qiu
et al. 2017). The system reorders candidate’s response and uses attention mecha-
nism with context to select the best feedback to users. Using AliMe to replace online
human customer service for most known questions became a trend since it released
the first version. AliMe is a typical customer service QA system in e-commerce
industry that answers millions of questions automatically per day. According to a
survey of daily questions suggested by Taobao app users on shopping problems,
statistical data revealed that except most are business questions, 5% of the remain-
ing questions are chitchat. The 5% questions on genuine demands motivate AliMe
to add a common sense open domain chat function. It has satisfactory performance
as both IR and generation-based system since the pre-trained seq2seq model is used
twice for response generation and reranked with attention to a set of responses from
information retrieval with knowledge originate based and seq2seq previously gener-
ated. Figure 9.35 shows the Seq2Seq model with attention learning.
Since AliMe has two parts in generation that use different formats to obtain
information as abovementioned. IR-based models use a natural language word
matching knowledge base, Seq2seq generative model, and a scoring model to re-
score output responses as they are generated is word embeddings with vectors. The
IR-based dataset consists of 9,164,834 QA pairs conversations by real customers
from business domain. Researchers used an inverted index to match these 9 million
conversations with input sentences containing the same words and used BM25 to
measure the similarity between input sentences and the selected questions to obtain
answers to the most similar questions as answers to input questions. Traditional
IR-based systems avoid problems where the system cannot answer common sense
type questions.

Fig. 9.35 Seq2Seq model with attention

232 9 Major NLP Applications

Microsoft used GRU to reduce computational power and response time span as
well as AliMe selected RNN GRU to improve response efficiency. During optimiza-
tion, beam search in decoder assisted to identify the highest conditional probability
to obtain optimizer response sentence within parameters. The performance showed
that IR+generation+rerank approach by seq2seq model and mean probability scor-
ing function evaluation obtained the highest score as compared with other methods.

9.4.2.2 Xiao Ice QA System

Xiao Ice (Zhou et al. 2020) is an AI companion sentient chatbot with more than 660
million users worldwide which takes Intelligent Quotient (IQ) and Emotional
Quotient (EQ) in system design as shown in Fig. 9.37. It focused on chitchat com-
pared with other commonly used QA systems. According to Conversation-turns Per
Session (CPS) evaluation score, its grade is 23 higher than most chatbots. Figure 9.36
shows a system architecture of Xiao Ice.
Xiao Ice exists on 11 social media platforms including WeChat, Tencent QQ,
Weibo, and Facebook as an industrial application. It has equipped with two-way
text-to-speech voice and can process text, images, voice, and video clips for
message-based conversations. Also, its core chat function can distinguish common
or specific domain topic chat types so that it can change topics easily and automati-
cally provide users with deeper domain knowledge. A dialog manager is like an
NLP general pipeline with dialog management to path conversation states such as
core chat contents for open or special domains to process data from different sources
are tractable. The Global State Tracker is a vector of Xiao Ice's responses to analyze
text strings for entities and empathy. It is vacant and gradually filled with rounds of

Fig. 9.36 Xiao Ice system architecture

9.4 Question-and-Answering Systems 233

conversations. Dialogue strategies are primarily designed for long-term users, based
on their feedbacks to enhance interactions engagement, optimize personality with
two or three levels achievements. A trigger mechanism is to change topic when the
chatbot repeats or answers information that are always valid, or when a user's feed-
back is mundane within three words. Once the user's input has a predefined format,
a skill selection part is activated to process different input. For example, images can
be categorized into different task-oriented scenarios. If an image is food related, the
user will be taken to a restaurant display, like a task completion by personal assis-
tants in advising weather information or making reservations, etc.
Xiao Ice has a few knowledge graphs in the data layer as its original datasets
come from popular forums such as Instagram in English or Douban in Chinese.
These datasets are categorized as multiple topics with a small knowledge base as
possible answers. It also follows the rules of updating knowledge base through
machine learning when new topics emerge. It is noted that not all new entities or
topics are collected unless the entity is contextually relevant, or a topic has higher
popularity or freshness in the news for rankings. User's personal interests can be
adjusted individually.
However, with so many features that can include the core part Empathetic
Computing as an add-on, it is not a mandatory part of a full chatbot, but a functional
and compelling feature to compete with the industry. The core of Xiao Ice is a RNN
language model that creates open and special domain knowledge. Figures 9.37 and
9.38 show an RNN-based neural response generator with examples of inconsistent
responses generated by seq2seq model in Xiao Ice QA system, respectively.

Fig. 9.37 RNN-based neural response generator

234 9 Major NLP Applications

Fig. 9.38 Examples of inconsistent responses generated using a seq2seq model

In general, response generation in AliMe uses seq2seq model to generate natural

language and rerank the optimizer for user’s answer, whereas Xiao Ice also has a
candidate generator and candidate ranking list. For the generator, one is a sequential
model trained by a pair of datasets learning the dialog format, the other is querying
the knowledge graph to obtain entities for related information stored in knowledge
base. Candidate ranking includes semantic computation and Xiao Ice personality
for answer optimization with information retrieval, neural model, and knowledge
graph selection.

9.4.2.3 TransferTransfo Conversational Agents

A QA system consisted of traditional and current mainstream methods, the above

systems used Seq2Seq model responsible for both language model and candidate
response optimizer. Since neural network is a data-driven model, its performance
relies on huge amount of big data. Transformer is a model architecture forgone
recurrence but entrusted in attention mechanism entirely to draw global dependen-
cies between input and output based on attention mechanism.
Open AI GPT-2 transfer learning architecture has an outstanding feature to
include decoder part layers advantages for response generation. The masked self-
attention implemented on GPT-2 can generate the next word based on acquired
information, understand the known text, predict, or use experience to fill up the
blank for next word to match with the whole article meaning.
GPT-2 fine-tune 40G pure text to learn natural language semantics, syntax with
target usage and suitable dataset scalability for specific NLP tasks. TransferTransfo
(Wolf et al. 2018) is a GBP-2 variant using persona-chat dataset to fine-tune the
original model, its generated utterance changes from long text to dialogue format.
TransferTransfo prototype is a pre-trained model on document-level continuous
sequence and paragraphs with a wide range of information. After that, fine-tune
strengthen input representation and use a multi-task learning scheme for adjust-
ments. Every input token included word and position embedding during input
representation.
For Transfer Learning system dialogue example as in Fig. 9.39, personal-chat
datasets in real-world can define users’ backgrounds and their interests as topics
during communications. The contexts contained are meaningful conversation that
can reveal empirical improvements in discriminative language understanding tasks.
Thus, Transformer is an evolutional system to imitate human behavior and promote
neural network model.
9.4 Question-and-Answering Systems 235

Fig. 9.39 Example dialog from PERSONA-CHAT dataset

Exercises
9.1. What is Information Retrieval (IR) in NLP? State and explain why IR is vital
for the implementation of NLP applications. Give two NLP applications to
illustrate.
9.2. In terms of implementation technology of Information Retrieval (IR) sys-
tems, what are the major difference between traditional and latest IR systems.
Give one IR system implementation example to support your answer.
9.3. What is Discourse Segmentation? State and explain why Discourse
Segmentation is critical for the implementation of Information Retrieval (IR)
systems.
9.4. What is Text Summarization (TS) in NLP? State and explain the relationship
and differences between TS system and IR (Information Retrieval) systems.
9.5. What are two basic approaches of Text Summarization (TS)? Give live exam-
ples of TS systems to discuss how they work by using these two approaches.
9.6. What are the major differences between Single vs Multiple documentation
summarization systems? State and explain briefly the related technologies
being used in these TS systems.
9.7. What are the major characteristics of contemporary Text Summarization (TS)
systems as compared with traditional TS systems in the past century? Give
live example(s) to support your answer.
9.8. What is a QA system in NLP? State and explain why QA system is critical to
NLP. Give two live examples to support your answer.
9.9. Choose any two industrial used QA systems and compare their pros and cons
in terms of functionality and system performance.
9.10. What is Transformer technology? State and explain how it can be used for the
implementation of QA system. Use a live example to support your answer.
236 9 Major NLP Applications

References

Agarwal, N., Kiran, G., Reddy, R. S. and Rosé, C. P. (2011) Towards Multi-Document
Summarization of Scientific Articles: Making Interesting Comparisons with SciSumm. In Proc.
of the Workshop on Automatic Summarization for Different Genres, Media, and Languages,
Portland, Oregon, pp. 8– 15.
Agrawal, K. (2020) Legal case summarization: An application for text summarization. In Proc. Int.
Conf. Comput. Commun. Informat. (ICCCI), pp. 1–6.
Aharon, M., Elad, M. and Bruckstein, A. (2006). K-SVD: An algorithm for designing overcom-
plete dictionaries for sparse representation. IEEE Transactions on signal processing, 54(11):
4311-4322.
Aizawa A. (2003). An information-theoretic perspective of tf–idf measures. Information Processing
& Management, 39(1): 45-65.
Alami, N., Meknassi, M and Rais, N. (2015). Automatic texts summarization: Current state of the
art. Journal of Asian Scientific Research, 5(1), 1-15.
Alomari, A., Idris, N., Sabri, A., and Alsmadi, I. (2022) Deep reinforcement and transfer learning
for abstractive text summarization: A review. Comput. Speech Lang. 71: 101276.
Banko, M., Mittal, V. O. and Witbrock, M. J. (2000) Headline Generation Based on Statistical
Translation. ACL 2000, pp. 318-325.
Baumel, T., Eyal, M. and Elhadad, M. (2018) Query Focused Abstractive Summarization:
Incorporating Query Relevance, Multi-Document Coverage, and Summary Length Constraints
into seq2seq Models. CoRR abs/1801.07704.
Barzilay, R., McKeown, K. and Elhadad, M. (1999) Information fusion in the context of multi-
document summarization. In Proceedings of ACL '99, pp. 550–557.
Baxendale, P. (1958) Machine-made index for technical literature - an experiment. IBM Journal of
Research Development, 2(4):354-361.
Brown TB, Mann B, Ryder N, et al (2020) Language models are few-shot learners. Adv Neural Inf
Process Syst 2020-Decem:pp.3–63
Carbonell, J. and Goldstein, J. (1998) The use of MMR, diversity-based reranking for reordering
documents and producing summaries. In Proceedings of SIGIR '98, pp. 335-336, NY, USA.
Chen, J. and Zhuge H. (2018) Abstractive text-image summarization using multi-modal atten-
tional hierarchical RNN. In Proc. Conf. Empirical Methods Natural Lang. Process., Brussels,
Belgium, pp. 4046–4056.
Chen H, Liu X, Yin D, Tang J (2017) A Survey on Dialogue Systems. ACM SIGKDD Explor
Newsl 19:25–35. https://doi.org/10.1145/3166054.3166058
Cheng, H. T. et al. (2016). Wide & deep learning for recommender systems. In Proceedings of the
1st workshop on deep learning for recommender systems, pp. 7-10.
Chi, L. and Hu, L. (2021) ISKE: An unsupervised automatic keyphrase extraction approach using
the iterated sentences based on graph method. Knowl. Based Syst. 223: 107014.
Chopra, S., Auli, M. and Rush, A. M. (2016) Abstractive Sentence Summarization with Attentive
Recurrent Neural Networks. HLT-NAACL 2016, pp. 93-98.
Cho K, Van Merriënboer B, Gulcehre C, et al (2014) Learning phrase representations using RNN
encoder-decoder for statistical machine translation. EMNLP 2014 - 2014 Conf Empir Methods
Nat Lang Process Proc Conf 1724–1734. https://doi.org/10.3115/v1/d14-1179
Church, K. W. (2017). Word2Vec. Natural Language Engineering, 23(1): 155-162.
CNN-DailyMail (2022) CNN/Daily-Mail Datasets. https://www.kaggle.com/datasets/gowrishan-
karp/newspaper-text-summarization-cnn-dailymail. Accessed 9 Aug 2022.
Columbia (2022). Columbia Newsblaster. http://newsblaster.cs.columbia.edu. Accessed 14
June 2022.
Croft, W. B. & Harper, D. J. (1979). Using probabilistic models of document retrieval without
relevance information. Journal of documentation, 35(4): 285-295.
Cui Y, Huang C, Lee R (2020) AI Tutor : A Computer Science Domain Knowledge Graph-Based
QA System on JADE platform. Int J Ind Manuf Eng 14:603–613
References 237

Dong, Y. (2018) A Survey on Neural Network-Based Summarization Methods. CoRR

abs/1804.04589.
DUC (2022) DUC Dataset. https://paperswithcode.com/dataset/duc-2004. Accessed 9 Aug 2022.
Edmundson, H. P. (1969) New Methods in Automatic Extracting. Journal of ACM 16(2): 264-285.
Elsaid, A., Mohammed, A., Ibrahim, L. F., Mohammed and Sakre, M. (2022) A Comprehensive
Review of Arabic Text Summarization. IEEE Access 10: 38012-38030.
Erkan, G. and Radev. D. R. (2004) LexRank: Graph-based Lexical Centrality as Salience in Text
Summarization. J. Artificial Intelligent Research 22: 457-479.
Evans, D. K. (2005) Similarity-based multilingual multidocument summarization. Technical
Report CUCS-014- 05, Columbia University.
Farsi, M., Hosahalli, D., Manjunatha, B., Gad, I., Atlam, E., Ahmed, A., Elmarhomy, G.,
Elmarhoumy and Ghoneim, O. (2021) Parallel genetic algorithms for optimizing the SARIMA
model for better forecasting of the NCDC weather data, Alexandria Eng. J., 60(1): 1299–1316.
Florian, R. (2002). Named entity recognition as a house of cards: Classifier stacking. In
Proceedings of the 6th conference on Natural language learning (COLING-02). https://doi.
org/10.3115/1118853.1118863.
Fraser, B. (1999). What are discourse markers? Journal of pragmatics, 31(7): 931-952.
Fukumoto, J. (2004) Multi-Document Summarization Using Document Set Type Classification. In
Proc. of NTCIR- 4, Tokyo, pp. 412-416.
Ghazvininejad M, Brockett C, Chang MW, et al (2018) A knowledge-grounded neural conversa-
tion model. 32nd AAAI Conf Artif Intell AAAI 2018 5110–5117
Gigaword (2022) Gigaword Datasets. https://huggingface.co/datasets/gigaword. Accessed 9
Aug 2022.
Gong, Y. and Liu, X. (2001) Generic Text Summarization Using Relevance Measure and Latent
Semantic Analysis. SIGIR 2001, pp. 19-25.
Google (2022) Google News. http://news.google.com. Accessed 14 June 2022.
Jain, D., Borah, M. D. and Biswas, A. (2020) Fine-tuning textrank for legal document summari-
zation: A Bayesian optimization-based approach. In Proc. Forum Inf. Retr. Eval., Hyderabad
India, pp. 41–48.
Jing, H. (2000) Sentence Reduction for Automatic Text Summarization. In Proceedings of the 6th
Applied Natural Language Processing Conference, Seattle, USA, pp. 310-315.
Halliday, M. A. K. and Hasan, R. (1976) Cohesion in English, Longman, London.
Hariharan, S., Ramkumar, T., Srinivasan, R. (2013) Enhanced Graph Based Approach for Multi
Document Summarization,” The International Arab Journal of Information Technology, 10 (4):
334-341.
Hasan, R. (1984) Coherence and Cohesive Harmony. In: Flood James (Ed.), Understanding
Reading Comprehension: Cognition, Language and the Structure of Prose. Newark, Delaware:
International Reading Association, pp. 181-219.
Hearst, M. A. (1997). Text tiling: Segmenting text into multi-paragraph subtopic passages.
Computational linguistics, 23(1): 33-64.
Hovy, E., and Lin, C. Y. (1999) Automated Text Summarization in SUMMARIST, In: Inderjeet
Mani and Mark T. Maybury (Eds.). Advances in Automatic Text Summarization, MIT Press,
pp. 18-24.
Keneshloo, Y., Shi, T., Ramakrishnan, N. and Reddy, C. K. (2019). Deep reinforcement learning
for sequence-to-sequence models. IEEE transactions on neural networks and learning systems,
31(7): 2469-2489.
Kupiec, J., Pedersen, J. and Chen, F. (1995) A Trainable Document Summarizer. In Proc. of the
18th annual international ACM SIGIR conference on Research and development in informa-
tion retrieval, pp. 68-73.
Langville, A. N. and Meyer, C. D. (2006) Google's PageRank and beyond - the science of search
engine rankings. Princeton University Press 2006, ISBN 978-0-691-12202-1, pp. I-VII, 1-224.
LCSTS (2022) LCSTS Dataset. https://www.kaggle.com/xuguojin/lcsts-dataset. Accessed 9
Aug 2022.
238 9 Major NLP Applications

Luhn, H. P. (1958) The Automatic Creation of Literature Abstracts. IBM J. Res. Dev. 2(2): 159-165.
Mahalakshmi, P. and Fatima, N. S. (2022) Summarization of Text and Image Captioning in
Information Retrieval Using Deep Learning Techniques. IEEE Access 10: 18289-18297.
Malki, Z., Atlam, E., Dagnew, G., Alzighaibi, A., Ghada, E. and Gad I. (2020) Bidirectional resid-
ual LSTM-based human activity recognition, Comput. Inf. Sci., 13(3):1–40.
Mani I. and Bloedorn, E. (1997) Multi-document summarization by graph search and matching.
AAAI/IAAI, vol. cmplg/ 9712004, pp. 622-628, 1997.
Mihalcea, R. and Tarau, P. (2004) TextRank: Bringing Order into Text. EMNLP 2004: 404-411
Nallapati, R., Zhai, F. and Zhou. B. (2017) Summarunner: A recurrent neural network-based
sequence model for extractive summarization of documents. AAAI 2017: 3075-3081.
arXiv:1611.04230
NewsInEssence (2022). NewsInEssence News. http://NewsInEssence.com. Accessed 14 June 2022.
NYT (2022) NYT Dataset. https://www.kaggle.com/datasets/manueldesiretaira/dataset-for-text-
summarization. Accessed 9 Aug 2022.
Mikolov T, Sutskever I, Chen K, et al (2013) Distributed representations of words and phrases and
their compositionality. Adv Neural Inf Process Syst 1–9
Ozsoy, M. G., Alpaslan, F. N. and Cicekli, I. (2011) Text summarization using Latent Semantic
Analysis. J. Inf. Sci. 37(4): 405-417.
Pervin S. and Haque M. (2013) Literature Review of Automatic Multiple Documents Text
Summarization, International Journal of Innovation and Applied Studies, 3(1) 121-129.
Qiu M, Li F-L, Wang S, et al (2017) AliMe Chat: A Sequence to Sequence and Rerank based Chatbot
Engine. In: Proceedings of the 55th Annual Meeting of the Association for Computational
Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Stroudsburg,
PA, USA, pp 498–503
Rath, G. J., Resnick A. and Savage, T. R. (1961) Comparisons of four types of lexical indicators
of content. Journal of the American Society for Information Science and Technology, 12(2):
126-130.
Reimers, N., and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-
networks, arXiv preprint arXiv:1908.10084.
Robertson, A. M. & Willett, P. (1998). Applications of n-grams in textual information systems.
Journal of Documentation, 54(1): 8-67.
Rothkopf, E. Z. (1971). Incidental memory for location of information in text. Journal of verbal
learning and verbal behavior, 10(6), 608-613.
Sadler, L. & Spencer, A. (2001). Syntax as an exponent of morphological features. In Yearbook of
morphology 2000, pp. 71-96. Springer.
Salton, G. (1989) Automatic Text Processing: the transformation, analysis, and retrieval of infor-
mation by computer. Addison- Wesley Publishing Company, USA.
Salton G, Wong A and Yang C S. (1975) A vector space model for automatic indexing.
Communications of the ACM, 18(11): 613-620.
See, A., Liu, P. J. and Manning, C. D. (2017) Get To The Point: Summarization with Pointer-
Generator Networks. ACL (1) 2017: 1073-1083.
Svore, K. M., Vanderwende L. and Burges, J.C. (2007) Enhancing Single document Summarization
by Combining RankNet and Third-party Sources. In Proc. of the Joint Conference on Empirical
Methods in Natural Language Processing and Computational Natural Language Learning,
pp. 448–457.
Taboada, M. & Mann, W. C. (2006). Applications of rhetorical structure theory. Discourse studies,
8(4): 567-588.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., Polosukhin,
I. (2017) Attention is All you Need. NIPS 2017: 5998-6008. arXiv:1706.03762.
Wan, X. (2008) An Exploration of Document Impact on Graph-Based Multi-Document
Summarization. Proc. of the Conference on Empirical Methods in Natural Language
Processing, Association for Computational Linguistics, pp. 755–762.
References 239

Weeds, J., Weir, D. and McCarthy, D. (2004). Characterising measures of lexical distributional sim-
ilarity. In COLING 2004: Proceedings of the 20th international conference on Computational
Linguistics, pp. 1015-1021.
Weibo (2022) Sina Weibo official site. https://weibo.com. Accessed 29 Sept 2022.
Whissell, J. S. & Clarke, C. L. (2011). Improving document clustering using Okapi BM25 feature
weighting. Information retrieval, 14(5): 466-487.
Wolf T, Sanh V, Chaumond J, Delangue C (2018) TransferTransfo: A Transfer Learning Approach
for Neural Network Based Conversational Agents
Yang, R., Bu, Z. and Xia, Z. (2012) Automatic Summarization for Chinese Text Using Affinity
Propagation Clustering and Latent Semantic Analysis. WISM 2012, pp. 543-550
Yang, J., Yi, X., Cheng, D. Z., Hong, L., Li, Y. and Wong, S. (2020). Mixed negative sampling
for learning two-tower neural networks in recommendations. In Proceedings of the Web
Conference 2020, pp. 441-447.
You O., Li W. and Lu, Q. (2009) An Integrated Multi-document Summarization Approach based
on Word Hierarchical Representation. In Proc. of the ACL-IJCNLP Conference, Singapore,
pp. 113–116.
Zhang, Y., Jin, R. and Zhou, Z. H. (2010). Understanding bag-of-words model: a statistical frame-
work. International Journal of Machine Learning and Cybernetics, 1(1): 43-52.
Zhou L, Gao J, Li D, Shum H-Y (2020) The Design and Implementation of XiaoIce, an Empathetic
Social Chatbot. Comput Linguist 46:53–93. https://doi.org/10.1162/coli_a_00368
Zhuang, S. & Zuccon, G. (2021). TILDE: Term independent likelihood moDEl for passage re-
ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and
Development in Information Retrieval, pp. 1483-1492.
Zhu T. and Zhao, X. (2012) An Improved Approach to Sentence Ordering For Multi-document
Summarization. IACSIT Hong Kong Conferences, IACSIT Press, Singapore, vol. 25, pp. 29-33.
Part II
Natural Language Processing Workshops
with Python Implementation in 14 Hours
Chapter 10
Workshop#1 Basics of Natural Language
Toolkit (Hour 1–2)

10.1 Introduction

Part 2 of this book will provide seven Python programming workshops on how each
NLP core component operates and integrates with Python-based NLP tools includ-
ing NLTK, spaCy, BERT, and Transformer Technology to construct a Q&A chatbot.
Workshop 1 will explore NLP basics including:
1. Concepts and installation procedures
2. Text processing function with examples using NLTK
3. Text analysis lexical dispersion plot in Python
4. Tokenization in text analysis
5. Statistical tools for text analysis

10.2 What Is Natural Language Toolkit (NLTK)?

NLTK (Natural Language Toolkit 2022) is one of the earliest Python-based NLP
development tool invented by Prof. Steven Bird and Dr. Edward Loper in the
Department of Computer and Information Science of University of Pennsylvania
with their classical book Natural Language Processing with Python published by
O'Reilly Media Inc. in 2009 (Bird et al. 2009). There are over 30 universities in
USA and 25 countries using NLTK for NLP related courses until present. This book
is considered as bible for anyone who wishes to learn and implement NLP applica-
tions using Python.
NLTK offers user-oriented interfaces with over 50 corpora and lexical resources
such as WordNet, a large lexical database of English. Nouns, verbs, adjectives, and
adverbs are grouped into sets of cognitive synonyms (synsets); each expresses a

© The Author(s), under exclusive license to Springer Nature Singapore Pte 243
Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_10
244 10 Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)

distinct concept which is an important lexical database in NLP developed by

Princeton University since 1980.
Other lexical databases and corpora are Penn Treebank Corpus, Open Multilingual
Wordnet, Problem Report Corpus, and Lin’s Dependency Thesaurus.
NLTK contains statistical-based text processing libraries of five fundamental
NLP enabling technologies and basic semantic reasoning tools including (Albrecht
et al. 2020; Antic 2021; Arumugam and Shanmugamani 2018; Hardeniya et al.
2016; Kedia and Rasu 2020; Perkins 2014):
• Word tokenization
• Stemming
• POS tagging
• Text classification
• Semantic analysis

10.3 A Simple Text Tokenization Example Using NLTK

Let us look at NLTK text tokenization using Jupyter Notebook (Jupyter 2022;
Wintjen and Vlahutin 2020) as below:

In[1] # Import NLTK package

import nltk

In[2] # Create a sample utterance 1 (utt1)

utt1 = "At every weekend, early in the morning. I drive my car to the car
center for car washing. Like clock-work."

In[3] # Display utt1

utt1
Out[3] 'At every weekend, early in the morning. I drive my car to the car center
for car washing. Like clock-work.'

In[4] # Create utterance tokens (utokens)

utokens = nltk.word_tokenize(utt1)

In[5] # Display utokens

utokens

Out[5] ['At', 'every', 'weekend', ',', 'early', 'in', 'the', 'morning', '.', 'I', 'drive',
'my', 'car', 'to', 'the', 'car', 'center', 'for', 'car', 'washing', '.', 'Like', 'clock-
work', '.']
10.4 How to Install NLTK? 245

10.4 How to Install NLTK?

Step 1 Install Python 3.X

Step 2 Install NLTK
2.1 Start CMD or other command line tool
2.2 Type pip install nltk
Figure 10.1 shows a screenshot of NLTK installation process.
Step 3 Install NLTK Data
Once NLTK is installed into Python, download NLTK data.
3.1 Run Python
3.2 Type the following to activate a NLTK downloader
import nltk
nltk.download()
Note: nltk.downloader() will invoke NLTK downloader automatically, a separate
window-based downloading module for users to download four NLP databanks into
their Python machines. They include (1) Collection libraries, (2) Corpora, (3)
Modules, and (4) other NLP packages. Figures 10.2, 10.3, and 10.4 show screen-
shots of NTLK downloader for Collection, Corpora, and NTLK models installations.

Fig. 10.1 Screenshot of NLTK installation process

Fig. 10.2 Screenshot of NTLK downloader of Collection library

246 10 Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)

Fig. 10.3 Screenshot of NTLK downloader of Corpora library

Fig. 10.4 Screenshot of NTLK downloader of NLTK models

10.5 Why Using Python for NLP?

Python toolkit and packages overtook C, C++, Java especially in data science, AI,
and NLP software development since 2000 (Albrecht et al. 2020; Kedia and Rasu
2020). There are several reasons to drive the changes because:
• it is a generic language without specific area unlike other language such as Java
and JavaScript specifically designed for web applications and websites
developments.
• it is easier to learn and user-friendly compared with C and C++ especially for
non-computer science students and scientists.
• its lists and list-processing data types provide an ideal environment for NLP
modelling and text analysis.
10.5 Why Using Python for NLP? 247

A Python program performs tokenization task to process text is shown below:

In[6] # Define utterance 2 (utt2)

utt2 = "Hello world. How are you?"

In[7] # Using split() method to split it into word tokens

utt2.split()

Out[7] ['Hello', 'world.', 'How', 'are', 'you?']

In[8] # Check the no of word tokens

nwords = len(utt2.split())
print ("'Hello world. How are you?' contains ",nwords," words.")

Out[8] 'Hello world. How are you?' contains 5 words.

Python codes perform word number counts from literature Alice’s Adventures in
Wonderland by Lewis Carroll (1832–1898) as below:

In[9] # Define method to count the number of word tokens in text file (cwords)
def cwords(literature):
try:
with open(literature, encoding='utf-8') as f_lit:
c_lit = f_lit.read()
except FileNotFoundError:
err = "Sorry, the literature " + literature + " does not exist."
print(err)
else:
w_lit = c_lit.split()
nwords = len(w_lit)
print("The literature " + literature + " contains " + str(nwords) +
" words.")

literature = 'alice.txt'
cwords(literature)

Out[9] The literature alice.txt contains 29465 words

This workshop has extracted four famous literatures from Project Gutenberg (2022):
1. Alice’s Adventures in Wonderland by Lewis Carroll (1832–1898) (alice.txt)
2. Little Women by Louisa May Alcott (1832–1888) (little_women.txt)
3. Moby Dick by Herman Melville (1819–1891) (moby_dick.txt)
4. The Adventures of Sherlock Holmes by Sir Arthur Conan Doyle (1859–1930)
(Adventures_Holmes.txt)
248 10 Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)

In[10] cwords('Adventures_Holmes.txt')
Out[10] The literature Adventures_Holmes.txt contains 107411 words.

10.6 NLTK with Basic Text Processing in NLP

NLTK are Python tools and methods to learn and practice starting from basic text
processing in NLP. They include:
• Text processing as lists of words
• Statistics on text processing
• Simple text analysis
NLTK provides 9 different types of text documents from classic literatures, Bible
texts, famous public speeches, news, and articles with personal corpus for text pro-
cessing. Let us start and load these text documents.

In[11] # Let's load some sample books from the nltk databank
import nltk
from nltk.book import *

Out[11]
10.7 Simple Text Analysis with NLTK 249

In[12] # Display the list of sample books

texts()
Out[12]

In[13] # Check text1

text1
Out[13] <Text: Moby Dick by Herman Melville 1851>

In[14] # To know more about text1, check this

text1?
In[15] # Import word_tokenize as wtoken
from nltk.tokenize import word_tokenize

# Open Adventures_Holmes.txt and performs tokenization

fholmes = open("Adventures_Holmes.txt","r",encoding="utf-8").read()
wtokens = word_tokenize(fholmes)
tholmes=nltk.text.Text(wtokens)

10.7 Simple Text Analysis with NLTK

Text analysis is to study a particular word or phrase occurred in a text document

such as literature or public speeches. NLTK has a "concordance()" function differ-
ent from ordinary search function. It does not only indicate occurrence but also
reveal neighboring words and phrases. Let us try texts examples from The Adventures
of Sherlock Holmes (Doyle 2019)
250 10 Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)

In[16] # Check concordance of word "Sherlock "

tholmes.concordance("Sherlock")
Out[16]

The above example shows all Sherlock occurrences indicating that Sherlock is
a special word linked with surname Holmes in text document.

Let’s look at word usage of extreme from the same literature:

In[17] # Check concordance of word "extreme"

tholmes.concordance("extreme")
Out[17]

Concordance techniques are means to learn grammars, words or phrases

called Use of English, also called Learn by Examples. In this example, we
learnt how to use the word extreme in various situations and scenarios.
10.7 Simple Text Analysis with NLTK 251

In[18] tholmes.similar("extreme")
Out[18] dense gathering

In[19] # Check concordance of word "extreme" in text2

text2.concordance("extreme")
Out[19]

In[20] # Check similar word "extreme" in text2

text2.similar("extreme")
Out[20]

In[21] # Check concordance word "extreme" in text4

text4.concordance ("extreme")
Out[21]

In[22] # Check similar word "extreme" in text4

text4.similar("extreme")
Out[22]

It showed that word usage of extreme varies by authors and text types e.g. it
has different styles in The Adventures of Sherlock Holmes as compared
with usage in Sense and Sensibility by Jane Austin (1775-1817) which is
more vivid but has standard and fixed usage in Inaugural Address Corpus.

The common_contexts() method is to examine contexts shared by two or more

words. The Adventures of Sherlock Holmes is used with common contexts of two
words extreme and huge.
First, call common contexts() function from object tholmes.

In[23] # Check common contexts on tholmes

tholmes.common_contexts(["extreme","huge"])
Out[23] No common contexts were found
252 10 Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)

which means after analyzing extreme and huge in The Adventures of Sherlock
Holmes, no common context meaning can be found.
Call concordance() function of these two words and check against the extracted
patterns as shown below:

In[24] # Check concordance word "extreme" in tholmes

tholmes.concordance("extreme")

Out[24]

In[25] # Check concordance word "huge" in tholmes

tholmes.concordance("huge")

Out[25]

Can you see how important it is in

1. NLP?
2. Use of English and technical writing?

Workshop 1.1 Simple Text Processing using NLTK

1. Try to use concordance(), similar(), and common_contexts() functions to
look for two more frequently used words usage.
2. Compare their usages from four sources: Moby Dick, Sense and Sensibility,
Inaugural Address Corpus, and Wall Street Journal.
3. Are there any pattern(s)?
4. What are their differences in Use of English?
10.8 Text Analysis Using Lexical Dispersion Plot 253

10.8 Text Analysis Using Lexical Dispersion Plot

Text analysis was learnt to study word patterns and common contexts in previous
workshop.
Dispersion Plot in Python NLTK is to identify occurrence frequencies of key-
words from the whole document.

10.8.1 What Is a Lexical Dispersion Plot?

Dispersion is quantification of each point deviation from the mean value in basic
statistics.
NLTK Dispersion Plot produces a plot showing words distribution throughout
the text. Lexical dispersion is used to indicate homogeneity of words (word tokens)
occurred in the corpus (text document) achieved by the dispersion_plot() in NLTK.
To start, let us use NLTK book object to call function dispersion_plot().
Note: requires pylab installation prior this function.
The following example uses text1 to verify basic information about
dispersion_plot().

In[26] text1.dispersion_plot?

10.8.2 Lexical Dispersion Plot Over Context Using Sense

and Sensibility

Are there any lexical patterns for positive words such as good, happy, and strong
versus negative words such as bad, sad, or weak in literature?

Workshop 1.2 Lexical Dispersion Plot over Context using Sense and
Sensibility
Use dispersion_plot to plot Lexical Dispersion Plot keywords: good, happy,
strong, bad, sad, and weak from Sense and Sensibility.
1. Study any lexical pattern between positive and negative keywords.
2. Check these patterns against Moby Dick to see if this pattern occurs and
explain.
3. Choose two other sentiment keywords to see if this pattern remains valid.
254 10 Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)

In[27] text2.dispersion_plot(["good", "happy", "strong", "bad", "sad", "weak"])

Out[27]

10.8.3 Lexical Dispersion Plot Over Time Using Inaugural

Address Corpus

Lexical usage is to analyze word pattern changes in written English over time. The
Inaugural Address Corpus addressed by US presidents of past 220 years is a text
document in NLTK book library to study lexical dispersion plot patterns changes on
keywords war, peace, freedom, and united for this workshop.

Workshop 1.3 Lexical Dispersion Plot over Time using Inaugural Address
Corpus
1. Use dispersion_plot to invoke Lexical Dispersion Plot for Inaugural
Address Corpus.
2. Study and explain lexical pattern changes for keywords America, citizens,
democracy, freedom, war, peace, equal, united.
3. Choose any two meaningful keywords and check for lexical pattern
changes.
10.9 Tokenization in NLP with NLTK 255

In[28] text4.dispersion_plot(["America" ,"citizens" ,"democracy",

"freedom", "war", "peace", "equal", "united"])
Out[28]

10.9 Tokenization in NLP with NLTK

10.9.1 What Is Tokenization in NLP?

A token can be words, part of a word, characters, numbers, punctuations, or sym-

bols. It is a principal constituent and complex NLP task due to every language has
own grammatical constructions to generate grammatic and syntactic rules.
Tokenization is an NLP process of dividing sentences/utterances from a text,
document, or speeches into chunks called tokens. By using tokenization, a vocabu-
lary from a document or corpus can be formed. Tokenization for sentence/utter-
ances Jane lent $100 to Peter early this morning is shown in Fig. 10.5.

Jane lent $100 to Peter early this morning.

Tokenization

Jane lent $ 100 to Peter early this morning .

Fig. 10.5 Tokenization example of a sample utterance “Jane lent $100 to Peter early this morning”
256 10 Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)

NLTK provides flexibility to tokenize any string of text using tokenize() function
as shown below:

In[29] # Create utterance 3 (utt3) and performs tokenization

utt3 = 'Jane lent $100 to Peter early this morning.'
wtokens = nltk.word_tokenize(utt3)
wtokens

Out[29] ['Jane', 'lent', '$', '100', 'to', 'Peter', 'early', 'this', 'morning', '.']

10.9.2 Different Between Tokenize() vs Split()

Python provides split() function to split a sentence of text into words as recalled in
Sect. 10.1 Let us see how it works with Tokenize() function.

In[30] # Use split() to perform word tokenization

words = utt3.split()
words
Out[30] ['Jane', 'lent', '$100', 'to', 'Peter', 'early', 'this', 'morning.']

Why are they different?

How is it important in
1. NLP?
2. Meanings?

Workshop 1.4 Tokenization on The Adventures of Sherlock Holmes with

NLTK
1. Read Adventures_Holmes.txt text file.
2. Save contents into a string object "holmes_doc".
3. Use split() to cut it into list object "holmes".
4. Count total number of words in the document.
5. Tokenize document using NLTK tokenize() function.
6. Count total number of tokens.
7. Compare the two figures.
(The file open part is provided to start with.)

In[31] # Workshop 10.4 Solution

with open('Adventures_Holmes.txt', encoding='utf-8') as f_lit:

dholmes = f_lit.read()
# Count number of words in the literature
…
10.9 Tokenization in NLP with NLTK 257

NLTK provides a simple way to count total number of tokens in a Text Document
using len() in NLTK package.
Try len(tholmes) will notice:

In[32] len(tholmes)

Out[32] 128366

10.9.3 Count Distinct Tokens

Text analysis is to study distinct words, or vocabulary occurred in a text document.

When text document is tokenized as token objects, Python can group them easily
into a set of distinct object using Set() method.
Set() in Python is to extract distinct objects of any types from a list of objects
with repeated instances.
Try the following using The Adventures of Sherlock Holmes will notice:

In[33] tholmes?

In[34] set(tholmes)

Out[34]

In[35] len(set(tholmes))

Out[35] 10048

This example showed that The Adventures of Sherlock Holmes contains 128,366
tokens, i.e. words and punctuations, and 10,048 distinct tokens, or types. Try other
literatures and see vocabulary can be learnt from these great literatures.
258 10 Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)

The following example shows how to sort distinct tokens using sorted() function.

In[36] sorted(set(tholmes)

Out[36]

Since books are tokenized in NLTK as a list book object, contents can be accessed
by using list indexing method as below:

In[37] # Access the First 20 tokens

tholmes[1:20]
Out[37]

In[38] # Access the MIDDLE content

tholmes[100:150]
Out[38]

In[39] # Aceess from the END

tholmes[-20:]
Out[39]

10.9.4 Lexical Diversity

10.9.4.1 Token Usage Frequency (Lexical Diversity)

Token usage frequency, also called Lexical Diversity is to divide the total number of
tokens by total number of token types as shown:

In[40] len(text1)/len(set(text1))
Out[40] 13.502044830977896
10.9 Tokenization in NLP with NLTK 259

In[41] len(text2)/len(set(text2))
Out[41] 20.719449729255086

In[42] len(text3)/len(set(text3))
Out[42] 16.050197203298673

In[43] len(text4)/len(set(text4))
Out[43] 15.251970074812968

Python codes above analyze token usage frequency of four literatures: Moby Dick,
Sense and Sensibility, Book of Genesis, and Inaugural Address Corpus. It has
usage frequency range from 13.5 to 20.7. What are the implications?

10.9.4.2 Word Usage Frequency

There are many commonly used words in English. The following example shows
the pattern of word usage frequency for the from above literatures.

In[44] text1.count('the')
Out[44] 13721

In[45] text1.count('the')/len(text1)*100
Out[45] 5.260736372733581

In[46] text2.count('the')/len(text2)*100
Out[46] 2.7271571452788606

In[47] text3.count('the')/len(text3)*100
Out[47] 5.386024483960325

In[48] text4.count('the')/len(text4)*100
Out[48] 6.2491416014283745

1. Are there any patterns found from these literatures?

2. Use other words of, a, I to study if there exists other pattern(s).
260 10 Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)

10.10 Basic Statistical Tools in NLTK

10.10.1 Frequency Distribution: FreqDist()

Text analysis is a NTLK tool that can tokenize a string or a book of text document.
Frequency Distribution—FreqDist() is an initial built-in method in NLTK to ana-
lyze frequency distribution of every token type in a text document.
Inaugural Address Corpus is used as an example to show how it works.

In[49] text4
Out[49] <Text: Inaugural Address Corpus>

In[50] FreqDist?

In[51] fd4 = FreqDist(text4)

In[52] fd4
Out[52] FreqDist({'the': 9555, ',': 7275, 'of': 7169, 'and': 5226, '.': 5011, 'to': 4477, 'in':
2604, 'a': 2229, 'our': 2062, 'that': 1769, ...})

10.10.1.1 FreqDist() as Dictionary Object

It is noted that FreqDist() will return key-value pairs from Dictionary object to
reflect the Key that store Token Type name and the Value which are corresponding
frequency of occurrence in a text. Since FreqDist() returns a Dictionary object,
keys() can be used to return the list of all Token Types as shown below.

In[53] token4 = fd4.keys()

token4
Out[53]
10.10 Basic Statistical Tools in NLTK 261

10.10.1.2 Access FreqDist of Any Token Type

Use list item access method to obtain frequency distribution of any token types. FD
value of token type for the is shown below.

In[54] fd4['the']
Out[54] 9555

1. What are five common word types (token types without punctuations) in any
text document?
2. Use FreqDist() to verify.

10.10.1.3 Frequency Distribution Plot from NLTK

NLTK is a useful tool to study the top frequency distribution token types for any
document using plot() function with FreqDist() method. FreqDist.plot() can also
plot the top XX frequently used token types in a text document.
1. Use fd3 to study FreqDist.plot() documentation using fd3.plot().
2. Plot top 30 frequently used token types from the Book of Genesis (Non-
Cumulative mode).
3. Do the same plot with Cumulative mode.

In[55] fd4.plot?

In[56] fd4.plot (30,cumulative=False)

Out[56]
262 10 Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)

In[57] fd4.plot (30,cumulative=True)

Out[57]

Workshop 1.5 Frequency Distribution Analysis on Classics Literatures

1. What are top 5 frequently used word types in the Book of Genesis (ignore
punctuations)?
2. Will it be the same with other great literatures?
3. Verify against (1) Moby Dick, (2) Sense and Sensibility, and (3) Inaugural
Address Corpus to see if they have the same patterns. Why or why not?
4. Why the study of common word types is also important in cryptography?

10.10.2 Rare Words: Hapax

Hapaxes are words that occur only once in a body of work whether it is a publication
or an entire language.
Ancient texts are full of hapaxes. For instance, in Shakespeare's Love's Labour's
Lost contains hapax honorificabilitudinitatibus which means able to achieve honors.
NLTK provides method hapaxes() under FreqDist object to list out all word types
that occurred once in text document.
Try FreqDist() with The Adventures of Sherlock Holmes and see how useful it is.

In[58] tholmes
Out[58] <Text: The Adventures of Sherlock Holmes by Arthur Conan...>

In[59] fd = FreqDist(tholmes)
10.10 Basic Statistical Tools in NLTK 263

In[60] hap = fd.hapaxes()

hap[1:50]
Out[60]

Workshop 1.6 Learn Vocabulary using Hapaxes

Hapaxes are useful for us to learn vocabulary contain more than 12 characters.
The following example uses hapaxes() with Python in-line function to
implement [w for w in hap1 if len(w) > 12]:
1. Create the Python script and extract vocabulary contain more than 12
characters from Moby Dick.
2. Select five meaningful vocabularies with their meanings.
3. Check with The Adventures of Sherlock Holmes to learn another five
vocabularies.
(Python script to generate vocabulary with over 12 characters are given.)

In[61] # Workshop 10.6 Solutions

voc12 = [w for w in hap if len(w) > 12]

voc12
Out[61]

10.10.3 Collocations

10.10.3.1 What Are Collocations?

A collocation is a work grouping for a set of words usually appeared together to

convey semantic meanings. The word collocation is originated from Latin word
meaning place together and was first introduced by Prof. John R Firth (1890–1960)
with his famous quote “You shall know a word by the company it keeps.”
264 10 Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)

There are many cases in English where strong collocations are word pairings
always appear together such as make and do, e.g. You make a cup of coffee, but you
do your work.
Collocations are frequently used in business settings when nouns are combined
with verbs or adjectives, e.g. set up an appointment, conduct a meeting, set the
price etc.

10.10.3.2 Collocations in NLTK

NLTK also provides a build-in method to handle collocations using NLTK

method—collocations().
The following example is to generate collocations lists from Moby Dick, Sense
and Sensibility, Book of Genesis, and Inaugural Address Corpus.
Let us look at some extracted collocation terms:

In[62] text1.collocations()

Out[62]

In[63] text2.collocations()

Out[63]

In[64] text3.collocations()

Out[64]

In[65] text4.collocations()

Out[65]
References 265

References

Albrecht, J., Ramachandran, S. and Winkler, C. (2020) Blueprints for Text Analytics Using
Python: Machine Learning-Based Solutions for Common Real World (NLP) Applications.
O’Reilly Media.
Antic, Z. (2021) Python Natural Language Processing Cookbook: Over 50 recipes to understand,
analyze, and generate text for implementing language processing tasks. Packt Publishing.
Arumugam, R. and Shanmugamani, R. (2018) Hands-On Natural Language Processing with
Python: A practical guide to applying deep learning architectures to your NLP applications.
Packt Publishing.
Bird, S., Klein, E., and Loper, E. (2009). Natural language processing with python. O'Reilly.
Doyle, A. C. (2019) The Adventures of Sherlock Holmes (AmazonClassics Edition).
AmazonClassics.
Gutenberg (2022) Project Gutenberg official site. https://www.gutenberg.org/ Accessed 16
June 2022.
Hardeniya, N., Perkins, J. and Chopra, D. (2016) Natural Language Processing: Python and
NLTK. Packt Publishing.
Jupyter (2022) Jupyter official site. https://jupyter.org/. Accessed 16 June 2022.
Kedia, A. and Rasu, M. (2020) Hands-On Python Natural Language Processing: Explore tools and
techniques to analyze and process text with a view to building real-world NLP applications.
Packt Publishing.
NLTK (2022) NLTK official site. https://www.nltk.org/. Accessed 16 June 2022.
Perkins, J. (2014). Python 3 text processing with NLTK 3 cookbook. Packt Publishing Ltd.
Wintjen, M. and Vlahutin, A. (2020) Practical Data Analysis Using Jupyter Notebook: Learn how
to speak the language of data by extracting useful and actionable insights using Python. Packt
Publishing.
WordNet (2022) WordNet official site. https://wordnet.princeton.edu/. Accessed 16 June 2022.
Chapter 11
Workshop#2 N-grams in NLTK
and Tokenization in SpaCy (Hour 3–4)

11.1 Introduction

Workshop 2 consists of two parts:

Part 1 will introduce N-gram language model using NLTK in Python and
N-grams class to generate N-gram statistics on any sentence, text objects, whole
document, literature to provide a foundation technique for text analysis, parsing and
semantic analysis in subsequent workshops.
Part 2 will introduce spaCy, the second important NLP Python implementation
tools not only for teaching and learning (like NLTK) but also widely used for NLP
applications including text summarization, information extraction, and Q&A chat-
bot. It is a critical mass to integrate with Transformer Technology in subsequent
workshops.

11.2 What Is N-Gram?

N-gram is an algorithm based on statistical language model (Bird et al. 2009;

Perkins 2014; Arumugam and Shanmugamani 2018), its basic idea is that contents
such as phonemes, syllables, letters, words, or base pairs in texts are operated by a
sliding window of size N to form a byte fragments sequence of length N
(Sidorov 2019).
N can be 1, 2, or other positive integer, although usually large N is not considered
because they rarely occur.
Each byte fragment is called a gram, and the frequency of all grams is counted
and filtered according to a pre-set threshold to form a list of key grams, which is the
text’s vector feature space, and each kind of gram in the list is a feature vector
dimension.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 267
Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_11
268 11 Workshop#2 N-grams in NLTK and Tokenization in SpaCy (Hour 3–4)

11.3 Applications of N-Grams in NLP

N-gram models are widely used (Albrecht et al. 2020; Arumugam and Shanmugamani
2018; Hardeniya et al. 2016; Kedia and Rasu 2020) in:
• Speech recognition where phonemes and sequences of phonemes are modeled
using a N-gram distribution.
• Parsing on words are modeled so that each N-gram is composed of N words. For
language identification, sequences of characters/graphemes (e.g. letters of the
alphabet) are modeled for different languages.
• Auto sentences completion
• Auto spell-check
• Semantic analysis

11.4 Generation of N-Grams in NLTK

NLTK (NLTK 2022; Bird et al. 2009; Perkins 2014) offers useful tools in NLP
processing.
Ngrams() function in NLTK facilitates N-gram operation.
Python code uses N-grams in NLTK to generate N-grams for any text string. Try
it and study how it works.
The following example is the first sentence of A Scandal in Bohemia from The
Adventures of Sherlock Holmes (Doyle 2019): To Sherlock Holmes she is always
“The Woman.” I have seldom heard him mention her under any other name, demon-
strating how N-gram generator works in NLTK.

In[1] import nltk

from nltk import ngrams
sentence = input( "Enter the sentence: " )
n = int(input( "Enter the value of n: " ))
n_grams = ngrams(sentence.split(), n)
for grams in n_grams:
print(grams)

Out[1]
11.4 Generation of N-Grams in NLTK 269

Here are the Bigrams. Let’s try Trigrams N=3.

In[2] import nltk

from nltk import ngrams
sentence = input( "Enter the sentence: " )
n = int(input( "Enter the value of n: " ))
n_grams = ngrams(sentence.split(), n)
for grams in n_grams:
print(grams)

Out[2]

How about Quadrigram N=4? Let’s use the same sentence.

In[3] import nltk

from nltk import ngrams
sentence = input( "Enter the sentence: " )
n = int(input( "Enter the value of n: " ))
n_grams = ngrams(sentence.split(), n)
for grams in n_grams:
print(grams)

Out[3]
270 11 Workshop#2 N-grams in NLTK and Tokenization in SpaCy (Hour 3–4)

NLTK offers an easy solution to generate N-gram of any N-number which are
useful in N-gram probability calculations and text analysis

Workshop 2.1 N-Grams on The Adventures of Sherlock Holmes

1. Read Adventures_Holmes.txt text file.
2. Save contents into a string object "holmes_doc".
3. Extract favorite paragraph from "holmes_doc" into "holmes_para".
4. Use above Python code to generate N-grams for N=3, N=4, and N=5.

11.5 Generation of N-Grams Statistics

Once N-grams are generated, the next step is to calculate term frequency (TF) of
each N-grams from a document to list out top items.
NLTK-based Python codes extend previous example to create N-grams statistics
to list out top 10 N-grams.
Let us try first two sentences of A Scandal in Bohemia from The Adventures of
Sherlock Holmes.

In[4] sentence

Out[4] 'To Sherlock Holmes she is always "The Woman". I have seldom heard him
mention her under any other name.'

Import RE package to do some simple text pre-processing:

In[5] import re, string

# get rid of all the XML markup

sentence = re.sub ('<.*>' , ' ', sentence)

# get rid of punctuation (except periods!)

punctuationNoPeriod = "[" + re.sub("\.","",string.punctuation) + "]"
sentence = re.sub(punctuationNoPeriod, "", sentence)

# first get individual words

tokenized = sentence.split()

# and get a list of all the bi-grams

Bigrams = ngrams(tokenized, 2)
11.5 Generation of N-Grams Statistics 271

Review N-grams to see how they work:

In[6] ngrams?

To generate N-gram statistics, first import “collections” class and invoke

Counter() method over Bigrams to perform N-gram statistics analysis.

In[7] import collections

# get the frequency of each bigram in our corpus
BigramFreq = collections.Counter(Bigrams)

# what are the ten most popular ngrams in this corpus?

BigramFreq.most_common(10)

Out[7]

It is noted that top 10 bigram frequency are all with count 1.

This is because the sample sentence is short and does not contain any bigram(s)
with a frequent bigram statistic. To sort out this problem, let us try a longer text.
The following example uses the whole first paragraph of A Scandal in Bohemia
from The Adventures of Sherlock Holmes and see whether it has a preferable result.
272 11 Workshop#2 N-grams in NLTK and Tokenization in SpaCy (Hour 3–4)

The first paragraph looks like this:

In[8]

Let’s review this first paragraph:

In[9] first_para

Use Python script to remove punctuation marks and tokenize the first_para object:

In[10] import re, string

# get rid of all the XML markup
first_para = re.sub ('<.*>' , ' ', first_para)

# get rid of punctuation (except periods!)

punctuationNoPeriod = "[" + re.sub("\.","",string.punctuation) + "]"
first_para = re.sub(punctuationNoPeriod, "", first_para)
# first get individual words
tokenized = first_para.split()

# and get a list of all the bi-grams

Bigrams = ngrams(tokenized, 2)

Use Counter() method of Collections class to calculate Bigram statistics of

first_para:
11.5 Generation of N-Grams Statistics 273

In[11] import collections

# get the frequency of each bigram in our corpus
BigramFreq = collections.Counter(Bigrams)

# what are the ten most popular ngrams in this corpus?

BigramFreq.most_common(10)

Out[11]

The results are satisfactory. It is noted that bigram in a has the most occurrence
frequency, i.e. three times while four other bigrams: Irene Adler, and that, for
the, his own have occurred twice each within the paragraph. Bigram in a, and
that and for the are frequently used English phrases which occurred in almost
every text document. How about To Sherlock and Irene Adler? There are two
N-gram types frequently used in N-gram Language Model studied in Chap. 2.
One is the frequently used N-gram phrase in English like in a, and that and for
that in our case. These bigrams are common phrases in other documents and
literature writings. Another is domain-specific N-grams. These types are only
frequently used in specific domain, documents, and genre of literatures. Hence,
To Sherlock and Irene Adler are frequently used related to this story only and
not in other situations.

Workshop 2.2 N-grams Statistics on The Adventures of Sherlock Holmes

1. Read Adventures_Holmes.txt text file.
2. Save contents into a string object "holmes_doc".
3. Generate a more representative N-gram statistic using the whole
holmes_doc.
4. Generate top 10 N-grams summary for N=3, N=4, and N=5.
5. Review results and comments on pattern(s) found.

Bigram analysis is required to examine which bigrams are commonly used not
only on single paragraph but also for the whole document or literature. Remember
in Workshop 1 NLTK has a built-in list of tokenized sample literatures in nltk.book.
Let us refer them first by using the nltk.book import statement.
274 11 Workshop#2 N-grams in NLTK and Tokenization in SpaCy (Hour 3–4)

In[12] # Let's load some sample books from the nltk databank
import nltk
from nltk.book import *

Out[12]

Check with text1 to see what they are:

In[13] text1
Out[13] <Text: Moby Dick by Herman Melville 1851>

or download using nltk.corpus.gutenberg.words() from Project Gutenberg of copy-

right clearance classic literature (Gutenberg, 2022). Let’s use this method to down-
load Moby Dick (Melville, 2006).

In[14] import nltk.corpus

from nltk.text import Text
moby = Text(nltk.corpus.gutenberg.words( 'melville-moby_dick.txt' ))

In[15] moby

Out[15] <Text: Moby Dick by Herman Melville 1851>

Review the first 50 elements of Moby Dick text object to see whether they are
tokenized.

In[16] moby [1:50]

11.5 Generation of N-Grams Statistics 275

Out[16]

Use Collections class and ngrams() method for Bigram statistics to identify top
20 most frequently bigrams occurred for the entire Moby Dick literature.

In[17] import collections

# and get a list of all the bi-grams
Bigrams = ngrams(moby, 2)

# get the frequency of each bigram in our corpus

BigramFreq = collections.Counter(Bigrams)

# what are the 20 most popular ngrams in this corpus?

BigramFreq.most_common(20)

Out[17]

Workshop 2.3 N-grams Statistics with removal of unnecessary punctuations

The results are average and unsatisfactory. It is noted that and, of the, s and in
the are top 4 bigrams occurred in the entire Moby Dick literature. It is average
since these bigrams are common English usage but original bigram statistics in
simple sentences required to remove all punctuations by:
1. List out all punctuations required to remove.
2. Revise above Python script to remove these punctuation symbol from the
token list.
3. Generate a top 20 Bigram summary for Moby Dick without punctuations.
4. Use sample method to generate (cleaned) Bigram statistics from Moby
Dick, Adventures of Sherlock Holmes, Sense and Sensibility, Book of
Genesis, Inaugural Address Corpus, and Wall Street Journal.
5. Verify results and comments of any pattern(s) found.
6. Try the same analysis for Trigram (N=3) and Quadrigram (N=4) to find
any pattern(s).
276 11 Workshop#2 N-grams in NLTK and Tokenization in SpaCy (Hour 3–4)

11.6 spaCy in NLP

11.6.1 What Is spaCy?

SpaCy (2022) is a free, open-source library for advanced NLP written in Python and
Cython programming languages.
The library is published under MIT license developed by Dr. Matthew Honnibal
and Dr. Ines Montani, founders of software company Explosion.
SpaCy is designed specifically for production use and build NLP applications to
process large volumes of text (Altinok 2021; Srinivasa-Desikan 2018; Vasiliev
2020) different from NLTK focused on teaching and learning perspective.
It also provides workflow pipelines for machine learning and deep learning tools
that can integrate with common platforms such as PyTorch, MXNet, and TensorFlow
with its machine learning library called Thinc. spaCy provides recurrent neural mod-
els such as convolution neural networks (CNN) by adopting Thinc for NLP imple-
mentation such as Dependency Parsing (DP), NER (Named Entity Recognition),
POS Tagging, and Text Classification and other advanced NLP applications such as
Natural Language Understanding (NLU) systems, Information Retrieval (IR),
Information Extraction (IE) systems, and Question-and-Answer Chatbot systems.
A spaCy system architecture is shown in Fig. 11.1, its major features support:
• NLP-based statistical models for over 19 commonly used languages,
• tokenization tools implementation for over 60 international languages,
• NLP pipeline components include NER, POS Tagging, DP, Text Classification,
and Chatbot implementation,
• integration with common Python platforms such as TensorFlow, PyTorch and
other high-level frameworks,
• integration with the latest Transformer and BERT technologies,
• user-friendly modular system packaging, evaluation, and deployment tools.

Fig. 11.1 System architecture of spaCy

11.7 How to Install spaCy? 277

11.7 How to Install spaCy?

SpaCy can be installed in MacOS/OSX, MS Windows, and Linux platforms (SpaCy

2022) as per other Python-based development tools like NLTK.
spaCy.io provides a one-stop-process for users to select own spaCy (1)
language(s) as trained pipelines, (2) optimal target in system efficiency vs accuracy
for NLP applications development based on a large dataset and lexical database, and
(3) download appropriate APIs and modules to maximize efficiency under CPU and
GPU hardware configuration. Figure 11.2 shows a Windows-based PIP download
environment using CUDA 11.3 GPU in English as trained pipelines and target for
speed efficiency over accuracy.

Fig. 11.2 Screenshot of spaCy configuration selection

278 11 Workshop#2 N-grams in NLTK and Tokenization in SpaCy (Hour 3–4)

11.8 Tokenization using spaCy

Tokenization is an operation in NLP. spaCy provides an easy-to-use scheme to

tokenize any text document into sentences like NLTK and further tokenize sen-
tences into words.
This section uses Adventures_Holmes.txt as example to demonstrate tokeniza-
tion in spaCy.

11.8.1 Step 1: Import spaCy Module

In[18] import spacy

11.8.2 Step 2: Load spaCy Module "en_core_web_sm".

Use en_core_web_md-3.2.0 package for English pipeline optimized for CPU in the
current platform with components including: tok2vec, tagger, parser, senter, ner,
attribute_ruler, lemmatizer.

In[19] nlp = spacy.load( "en_core_web_sm" )

11.8.3 Step 3: Open and Read Text File "Adventures_Holmes.

txt" Into file_handler "fholmes"

Note: Since text file already exists, skip try-except module to save program-
ming steps.

In[20] fholmes = open( "Adventures_Holmes.txt", "r", encoding="utf-8")

11.8.4 Step 4: Read Adventures of Sherlock Holmes

Use read() method to read whole text document as a complex string object "holmes".
11.8 Tokenization using spaCy 279

In[21] holmes = fholmes.read()

holmes
Out[21]

11.8.5 Step 5: Replace All Newline Symbols

Replace all newline characters "\n" into space characters.

In[22] holmes = holmes.replace( "\n", " " )

11.8.6 Step 6: Simple Counting

Review total number of characters in The Adventures of Sherlock Holmes and exam-
ine the result document.

In[23] len (holmes)

Out[23] 580632

In[24] holmes

Out[24]
280 11 Workshop#2 N-grams in NLTK and Tokenization in SpaCy (Hour 3–4)

11.8.7 Step 7: Invoke nlp() Method in spaCy

SpaCy nlp() method is an important Text Processing Pipeline to initialize nlp object
(English in our case) for NLP processing such as tokenization. It will convert any
text string object into a nlp object.
Study nlp() docstring to see how it works.

In[25] nlp?

In[26] holmes_doc = nlp(holmes)

In[27] holmes_doc

Out[27]

11.8.8 Step 8: Convert Text Document Into Sentence Object

SpaCy is practical for text document tokenization to convert text document object
into (1) sentence objects and (2) tokens.
This example uses for-in statement to convert the whole Sherlock Holmes docu-
ment into holmes_sentences.
11.8 Tokenization using spaCy 281

In[28] holmes_sentences = [sentence.text for sentence in holmes_doc.sents]

holmes_sentences
Out[28]

Examine the structure of spaCy sentences and see what can be found.

In[29] holmes_sentences?

Study the numbers of sentences contained in The Adventures of Sherlock Holmes.

In[30] len (holmes_sentences)

Out[30] 6830
282 11 Workshop#2 N-grams in NLTK and Tokenization in SpaCy (Hour 3–4)

List out sentence numbers 50th – 59th to review.

In[31] holmes_sentences[50:60]

Out[31]

11.8.9 Step 9: Directly Tokenize Text Document

Tokenize text document into word tokens by using “token” object in spaCy instead
of text document object extraction into sentence list object. Study how it operates.

In[32] holmes_words = [token.text for token in holmes_doc]

holmes_words [130:180]
Out[32]

In[33] holmes_words?
11.8 Tokenization using spaCy 283

In[34] len (holmes_words)

Out[34] 133749

In[35] nltk_homles_tokens = nltk.word_tokenize(holmes)

In[36] nltk_homles_tokens [104:153]

Out[36]

According to extracted tokens, they seem to be identical.

1. Are they 100% identical?
2. What is/are the difference(s)?
3. Which one is better?

Workshop 2.4 SpaCy or NLTK: Which one is Faster?

In many applications, especially in AI and NLP application, speed (i.e. efficiency) is
one of the most important consideration because:
1. Many AI and NLP applications involve a huge data/database/databank for
system training with huge population size, e.g. Lexical database of English and
Chinese. So, whether an NLP engine/application is fast enough in every NLP
operation such as tokenization, tagging and POS tagging, and parsing is an
important factor.
2. In many AI-based related NLP application such as Deep Learning for real-time
Information Extraction, it involves tedious network training and learning
process, how efficient of every NLP operation is a critical process to decide
whether NLP application can be used in real-world scenario.
This workshop studies how efficient NLTK vs spaCy in terms of text document
Tokenization.
To achieve this, integrate Python codes of NTLK/spaCy document tokenization
with Timer object - time.
1. Implement tokenization codes in NTLK and spaCy to time tokenization time
by using time object, the following codes can be used as reference.
2. Examine time taken for Tokenization process of "Adventures_Holmes.txt"
using NTLK vs spaCy methods.
3. Which one is faster? or are they similar? Why?
4. How about Document->Text efficiency? Compare NTLK vs spaCy on
Doc->Text efficiency.
Hint: Like spaCy, NLTK can also implement Document->Text by two simple codes:
nltk_tokenizer = nltk.data.load("tokenizers/punkt/english.pickle")
nltk_sentences = tokenizer.tokenize(holmes) # holmes is the text document string
object
284 11 Workshop#2 N-grams in NLTK and Tokenization in SpaCy (Hour 3–4)

In[37] # Sample code for Efficiency Performance of the NLP Engine

import nltk # or spacy
import time
start = time.time()
#
# YOUR NTLK or spaCy Tokenization codes
#
print( "Time taken: %s s" % (time.time() - start))

Out[37] 0.0s

References

Albrecht, J., Ramachandran, S. and Winkler, C. (2020) Blueprints for Text Analytics Using
Python: Machine Learning-Based Solutions for Common Real World (NLP) Applications.
O’Reilly Media.
Altinok, D. (2021) Mastering spaCy: An end-to-end practical guide to implementing NLP applica-
tions using the Python ecosystem. Packt Publishing.
Arumugam, R., & Shanmugamani, R. (2018). Hands-on natural language processing with python.
Packt Publishing.
Bird, S., Klein, E., and Loper, E. (2009). Natural language processing with python. O'Reilly.
Doyle, A. C. (2019) The Adventures of Sherlock Holmes (AmazonClassics Edition).
AmazonClassics.
Gutenberg (2022) Project Gutenberg official site. https://www.gutenberg.org/ Accessed 16
June 2022.
Hardeniya, N., Perkins, J. and Chopra, D. (2016) Natural Language Processing: Python and
NLTK. Packt Publishing.
Melville, H. (2006) Moby Dick. Hard Press.
Kedia, A. and Rasu, M. (2020) Hands-On Python Natural Language Processing: Explore tools and
techniques to analyze and process text with a view to building real-world NLP applications.
Packt Publishing.
NLTK (2022) NLTK official site. https://www.nltk.org/. Accessed 16 June 2022.
Perkins, J. (2014). Python 3 text processing with NLTK 3 cookbook. Packt Publishing Ltd.
SpaCy (2022) spaCy official site. https://spacy.io/. Accessed 16 June 2022.
Sidorov, G. (2019) Syntactic n-grams in Computational Linguistics (SpringerBriefs in Computer
Science). Springer.
Srinivasa-Desikan, B. (2018). Natural language processing and computational linguistics: A prac-
tical guide to text analysis with python, gensim, SpaCy, and keras. Packt Publishing, Limited.
Vasiliev, Y. (2020) Natural Language Processing with Python and spaCy: A Practical Introduction.
No Starch Press.
Chapter 12
Workshop#3 POS Tagging Using NLTK
(Hour 5–6)

12.1 Introduction

In Chap. 3, we studied basic concepts and theories related to Part-of-Speech (POS)

and various POS tagging techniques. This workshop will explore how to implement
POS tagging by using NLTK starting from a simple recap on tokenization tech-
niques and two fundamental processes in word-level progressing: stemming and
stop-word removal, which will introduce two types of stemming techniques: Porter
Stemmer and Snowball Stemmer that can be integrated with WordCloud commonly
used in data visualization followed by the main theme of this workshop with the
introduction of PENN Treebank Tagset and to create your own POS tagger.

12.2 A Revisit on Tokenization with NLTK

Text sentences are divided into subunits first and map into vectors in most NLP
tasks. These vectors are fed into a model to encode where output is sent to a down-
stream task for results. NLTK (NLTK 2022) provides methods to divide text into
subunits as tokenizers. Twitter sample corpus is extracted from NLTK to perform
tokenization (Hardeniya et al. 2016; Kedia and Rasu 2020; Perkins 2014) in proce-
dures below (Albrecht et al. 2020; Antic 2021; Bird et al. 2009):
1. Import NLTK package.
2. Import Twitter sample data.
3. List out fields.
4. Get Twitter string list.
5. List out first 15 Twitters.
6. Tokenize twitter.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 285
Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_12
286 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

Let us start with the import of NLTK package and download Twitter samples
provided by NLTK platform.

In[1] # Import NLTK

import nltk

# Download twitter_samples
# nltk.download('twitter_samples')

Import twitter samples dataset as twtr and check file id using fileids() method:

In[2] # Import twitter samples from NTLK corpus (twtr)

from nltk.corpus import twitter_samples as twtr

In[3] # Display Field IDs

twtr.fileids()
Out[3] ['negative_tweets.json','positive_tweets.json', 'tweets.20150430-23406.json']

Review first 5 Twitter messages:

In[4] # Assign sample twitters (stwtr)

stwtr = twtr.strings('tweets.20150430-223406.json')

In[5] # Display the first 5 sample twitters

stwtr[:5]
Out[5]

Import word_tokenize method from NLTK, name as w_tok to perform tokeniza-

tion on 5th Twitter message:

In[6] # Import NLTK word tokenizer

from nltk.tokenize import word_tokenize as w_tok
12.2 A Revisit on Tokenization with NLTK 287

In[7] # tokenize stwtr[4]

w_tok(stwtr[4])

Out[7]

NLTK offers tokenization for punctuation and spaces wordpunct_tokenize.

Let’s use the 5th Twitter message to see how it works.

In[8] from nltk.tokenize import wordpunct_tokenize as wp_tok

wp_tok(stwtr[4])
Out[8]

It can also tokenize words between hyphens and other punctuations. Further,
NLTK’s regular expression (RegEx) tokenizer can build custom tokenizers:

In[9] # Import the RegEx tokenizer

from nltk import regexp_tokenize as rx_tok
rx_pattern1 = '\w+'
rx_tok(stwtr[4],rx_pattern1)
Out[9]

A simple regular expression filtered out words with alphanumeric

characters only, but not punctuations in previous code. Another regu-
lar expression can detect and filter out both words containing alpha-
numeric characters and punctuation marks in the following code:
288 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

In[10] # Create Rx pattern2 and perform the RX tokenize again

rx_pattern2 = '\w+|[!,\-,]'
rx_tok(stwtr[4],rx_pattern2)
Out[10]

12.3 Stemming Using NLTK

After tokenization has sentences divided into words, stemming is a procedure to

unify words and extract the root, base form of each word, e.g. stemming of word
compute is shown in Fig. 12.1.

12.3.1 What Is Stemming?

Stemming usually removes prefixes or suffixes such as -er, -ion, -ization from words
to extract the base or root form of a word, e.g. computers, computation, and com-
puterization. Although these words spell differently but shared identical concept
related to compute, so compute is the stem of these words.

Fig. 12.1 Stemming of compute

12.3 Stemming Using NLTK 289

12.3.2 Why Stemming?

There is needless to extract every single word in a document but only concept or
notion they represent such as information extraction and topic summarization in
NLP applications. It can save computational capacity and preserve overall meaning
of the passage. Stemming technique is to extract the overall meaning or words’ base
form instead of distinct words.
Let us look at how to perform stemming on text data.

12.3.3 How to Perform Stemming?

NLTK provides practical solution to implement stemming without sophisticated

programming. Let us try two commonly used methods: (1) Porter Stemmer and (2)
Snowball Stemmer in NLP.

12.3.4 Porter Stemmer

Porter Stemmer is the earliest stemming technique used in 1980s. Its key procedure
is to remove words common endings and parse into generic forms. This method is
simple and used in many NLP applications effectively.
Import Porter Stemmer from NLTK library:

In[11] # Import PorterStemmer as p_stem

from nltk.stem.porter import PorterStemmer as p_stem

Try to stem words like computer.

In[12] p_stem().stem("computer")
Out[12] 'comput'

PorterStemmer simply removes suffix -er when processing computer to

acquire compute which is incorrect. Hence this stemmer is basic.
290 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

Next, try to stem dogs to see what happens.

In[13] p_stem().stem("dogs")
Out[13] 'dog'

For the above code, dogs are converted from plural to singular, remove
suffix -s and convert to dog.

Let’s try more, say traditional.

In[14] p_stem().stem("traditional")
Out[14] 'tradit'

Stemmer may output an invalid word when dealing with special words e.g.
tradit is acquired if suffix -ional is removed. tradit is not a word in English,
it is a root form.

Let’s work on words in plural form. There are 26 words extracted from a – z in
plural form to perform PorterStemming:

In[15] # Define some plural words

w_plu = ['apes','bags','computers','dogs','egos','frescoes','generous','hats','iglo
os','jungles', 'kites','learners','mice','natives','openings','photos','queries','rats',
'scenes', 'trees','utensils','veins','wells','xylophones','yoyos','zens']

In[16] from nltk.stem.porter import PorterStemmer as p_stem

w_sgl = [p_stem().stem(wplu) for wplu in w_plu]
print(' '.join(w_sgl))

Out[16] ape bag comput dog ego fresco gener hat igloo jungl kite learner mice nativ
open photo queri rat scene tree utensil vein well xylophon yoyo zen

Porter Stemming will remove suffixes -s or -es to extract root form, that may
result in single form such as apes, bags, dogs, etc. But in some cases, it will
generate non-English words such as gener, jungl and queri.
12.3 Stemming Using NLTK 291

Workshop 3.1 Try to stem a paragraph from The Adventures of Sherlock

Holmes
1. Read Adventures_Holmes.txt text file from The Adventures of Sherlock
Holmes (Doyle 2019; Gutenberg 2022)
2. Save contents into a string object "holmes_doc"
3. Extract a paragraph and tokenize it
4. Use Porter Stemming and output a list of stemmed words.

12.3.5 Snowball Stemmer

Snowball Stemmer provides improvement in stemming results as compared with

Porter Stemmer and provides multi-language stemming solution. One can check
languages using languages() method. Import from NLTK package to invoke
Snowball Stemmer:

In[17] # Import Snowball Stemmer as s_stem

from nltk.stem.snowball import SnowballStemmer as s_stem

Review what languages Snowball stemmer can support:

In[18] # Display the s_stem language set

print(s_stem.languages)

Out[18] ('arabic', 'danish', 'dutch', 'english', 'finnish', 'french', 'german', 'hungarian',

'italian', 'norwegian', 'porter', 'portuguese', 'romanian', 'russian', 'spanish',
'swedish')

Snowball Stemmer provides a variety of solutions in commonly used languages

from Arabic to Swedish.

In[19] # Import Snowball Stemmer as s_stem and assign to English language

from nltk.stem.snowball import SnowballStemmer as s_stem
s_stem_ENG = s_stem(language="english")

Use same list of plural words (w_plu) to check how it works in Snowball
Stemmer for comparison:

In[20] # Display the list of plural words

w_plu
292 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

Out[20] ['apes', 'bags', 'computers', 'dogs', 'egos', 'frescoes', 'generous', 'hats',

'igloos', 'jungles', 'kites', 'learners', 'mice', 'natives', 'openings', 'photos',
'queries', 'rats', 'scenes', 'trees', 'utensils', 'veins', 'wells', 'xylophones',
'yoyos', 'zens']

In[21] # Apply Snowball Stemmer onto the plural words

sgls = [s_stem_ENG.stem(wplu) for wplu in w_plu]
print(' '.join(sgls))

Out[21] ape bag comput dog ego fresco generous hat igloo jungl kite learner mice
nativ open photo queri rat scene tree utensil vein well xylophon yoyo zen

Try to compare with previous stemmer. What are the differences?

1. Snowball Stemmer achieved similar results as Porter Stemmer in most cases

except in generously where Snowball Stemmer came up with a meaningful
root form generous instead of gener in Porter Stemmer
2. Try some plural words to compare performance between Porter Stemmer vs
Snowball Stemmer

12.4 Stop-Words Removal with NLTK

12.4.1 What Are Stop-Words?

There are input words and utterances to filter out impractical stop-words in NLP
preprocessing such as: a, is, the, of, etc.
NLTK already provides a built-in stop-words package for this function. Let us
see how it works.

12.4.2 NLTK Stop-Words List

Import stopwords module and call stopwords.words() method to list out all stop-
words in English.
12.4 Stop-Words Removal with NLTK 293

In[22] # Import NLTK stop-words as wstops

from nltk.corpus import stopwords as wstops
print(wstops.words('english'))

Out[22]

1. Stop-words corpus size is not large.

2. All stop-words are commonly used in many documents. They effect stor-
age and system efficiency in NLP applications if they are not removed.
3. This stop-word corpus is incomplete and subjective. There may be words
considered as stop-words not included in this databank.

Use stopwords.fileids() function to review how many languages library of stop-

words NLTK contains.

In[23] # Import NLTK stop-words as wstops and display the FILE_IDs

from nltk.corpus import stopwords as wstops
print(wstops.fileids())
Out[23]
294 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

12.4.3 Try Some Texts

The above list shows all stop-words. Let us use a simple utterance:

In[24] # Import NLTK stop-words as wstops

from nltk.corpus import stopwords as wstops
wstops_ENG = wstops.words('english')
utterance = "Try to test for the stop word remove function to see how it
works."
utterance_clean =[w for w in utterance.split()
if w not in wstops_ENG]

Review results:

In[25] # Display the cleaned utterance

utterance_clean
Out[25] ['We', 'look', 'words', 'removed', 'text', 'following', 'code']

1. All commonly used stop-words such as to, for, the, it, are removed as
shown in the example.
2. It has little effect to overall meaning of the utterance.
3. It requires same computational time and effort.

The following example uses Hamlet from The Complete Works of Shakespeare to
demonstrate how stop-words are removed from text processing in NLP.

In[26] # Import the Gutenberg library from NLTK

from nltk.corpus import gutenberg as gub
hamlet = gub.words('shakespeare-hamlet.txt')
hamlet_clean = [w for w in hamlet if w not in wstops_ENG]

In[27] len(hamlet_clean)*100.0/len(hamlet)
Out[27] 69.26124197002142

This classic literature contains deactivated words. Nevertheless, these stop-

words are unmeaningful in many NLP tasks that may affect results, so most
of them are removed during pre-processing.
12.4 Stop-Words Removal with NLTK 295

12.4.4 Create Your Own Stop-Words

Stop-word corpus can extract a list of string that can add any stop-words with sim-
ple append() function, but it is advisable to create a new stop-word library object
name to begin.

12.4.4.1 Step 1: Create Own Stop-Word Library List

In[28] My_sws = wstops.words('english')

12.4.4.2 Step 2: Check Object Type and Will See It Has a Simple List

In[29] My_sws?

12.4.4.3 Step 3: Study Stop-Word List

In[30] My_sws
Out[30]

12.4.4.4 Step 4: Add New Stop-Word "sampleSW" Using Append()

In[31] My_sws.append('sampleSW')
My_sws[160:]
Out[31]
296 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

Try this to see how it works.

In[32] # Import word_tokenize as w_tok

from nltk.tokenize import word_tokenize as w_tok

# Create the sample utterance

utterance = "This is a sample utterance which consits of eg as stop word
sampleSW."

# Tokenize the utterance

utt_toks = w_tok(utterance)

# Stop word removal

utt_nosw = [w for w in utt_toks if not w in My_sws]

# Display utterance without My stopwords

print(utt_nosw)

Out[32] ['This', 'sample', 'utterance', 'consits', 'eg', 'stop', 'word', '.']

Workshop 3.2 Stop-word Filtering on The Adventures of Sherlock Holmes

Use stop-word filtering technique for The Adventures of Sherlock Holmes:
1. Read Adventures_Holmes.txt text file
2. Save contents into a string object "holmes_doc"
3. Use stop-word technique just learnt to tokenize holmes_doc
4. Generate a list of word tokens with stop-words removed
5. Check any 3 possible stop-words to add into own stop-word list
6. Regenerate a new token list with additional stop-word removed

12.5 Text Analysis with NLTK

When text data has been processed and tokenized, basic analysis are required to
calculate words or tokens, their distribution and usage frequency in NLP tasks. This
allows understanding of main contents and topics accuracy in the document. Import
a sample webtext (Firefox.txt) from NLTK library.
12.5 Text Analysis with NLTK 297

In[33] # Import webtext as wbtxt

from nltk.corpus import webtext as wbtxt

# Create sample webtext

wbtxt_s = wbtxt.sents('firefox.txt')
wbtxt_w = wbtxt.words('firefox.txt')

# Display total nos of webtext sentences in firefox.txt

len(wbtxt_s)
Out[33] 1138

Review the number of words as well.

In[34] # Display total nos of webtext words in firefox.txt

len(wbtxt_w)
Out[34] 102457

FireFox.txt contains sample texts extracted from Firefox discussion forum to

serve as useful dataset for basic text-level analysis in NLP.

It can also obtain vocabulary size by passing through a set as shown in the fol-
lowing code:

In[35] # Define vocabulary object (vocab)

vocab = set(wbtxt_w)

# Display the size of Vocab

len(vocab)
Out[35] 8296

nltk.FreqDist() function is used to generate words frequency distribution occurred

in the whole text as shown:

In[36] # Define Frequency Distribution object

fdist = nltk.FreqDist(wbtxt_w)

In[37] sorted(fdist, key=fdist.getitem,reverse=True)[0:30]

298 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

Out[37]

The above code generates top 30 frequently used words and punctuations in
the whole text. in, to and the are top 3 on the list like other literatures as
Firefox.txt text is the collection of users’ discussion messages and contents
about Firefox browser like conversations.

To exclude stop-words such as the, and not, use the following code to see f words
frequency distribution longer than 3.

In[38] # Import Matplotlib pyplot object

import matplotlib.pyplot as pyplt
pyplt.figure(figsize=(20, 8))
lwords = dict([(k,v) for k,v in fdist.items() if len(k)>3])
fdist = nltk.FreqDist(lwords)
fdist.plot(50,cumulative=False)

Out[38]

Exclude stop-words such as the, and, is, and create a tuple dictionary to
record words frequency. Visualize and transform them into a NLTK fre-
quency distribution graph based on this dictionary as shown above.
12.6 Integration with WordCloud 299

Workshop 3.3 Text Analysis on The Adventures of Sherlock Holmes

1. Read Adventures_Holmes.txt text file
2. Save contents into a string object "holmes_doc"
3. Use stop-word technique from tokenize holmes_doc
4. Generate a word tokens list with stop-words removed
5. Use the technique learnt to plot first 30 frequently occurred words from
this literature
6. Identify any special pattern related to word distribution. If no, try first 50
ranking words

12.6 Integration with WordCloud

See Fig. 12.2.

12.6.1 What Is WordCloud?

Wordcloud, also known as tag cloud, is a data visualization method commonly used
in many web statistics and data analysis scenarios. It is a graphical representation of
all words and keywords in sizes and colors. A word has the largest and bold in word
cloud means it occurs frequently in the text (dataset).

Fig. 12.2 A sample WordCloud

300 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

To generate frequency distribution of all words occurred in a text document, the

most natural way is to generate statistics in a WordCloud.
Python provides a built-in WordCloud package "WordCloud".
It can obtain an intuitive visualization of words used in the text from the fre-
quency distribution.
Install wordcloud package first using the pip install command:
pip install wordcloud.
Once Wordcloud package is installed, import wordcloud package using import
command and invoke the frequency generator with generate_from frequen-
cies() method:

In[39] # Import WordCloud as wCloud

from wordcloud import WordCloud as wCloud

In[40] wcld = wCloud().generate_from_frequencies(fdist)

In[41] Import matplotlib.pyplot as pyplt

pyplt.figure(figsize=(20, 8))
pyplt.imshow(wcld, interpolation='bilinear')
pyplt.axis("off")
pyplt.show()
Out[41]

Workshop 3.4 WordCloud for The Adventures of Sherlock Holmes

1. Read Adventures_Holmes.txt text file
2. Save contents into a string object "holmes_doc"
3. Use stop-word technique from tokenize holmes_doc
4. Generate word tokens list with stop-words removed
5. Extract top 100 frequently words occurred from this literature
6. Generate Wordcloud for this literature
12.7 POS Tagging with NLTK 301

12.7 POS Tagging with NLTK

The earlier part of this workshop had studied several NLP preprocessing tasks:
tokenization, stemming, stop-word removal, word distribution in text corpus and
data visualization using WordCloud. This section will explore POS tagging
in NLTK.

12.7.1 What Is POS Tagging?

Part-of-Speech (POS) refers to words categorization process in a sentence/utterance

into specific syntactic or grammatical functions.
There are 9 major POS in English: Nouns, Pronouns, Adjectives, Verbs,
Prepositions, Adverbs, Determiners, interjection, and Conjunctions. POS tagging is
to assign POS tags into each word token in the sentence/utterance.
NTLK supports commonly used tagset such as PENN Treebank (Treebank 2022)
and Brown corpus. It allows to create own tags used for specific NLP applications.

12.7.2 Universal POS Tagset

A tagset consists of 12 universal POS categories which is constructed to facilitate

future requirements for unsupervised induction of syntactic structure. When is com-
bined with original treebank data, this universal tagset and mapping produce a data-
set consisting of common POS in 22 languages (Albrecht et al. 2020; Antic 2021;
Bird et al. 2009).
Figure 12.3 shows a table of universal POS tagset in English.

Fig. 12.3 Table of Universal POS Tagset in English

302 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

12.7.3 PENN Treebank Tagset (English and Chinese)

English Penn Treebank Tagset is used with English corpora developed by Prof.
Helmut Schmid in TC project at the Institute for Computational Linguistics of the
University of Stuttgart (TreeBank 2022). Figure 12.4 shows an original 45 used
Penn Treebank Tagset.
A recent version of this English POS Tagset can be found at Sketchengine.eu
(Sketchengine 2022a) and Chinese POS Tagset (Sketchengine 2022b).
NLTK provides direct mapping from tagged corpus such as Brown Corpus
(NLTK 2022) to universal tags for implementation, e.g. tags VBD (for past tense
verb) and VB (for base form verb) map to VERB only in universal tagset.

In[42] # Import Brown Corpus as bwn

from nltk.corpus import brown as bwn

Fig. 12.4 Original 45 used Penn Treebank Tagset

12.7 POS Tagging with NLTK 303

In[43] bwn.tagged_words()[0:40]
Out[43]

Fulton is tagged as NP-TL in example code above, a proper noun (NP) appears in a
title (TL) context in Brown corpus that mapped to noun in universal tagset. These
subcategories are to be considered instead of generalized universal tags in NLP
application

12.7.4 Applications of POS Tagging

POS tagging is commonly used in many NLP applications ranging from Information
Extraction (IE), Named Entity Recognition (NER) to Sentiment Analysis and
Question-&-Answering systems.
Try the following and see how it works:

In[44] # Import word_tokenize and pos_tag as w_tok and p_tag

from nltk.tokenize import word_tokenize as w_tok
from nltk import pos_tag as p_tag

# Create and tokenizer two sample utterances utt1 and utt2

utt1 = w_tok("Give me a call")
utt2 = w_tok("Call me later")

Review these two utterances’ POS tags:

In[45] p_tag(utt1, tagset='universal' )

Out[45] [('Give', 'VERB'), ('me', 'PRON'), ('a', 'DET'), ('call', 'NOUN')]
304 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

In[46] p_tag(utt2, tagset='universal' )

Out[46] [('Call', 'VERB'), ('me', 'PRON'), ('later', 'ADV')]

1. The word call is a noun in text 1 and a verb in text 2.

2. POS tagging is used to identify a person, a place, or a location, based on the
tags in NER.
3. NLTK also provides a classifier to identify such entities in text as shown in
the following code:

In[47] utt_untag = w_tok("My dad was born in South America")

utt_untag

Out[47] ['My', 'dad', 'was', 'born', 'in', 'South', 'America']

In[48] utt_tagged = p_tag(utt_untag)

utt_tagged
Out[48] [('My', 'PRP$'), ('dad', 'NN'), ('was', 'VBD'), ('born', 'VBN'),
('in', 'IN'), ('South', 'NNP'), ('America', 'NNP')]

In[49] # Import svgling package

import svgling

# Import NLTK.ne_chunk as chunk

from nltk import ne_chunk as chunk

# Display POS Tags chunk

chunk(utt_tagged)
Out[49]

NLTK chunk() function is applied to NER to identify the chunker South

America as a geopolitical entity (GPE) in this example. So far, there are
examples using NLTK’s built-in taggers. Next section will look at how to
develop own POS tagger.
12.7 POS Tagging with NLTK 305

In[50] # Try another example

utt_tok = w_tok("Can you please buy me Haagen-Dazs Icecream? It's
$30.8.")
print("Tokens are: ", utt_tok)

Out[50] Tokens are: ['Can', 'you', 'please', 'buy', 'me', 'Haagen-Dazs', 'Icecream', '?',
'It', "'s", '$', '30.8', '.']

In[51] utt_tagged = p_tag(utt_tok)

chunk(utt_tagged)
Out[51]

1. The system treats ‘$’, ‘30.8’, and ‘.’ as separate tokens in this example. It
is crucial because contractions have their own semantic meanings and own
POS leading to the ensuing part of NLTK library POS tagger.
2. POS tagger in NLTK library outputs specific tags for certain words.
3. However, it makes a mistake in this example. Where is it?
4. Compare POS Tagging for the following sentence to identify problem.
Explain.

In[52] # Try one more example

utt_tok = w_tok("Can you please buy me New-Zealand Icecream? It's
$30.8.")
print("Tokens are: ", utt_tok)
utt_tagged = nltk.pos_tag(utt_tok)
chunk(utt_tagged)
Out[52] Tokens are: ['Can', 'you', 'please', 'buy', 'me', 'New-Zealand', 'Icecream', '?',
'It', "'s", '$', '30.8', '.']
306 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

Workshop 3.5 POS Tagging on The Adventures of Sherlock Holmes

1. Read Adventures_Holmes.txt text file
2. Save contents into a string object "holmes_doc"
3. Extract three typical sentences from three stories of this literature
4. Use POS Tagging to these sentences
5. Use ne_chunk function to display POS tagging tree for these three
sentences
6. Compare POS Tags among these example sentences and examine on
how they work

12.8 Create Own POS Tagger with NLTK

This section will create own POS tagger using NLTK’s tagged set corpora and
sklearn Random Forest machine learning model.
The following example demonstrates a classification task to predict POS tag for
a word in a sentence using NLTK treebank dataset for POS tagging, and extract
word prefixes, suffixes, previous and neighboring words as features for system
training.
Import all necessary Python packages as below:

In[53] # Import all necessary Python packages

import nltk
import numpy as np
from nltk import word_tokenize as w_tok
import matplotlib.pyplot as pyplt
%matplotlib inline
from sklearn.feature_extraction import DictVectorizer as DVect
from sklearn.model_selection import train_test_split as tt_split
from sklearn.ensemble import RandomForestClassifier as RFClassifier
from sklearn.metrics import accuracy_score as a_score
from sklearn.metrics import confusion_matrix as c_matrix
12.8 Create Own POS Tagger with NLTK 307

In[54] # Define the ufeatures() class

def ufeatures(utt, idx):
ftdist = {}
ftdist['word'] = utt[idx]
ftdist['dist_from_first'] = idx - 0
ftdist['dist_from_last'] = len(utt) - idx
ftdist['capitalized'] = utt[idx][0].upper() == utt[idx][0]
ftdist['prefix1'] = utt[idx][0]
ftdist['prefix2'] = utt[idx][:2]
ftdist['prefix3'] = utt[idx][:3]
ftdist['suffix1'] = utt[idx][-1]
ftdist['suffix2'] = utt[idx][-2:]
ftdist['suffix3'] = utt[idx][-3:]
ftdist['prev_word'] = '' if idx==0 else utt[idx-1]
ftdist['next_word'] = '' if idx==(len(utt)-1) else utt[idx+1]
ftdist['numeric'] = utt[idx].isdigit()
return ftdist

In[55] # Define the Retreive Untagged Utterance (RUutterance) class

def RUutterance(utt_tagged):
[utt,t] = zip(*utt_tagged)
return list(utt)

Function ufeatures() converts input text into a dict object of features,

whereas each utterance is passed with corresponding index of current
token word which features are extracted. Let’s use treebank tagged utter-
ances with universal tags to label and train data:

In[56] utt_tagged = nltk.corpus.treebank.tagged_sents(tagset='universal')

In[57] utt_tagged
Out[57]
308 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

1. In this example, universal tags are used for simplicity.

2. Of course, one can also use fine-gained treebank POS Tags for imple-
mentation.
3. Once do so, can now extract the features for each tagged utterance in
corpus with training labels.
Use following code to extract the features:

In[58] # Define Extract Feature class (exfeatures)

def exfeatures(utt_tag):
utt, tag = [], []

for ut in utt_tag:
for idx in range(len(ut)):
utt.append(ufeatures(RUutterance(ut), idx))
tag.append(ut[idx][1])

return utt, tag

In[59] X,y = exfeatures(utt_tagged)

This example uses DVect to convert feature-value dictionary into training vectors.
If the number of possible values for suffix3 feature is 40, there will be 40 features
in output. Use following code to DVect:

In[60] # Define sample size

nsize = 10000

# Invoke Dict Vectorizer

dvect = DVect(sparse=False)

Xtran = dvect.fit_transform(X[0:nsize])
ysap = y[0:nsize]

This example has a sample size of 10,000 utterances which 80% of the dataset
is used for training and other 20% is used for testing. RF (Random Forecast)
Classifier is used as POS tagger model as shown:

In[61] Xtrain,Xtest,ytrain,ytest = tt_split(Xtran, ysap, test_size=0.2,

random_state=123)
12.8 Create Own POS Tagger with NLTK 309

In[62] rfclassifier = RFClassifier(n_jobs=4)

rfclassifier.fit(Xtrain,ytrain)

Out[62] RandomForestClassifier(n_jobs=4)

After system training, can perform POS Tagger validation by using some
sample utterances. But before passing to ptag_predict() method, extract fea-
tures are required by ufeatures() method as shown:

In[63] # Define the POS Tags Predictor class (ptag_predict)

def ptag_predict(utt):
utt_tagged = []
fts = [ufeatures(utt, idx) for idx in range(len(utt))]
fts = dvect.transform(fts)
tgs = rfclassifier.predict(fts)
return zip(utt, tgs)

Convert utterance into corresponding features with ufeatures() method. The

features dictionary extracted from this method is vectorized using previously
trained dvect:

In[64] # Test with a sample utterance (utt3)

utt3 = "It is an example for POS tagger"
for utt_tagged in ptag_predict(utt3.split()):
print(utt_tagged)
Out[64] ('It', 'PRON') ('is', 'VERB') ('an', 'DET') ('example', 'NOUN')
('for', 'ADP') ('POS', 'NOUN') ('tagger', 'NOUN')

Use a sample utterance “utt3” and invoke ptag_predict() method to output

tags for each word token inside utt3 and review for accuracy afterwards.

In[65] predict = rfclassifier.predict(Xtest)

In[66] a_score(ytest,predict)
Out[66] 0.9355
310 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

The overall a_score has approximately 93.6% accuracy rate and satisfactory.
Next, let’s look at confusion matrix (c-mat) to check how well can POS tagger
perform.

In[67] c_mat = c_matrix(ytest,predict)

In[68] pyplt.figure(figsize=(10,10))
pyplt.xticks(np.arange(len(rfclassifier.classes_)),rfclassifier.classes_)
pyplt.yticks(np.arange(len(rfclassifier.classes_)),rfclassifier.classes_)
pyplt.imshow(c_mat, cmap=pyplt.cm.Blues)
pyplt.colorbar()

Out[68]

Use classes from RF classifier as x and y labels to create a c-mat (confusion

matrix). These labels are POS tags used for system training. The plot that fol-
lows shows a pictorial representation of confusion matrix.

Use classes from random forest classifier as x and y labels in the code for plotting
confusion matrix.
12.8 Create Own POS Tagger with NLTK 311

It looks like the tagger performs relatively well for nouns, verbs, and determiners
in sentences reflected in dark regions of the plot. Let’s look at some top features of
the model from following code:

In[69] flist = zip(dvect.get_feature_names_out(),

rfclassifier.feature_importances_)
sfeatures = sorted(flist,key=lambda x: x[1], reverse=True)
print(sfeatures[0:20])
Out[69]

1. The RF feature importance is stored in Python feature_importances list. Some

of the suffix features have higher importance scores than others
2. For instances, words ending with -ed are usually verbs in past tense which
make sense in many situations, and punctuations like commas may affect POS
tagging performance in some situations

Workshop 3.6 Revisit POS Tagging on The Adventures of Sherlock Holmes

with Additional Tagger
1. Read Adventures_Holmes.txt text file
2. Save contents into a string object "holmes_doc"
3. Extract three typical sentences from three stories of this literature
4. Use method learnt to create own POS taggers. What are new POS tags to add
or use?
5. Try new POS taggers for these three typical sentences and compare results
with previous workshop
312 12 Workshop#3 POS Tagging Using NLTK (Hour 5–6)

References

Albrecht, J., Ramachandran, S. and Winkler, C. (2020) Blueprints for Text Analytics Using
Python: Machine Learning-Based Solutions for Common Real World (NLP) Applications.
O’Reilly Media.
Antic, Z. (2021) Python Natural Language Processing Cookbook: Over 50 recipes to understand,
analyze, and generate text for implementing language processing tasks. Packt Publishing.
Bird, S., Klein, E., and Loper, E. (2009). Natural language processing with python. O'Reilly.
Doyle, A. C. (2019) The Adventures of Sherlock Holmes (AmazonClassics Edition).
AmazonClassics.
Gutenberg (2022) Project Gutenberg official site. https://www.gutenberg.org/ Accessed 16
June 2022.
Hardeniya, N., Perkins, J. and Chopra, D. (2016) Natural Language Processing: Python and
NLTK. Packt Publishing.
Kedia, A. and Rasu, M. (2020) Hands-On Python Natural Language Processing: Explore tools and
techniques to analyze and process text with a view to building real-world NLP applications.
Packt Publishing.
NLTK (2022) NLTK official site. https://www.nltk.org/. Accessed 16 June 2022.
Perkins, J. (2014). Python 3 text processing with NLTK 3 cookbook. Packt Publishing Ltd.
Sketchengine (2022a) Recent version of English POS Tagset by Sketchengine. https://www.
sketchengine.eu/english-treetagger-pipeline-2/. Accessed 21 June 2022.
Sketchengine (2022b) Recent version of Chinese POS Tagset by Sketchengine. https://www.
sketchengine.eu/chinese-penn-treebank-part-of-speech-tagset/. Accessed 21 June 2022.
Treebank (2022) Penn TreeBank Release 2 official site. https://catalog.ldc.upenn.edu/docs/
LDC95T7/treebank2.index.html. Accessed 21 June 2022.
Chapter 13
Workshop#4 Semantic Analysis and Word
Vectors Using spaCy (Hour 7–8)

13.1 Introduction

In Chaps. 5 and 6, we studied the basic concepts and theories related to meaning
representation and semantic analysis. This workshop will explore how to use spaCy
technology to perform semantic analysis starting from a revisit on word vectors
concept, implement and pre-train them followed by the study of similarity method
and other advanced semantic analysis.

13.2 What Are Word Vectors?

Word vectors (Albrecht et al. 2020; Bird et al. 2009; Hardeniya et al. 2016; Kedia
and Rasu 2020; NLTK 2022) are practical tools in NLP.
A word vector is a dense representation of a word. Word vectors are important
for semantic similarity applications like similarity calculations between words,
phrases, sentences, and documents, e.g. they provide information about synonym-
ity, semantic analogies at word level.
Word vectors are produced by algorithms to reflect similar words appear in simi-
lar contexts. This paradigm captures target word meaning by collecting information
from surrounding words which is called distributional semantics.
They are accompanied by associative semantic similarity methods including
word vector computations such as distance, analogy calculations, and visualization
to solve NLP problems.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 313
Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_13
314 13 Workshop#4 Semantic Analysis and Word Vectors Using spaCy (Hour 7–8)

In this workshop, we are going to cover the following main topics (Altinok 2021;
Arumugam and Shanmugamani 2018; Perkins 2014; spaCy 2022; Srinivasa-
Desikan 2018; Vasiliev 2020):
• Understanding word vectors.
• Using spaCy’s pre-trained vectors.
• Advanced semantic similarity methods.

13.3 Understanding Word Vectors

Word vectors, or word2vec are important quantity units in statistical methods to

represent text in statistical NLP algorithms. There are several ways of text vector-
ization to provide words semantic representation.

13.3.1 Example: A Simple Word Vector

Let us look at a basic way to assign words vectors:

• Assign an index value to each word in vocabulary and encode this value into a
sparse vector.
• Consider tennis as vocabulary and assign an index to each word according to
vocabulary order as in Fig. 13.1:
Vocabulary word vector will be 0, except for word corresponding index value
position as in Fig. 13.2:

Fig. 13.1 A basic word

vector example consists of
9 words
13.3 Understanding Word Vectors 315

Fig. 13.2 Word vectors

corresponding index value
consists of 9 words

Fig. 13.3 Word vector

matrix for I play tennis
today

Since each row corresponds to one word, a sentence represents a matrix, e.g. I
play tennis today is represented by matrix as in Fig. 13.3:
Vectors length is equal to word numbers in vocabulary as shown above. Each
dimension is apportioned to one word explicitly. When applying this encoding vec-
torization to text, each word is replaced by its vector and the sentence is transformed
into a (N, V) matrix, where N is words number in sentence and V is vocabulary size.
This text representation is easy to compute, debug, and interpret. It looks good so
far but there are potential problems:
• Vectors are sparse. Each vector contains many 0 s but has one 1. If words have
similar meanings, they can group to share dimensions, this vector will deplete
space. Also, numerical algorithms do not accept high dimension and sparse vec-
tors in general.
• A sizeable vocabulary is comparable to high dimensions vectors that are imprac-
tical for memory storage and computation.
316 13 Workshop#4 Semantic Analysis and Word Vectors Using spaCy (Hour 7–8)

• Similar words do not assign with similar vectors resulting in unmeaningful vec-
tors, e.g. cheese, topping, salami, and pizza have related meanings but have unre-
lated vectors. These vectors depend on corresponding word’s index and assign
randomly in vocabulary, indicating that one-hot encoded vectors are incapable to
capture semantic relationships and against word vectors’ purpose to answer pre-
ceding list concerns.

13.4 A Taste of Word Vectors

A word vector is a fixed-size, dense, and real-valued vector. It is a learnt representa-

tion of text where semantic similar words correspond to similar vectors and a solu-
tion to preceding problems.

This is a 50-dimensional vector for word the, these dimensions have floating points:
1. What do dimensions represent?
2. These individual dimensions do not have inherent meanings typically but
instead they represent vector space locations, and the distance between these
vectors indicates the similarity of corresponding words’ meanings
3. Hence, a word’s meaning is distributed across dimensions
4. This type of word’s meaning representation is called distributional semantics

Use word vector visualizer for TensorFlow from (TensorFlow 2022) Google
which offers word vectors for 10,000 words. Each vector is 200-dimensional and
projected into three dimensions for visualization. Let us look at the representation
of tennis as in Fig. 13.4:

tennis is semantically grouped with other sports, i.e. hockey, basketball, chess, etc.
Words in proximity are calculated by their cosine distances as shown in Fig. 13.5
13.4 A Taste of Word Vectors 317

Fig. 13.4 Vector representation of tennis and semantic similar words

Word vectors are trained on a large corpus such as Wikipedia which included to
learn proper nouns representations, e.g. Alice is a proper noun represented by vector
as in Fig. 13.6:
It showed that all vocabulary input words are in lower cases to avoid multiple
representations of the same word. Alice and Bob are person names to be listed. In
addition, lewis and carroll have relevance to Alice because of the famous literature
Alice’s Adventures in Wonderland written by Lewis Carroll. Further, it also showed
syntactic category of all neighboring words are nouns but not verbs.
Word vectors can capture synonyms, antonyms, and semantic categories such as
animals, places, plants, names, and abstract concepts.
318 13 Workshop#4 Semantic Analysis and Word Vectors Using spaCy (Hour 7–8)

Fig. 13.5 tennis proximity

words in three-dimensional
space
13.5 Analogies and Vector Operations 319

Fig. 13.6 Vector representation of alice

13.5 Analogies and Vector Operations

Word vectors capture semantics, support vector addition, subtraction, and analo-
gies. A word analogy is a semantic relationship between a pair of words. There are
many relationship types such as synonymity, anonymity, and whole-part relation.
Some example pairs are (King—man, Queen—woman), (airplane—air, ship—sea),
(fish—sea, bird—air), (branch—tree, arm—human), (forward—backward, absent—
present) etc.
For example, gender mapping represents Queen and King as Queen—Woman +
Man = King. If woman is subtracted by Queen and add Man instead to obtain King.
Then, this analogy interprets queen is attributed to king as woman is attributed to
man. Embeddings can generate analogies such as gender, tense, and capital city as
shown in Fig. 13.7:
320 13 Workshop#4 Semantic Analysis and Word Vectors Using spaCy (Hour 7–8)

Fig. 13.7 Analogies created by word vectors

13.6 How to Create Word Vectors?

There are many ways to produce and pre-trained word vectors:

1. word2vec is a name of statistical algorithm created by Google to produce word
vectors. Word vectors are trained with a neural network architecture to process
windows of words and predicts each word vector depending on surrounding
words. These pre-trained word vectors can be downloaded from Synthetic (2022).
2. Glove vectors are invented by Stanford NLP group. This method depends on
singular value decomposition used in word co-occurrences matrix. The pre-
trained vectors are available at nlp.stanford.edu (Stanford 2022).
3. fastText (FastText 2022) was created by Facebook Research like word2vec.
word2vec predicts words based on their surrounding context, while fastText pre-
dicts subwords i.e. character N-grams. For example, the word chair generates
the following subwords:

13.7 spaCy Pre-trained Word Vectors

Word vectors are part of many spaCy language models. For instance, en_core_web_
md model ships with 300-dimensional vectors for 20,000 words, while en_core_
web_lg model ships with 300-dimensional vectors with a 685,000 words vocabulary.
Typically, small models (names end with sm) do not include any word vectors
but context-sensitive tensors. Semantic similarity calculations can perform but
results will not be accurate as word vector computations.
A word’s vector is via token.vector method. Let us look at this method using
code query word vector for banana:
13.7 spaCy Pre-trained Word Vectors 321

In[1] # Import spaCy and load the en_core_web_md model

import spacy
nlp = spacy.load("en_core_web_md")

# Create a sample utterance (utt1)

utt1 = nlp("I ate a banana.")

In[2] import en_core_web_md

nlp = en_core_web_md.load()

Use the following script to show Word Vector for banana:

In[3] utt1[3].vector
Out[3]

In this example, token.vector returns a NumPy ndarray.

Use the following command to call NumPy methods for result.

In[4] type(utt1[3].vector)
Out[4] numpy.ndarray
322 13 Workshop#4 Semantic Analysis and Word Vectors Using spaCy (Hour 7–8)

In[5] utt1[3].vector.shape
Out[5] (300,)

Query Python type of word vector in this code segment. Then, invoke shape()
method of NumPy array on the vector.

Doc and Span objects also have vectors. A sentence vector or a span is the aver-
age of words’ vectors. Run following code and view results:

In[6] # Create second utterance (utt2)

utt2 = nlp("I like a banana,")
utt2.vector
utt2[1:3].vector
Out[6]
13.8 Similarity Method in Semantic Analysis 323

Only words in model’s vocabulary have vectors, words are not in vocabulary are
called out-of-vocabulary (OOV) words. token.is_oov and token.has_vector are
two methods to query whether a token is in the model’s vocabulary and has a
word vector:

In[7] # Create the utterance 3

utt3 = nlp("You went there afskfsd.")

In[8] for token in utt3:

print( "Token is: ",token, "OOV: ", token.is_oov, "Token has vector:" ,
token.has_vector)
Out[8]

This is basically how to use spaCy’s pretrained word vectors. Next, discover
how to invoke spaCy’s semantic similarity method on Doc, Span, and Token
objects.

13.8 Similarity Method in Semantic Analysis

Every container type object has a similarity method that allows to calculate seman-
tic similarity of other container objects by comparing word vectors in spaCy.
Semantic similarity between two container objects is different container types. For
instance, a Token object to a Doc object and a Doc object to a Span object.
The following example computes two Span objects similarity:

In[9] # Create utt4 and utt5 and measure the similarity

utt4 = nlp("I visited England.")
utt5 = nlp("I went to London.")
utt4[1:3].similarity(utt5[1:4])
Out[9] 0.6539691090583801
324 13 Workshop#4 Semantic Analysis and Word Vectors Using spaCy (Hour 7–8)

Compare two Token objects, London and England:

In[10] utt4[2]
Out[10] England

In[11] utt4[2].similarity(utt5[3])
Out[11] 0.7389127612113953

The sentence’s similarity is computed by calling similarity() on Doc objects:

In[12] utt4.similarity(utt5)
Out[12] 0.8771558796234277

1. The preceding code segment calculates semantic similarity between two sentences
I visited England and I went to London
2. Similarity score is high enough to consider both sentences are similar (similarity
degree ranges from 0 to 1, 0 represents unrelated and 1 represents identical)

similarity() method returns 1 compare an object to itself unsurprisingly:

In[13] utt4.similarity(utt4)
Out[13] 1.0

Judge the distance with numbers is complex but review vectors on paper can
understand how vocabulary word groups are formed.
Code snippet below visualizes a vocabulary of two graphical semantic classes.
The first word class is for animals and the second class is for food.

In[14] import matplotlib.pyplot as plt

from sklearn.decomposition import PCA
import numpy as np
import spacy

nlp = spacy.load( "en_core_web_md" )

vocab = nlp( "cat dog tiger elephant bird monkey lion cheetah burger pizza
food cheese wine salad noodles macaroni fruit vegetable" )

words = [word.text for word in vocab]

13.8 Similarity Method in Semantic Analysis 325

Create Word Vector vecs:

In[15] vecs = np.vstack([word.vector for word in vocab if word.has_vector])

Use PCA (Principal Component Analysis) similarity analysis and plot similarity
results with plt class.

In[16] pca = PCA(n_components=2)

vecs_transformed = pca.fit_transform(vecs)
plt.figure(figsize=(20,15))
plt.scatter(vecs_transformed[:,0], vecs_transformed[:,1])
for word, coord in zip(words, vecs_transformed):
x,y = coord
plt.text(x,y,word, size=15)
plt.show()
Out[16]

1. Import matplotlib library to create a graph.

2. Next two imports are for vectors calculation.
3. Import spaCy and create a nlp object.
4. Create a Doc object from vocabulary.
5. Stack word vectors vertically by calling np.vstack.
6. Project vectors into a two-dimensional space for visualization since they are
300-dimensional. Extract two principal components via principal component
analysis (PCA) for projection.
7. Create a scatter plot for rest of the code to deal with matplotlib function calls.
326 13 Workshop#4 Semantic Analysis and Word Vectors Using spaCy (Hour 7–8)

It showed that spaCy word vectors can visualize two semantic classes that are
grouped. The distance between animals is reduced and uniformly distributed, while
food class formed groups within the group.

Workshop 4.1 Word Vector Analysis on The Adventures of Sherlock Holmes

In this workshop, we have just learnt how to use spaCy to produce word vector to
compare the similarity of two text objects/document. Try to use The Adventures
of Sherlock Holmes (Doyle 2019; Gutenberg 2022) to select two "presentative"
texts from this detective story:
1. Read Adventures_Holmes.txt text file.
2. Save contents into a string object "holmes_doc".
3. Plot Semantic Graphs for these two texts.
4. Perform Similarity text for these two documents. See what are found.

13.9 Advanced Semantic Similarity Methods with spaCy

It has learnt that spaCy’s similarity method can calculate semantic similarity to
obtain scores but there are advanced semantic similarity methods to calculate words,
phrases, and sentences similarity.

13.9.1 Understanding Semantic Similarity

It is necessary to identify example characteristics when collecting data or text data

(any sort of data), i.e. calculate two text similarity scores. Semantic similarity is a
metric to define the distance between texts based on semantics texts.
A mathematics is basically a distance function. Every metric induces a topology
on vector space. Word vectors are vectors that can be used to calculate the distance
between them as a similarity score.
There are two commonly used distance functions: (1) Euclidian distance and (2)
cosine distance.

13.9.2 Euclidian Distance

Euclidian distance counts on vector magnitude and disregards orientation. If a vec-

tor is drawn from an origin, let us call it a dog vector to another point, call a cat
vector and subtract one vector from and other, the distance represents the magnitude
of vectors is shown in Fig. 13.8.
If two or more semantically similar words (canine, terrier) to dog and make it a
text of three words, i.e. dog canine terrier. Obviously, the dog vector will now grow
in magnitude, possibly in the same direction. This time, the distance will be much
13.9 Advanced Semantic Similarity Methods with spaCy 327

Fig. 13.8 Euclidian

distance between two
vectors: dog and cat

Fig. 13.9 Distance

between dog and cat, as
well as the distance
between dog canine terrier
and cat

bigger due to geometry, although the semantics of first piece of text (now dog canine
terrier) remain the same.
This is the main drawback of using Euclidian distance for semantic similarity as
the orientation of two vectors in the space is not considered. Figure 13.9 illustrates
the distance between dog and cat, and the distance between dog canine terrier
and cat.
How can we fix this problem? There is another way of calculating similarity
called cosine similarity to address this problem.

13.9.3 Cosine Distance and Cosine Similarity

Contrary to Euclidian distance, cosine distance is more concerned with the orienta-
tion of two vectors in the space. The cosine similarity of two vectors is basically the
cosine of angle created by these two vectors. Figure 13.10 shows the angle between
dog and cat vectors:
328 13 Workshop#4 Semantic Analysis and Word Vectors Using spaCy (Hour 7–8)

Fig. 13.10 The angle

between dog and cat
vectors. Here, the semantic
similarity is calculated by
cos(θ)

The maximum similarity score that is allowed by cosine similarity is 1. This is

obtained when the angle between two vectors is 0 degree (hence, the vectors coin-
cide). The similarity between two vectors is 0 when the angle between them is 90
degrees.
Cosine similarity provides scalability when vectors grow in magnitude. If one of
the input vectors is expanded as in Fig. 13.10, the angle between them remains the
same and so as the cosine similarity score.
Note that here is to calculate semantic similarity score and not distance. The
highest possible value is 1 when vectors coincide, while the lowest score is 0 when
two vectors are perpendicular. The cosine distance is 1 – cos(θ) which is a distance
function.
spaCy uses cosine similarity to calculate semantic similarity. Hence, calling the
similarity method helps to perform cosine similarity calculations.
So far, we have learnt to calculate similarity scores, but still have not discovered
words meaning. Obviously, not all words in a sentence have the same impact on the
semantics of sentence. The similarity method will only calculate the semantic simi-
larity score, but the right keywords are required for calculation results comparison.
Consider the following text snippet:

If is interested in finding the biggest mammals on the planet, the phrases biggest
mammals and in the world will be keywords. By comparing these phrases with the
search phrases largest mammals and on the planet should give a high similarity
score. But if is interested in finding out about places in the world, California will be
keyword. California is semantically like word geography and more suitably, the
entity type is a geographical noun.
Since we have learnt how to calculate similarity score, the next section will learn
about where to look for the meaning. It will cover a case study on text categorization
before improving task results via key phrase extraction with similarity score
calculations.
13.9 Advanced Semantic Similarity Methods with spaCy 329

13.9.4 Categorizing Text with Semantic Similarity

Determining two sentences’ semantic similarity can categorize texts into predefined
categories or spot only the relevant texts. This case study will filter users’ comments
in an e-commerce website related to the word perfume. Suppose to evaluate the fol-
lowing comments:

Here, it is noted that only the second sentence is related. This is because it con-
tains the word fragrance and adjectives describing scents. To understand which sen-
tences are related, can try several comparison strategies.
To start, compare perfume to each sentence. Recall that spaCy generates a word
vector for a sentence by averaging the word vector of its tokens. The following code
snippet compares preceding sentences to perfume search key:

In[17] utt6 = nlp( "I purchased a science fiction book last week. I loved everything
related to this fragrance: light, floral and feminine... I purchased a bottle of
wine. " )
key = nlp( "perfume" )
for utt in utt6.sents:
print(utt.similarity(key))

Out[17]

The following steps are performed:

Create a Doc object with three preceding sentences. For each sentence, calculate
similarity score with perfume and print the score by invoking similarity() method on
the sentence. The degree of similarity between perfume and the first sentence is
minute, indicating that this sentence is irrelevant to the search key. The second sen-
tence looks relevant which means that semantic similarity is correctly identified.
How about the third sentence? The script identified that the third sentence is
relevant somehow, most probably because it includes the word bottle and perfumes
are sold in bottles. The word bottle appears in similar contexts with the word per-
fume. For this reason, the similarity score of this sentence and search key is not
small enough; also, the scores of second and third sentences are not distant enough
to make the second sentence significant.
In practice, long texts such as web documents can be dealt with but averaging
over them diminish the importance of keywords.
330 13 Workshop#4 Semantic Analysis and Word Vectors Using spaCy (Hour 7–8)

Let us look at how to identify key phrases in a sentence to improve

performance.

13.9.5 Extracting Key Phrases

Semantic categorization is more effectively to extract important word phrases and

compare them to the search key. Instead of comparing the key to different parts of
speech, we can compare the key to noun phrases. Noun phrases are subjects, direct
objects, and indirect objects of sentences that convey high percentages of sentences
semantics.
For example, in sentence Blue whales live in California, focuses will likely be on
blue whales, whales, California, or whales in California.
Similarly, in the preceding sentence about perfume, the focused is to pick out
fragrance the noun. Different semantic tasks may need other context words such as
verbs to decide what the sentence is about, but for semantic similarity, noun phrases
convey significant weights.
What is a noun phrase? A noun phrase (NP) is a group of words that consist of a
noun and its modifiers. Modifiers are usually pronouns, adjectives, and determiners.
The following phrases are noun phrases:

spaCy extracts noun phases by parsing the output of dependency parser. It can
identify noun phrases of a sentence by using doc.noun_chunks method:

In[18] utt7 = nlp( "My beautiful and cute dog jumped over the fence" )

In[19] utt7.noun_chunks
Out[19] <generator at 0x1932f2de900>

In[20] list(utt7.noun_chunks)
Out[20] [My beautiful and cute dog, the fence]
13.9 Advanced Semantic Similarity Methods with spaCy 331

Let us modify the preceding code snippet. Instead of comparing the search key
perfume to the entire sentence, this time will only compare it with sentence’s
noun chunks:

In[21] for utt in utt7.sents:

nchunks = [nchunk.text for nchunk in utt.noun_chunks]
nchunk_utt = nlp(" ".join(nchunks))
print(nchunk_utt.similarity(key))

Out[21] 0.27409999728254997

The following is performed for the preceding code:

1. Iterate over sentences
2. Extract noun chunks and store them in a Python list for each sentence
3. Join noun chunks in the list into a Python string and convert it into a Doc object
4. Compare this Doc object of noun chunks to search key perfume to determine
semantic similarity scores

If these scores are compared with previous scores, it is noted that that the first
sentence remains irrelevant, so its score decreased marginally but the second sen-
tence’s score increased significantly. Also, the second and third sentences scores are
distant from each other to reflect that second sentence is the most related sentence.

13.9.6 Extracting and Comparing Named Entities

In some cases, it can focus to extract proper nouns instead of every noun. Hence, it
is required to extract named entities. Let us compare the following paragraphs:

The codes should be able to recognize that first two paragraphs are about large
technology companies and their products, whereas the third paragraph is about a
geographic location.
Comparing all noun phrases in these sentences may not be helpful because many
of them such as volume are irrelevant to categorization. The topics of these para-
graphs are determined by phrases within them, that is, Google Search, Google,
Microsoft Bing, Microsoft, Windows, Dead Sea, Jordan Valley, and Israel. spaCy
can identify these entities:
332 13 Workshop#4 Semantic Analysis and Word Vectors Using spaCy (Hour 7–8)

In[22] utt8 = nlp( "Google Search, often referred as Google, is the most popular
search engine nowadays. It answers a huge volume of queries every day." )
utt9 = nlp( "Microsoft Bing is another popular search engine. Microsoft is
known by its star product Microsoft Windows, a popular operating system
sold over the world." )
utt10 = nlp( "The Dead Sea is the lowest lake in the world, located in the
Jordan Valley of Israel. It is also the saltiest lake in the world." )

In[23] utt8.ents
Out[23] (Google Search, Google, every day)

In[24] utt9.ents
Out[24] (Microsoft Bing, Microsoft, Microsoft Windows)

In[25] utt10.ents
Out[25] (the Jordan Valley, Israel)

Since words are extracted for comparison, let’s calculate similarity scores:

In[26] ents1 = [ent.text for ent in utt8.ents]

ents2 = [ent.text for ent in utt9.ents]
ents3 = [ent.text for ent in utt10.ents]
ents1 = nlp(" ".join(ents1))
ents2 = nlp(" ".join(ents2))
ents3 = nlp(" ".join(ents3))

In[27] ents1.similarity(ents2)
Out[27] 0.5394545341415748

In[28] ents1.similarity(ents3)
Out[28] 0.48605042335384385
In[29] ents2.similarity(ents3)
Out[29] 0.39674953175052086
References 333

These figures revealed that the highest level of similarity exists between first and
second paragraphs, which are both about large tech companies. The third
paragraph is unlike other paragraphs. How can this calculation be obtained by
using word vectors only? It is probably because words Google and Microsoft often
appear together in news and other social media text corpora, hence producing
similar word vectors

This is the conclusion of advanced semantic similarity methods section with dif-
ferent ways to combine word vectors with linguistic features such as key phrases
and named entities.

Workshop 4.2 Further Semantic Analysis on The Adventures of Sherlock

Holmes
It has learnt to further improve semantic Analysis results on document similarity
comparison by extracting (1) key phrases; (2) and comparing names entities. Try
to use these techniques on The Adventures of Sherlock Holmes:
1. Extract three "representative texts" from this novel
2. Perform key phrases extraction to improve the similarity rate as compared
with Workshop 4.1 results
3. Extract and compare name entities to identify significant name entities from
this literature to further improve semantic analysis performance
4. Remember to plot semantic diagram to show how these entities and keywords
are related
5. Discuss and explain what are found

References

Albrecht, J., Ramachandran, S. and Winkler, C. (2020) Blueprints for Text Analytics Using
Python: Machine Learning-Based Solutions for Common Real World (NLP) Applications.
O’Reilly Media.
Altinok, D. (2021) Mastering spaCy: An end-to-end practical guide to implementing NLP applica-
tions using the Python ecosystem. Packt Publishing.
Arumugam, R., & Shanmugamani, R. (2018). Hands-on natural language processing with python.
Packt Publishing.
Bird, S., Klein, E., and Loper, E. (2009). Natural language processing with python. O'Reilly.
Doyle, A. C. (2019) The Adventures of Sherlock Holmes (AmazonClassics Edition).
AmazonClassics.
FastText (2022) FastText official site. https://fasttext.cc/. Accessed 22 June 2022.
Gutenberg (2022) Project Gutenberg official site. https://www.gutenberg.org/ Accessed 16
June 2022.
Hardeniya, N., Perkins, J. and Chopra, D. (2016) Natural Language Processing: Python and
NLTK. Packt Publishing.
Kedia, A. and Rasu, M. (2020) Hands-On Python Natural Language Processing: Explore tools and
techniques to analyze and process text with a view to building real-world NLP applications.
Packt Publishing.
NLTK (2022) NLTK official site. https://www.nltk.org/. Accessed 16 June 2022.
Perkins, J. (2014). Python 3 text processing with NLTK 3 cookbook. Packt Publishing Ltd.
SpaCy (2022) spaCy official site. https://spacy.io/. Accessed 16 June 2022.
334 13 Workshop#4 Semantic Analysis and Word Vectors Using spaCy (Hour 7–8)

Srinivasa-Desikan, B. (2018). Natural language processing and computational linguistics: A prac-

tical guide to text analysis with python, gensim, SpaCy, and keras. Packt Publishing Limited.
Stanford (2022) NLP.stanford.edu Glove official site. https://nlp.stanford.edu/projects/glove/.
Accessed 22 June 2022.
Synthetic (2022) Synthetic Intelligent Network site on Word2Vec Model. https://developer.syn.
co.in/tutorial/bot/oscova/pretrained-vectors.html#word2vec-and-glove-models. Accessed 22
June 2022.
TensorFlow (2022) TensorFlow official site. https://projector.tensorflow.org/. Accessed 22
June 2022.
Vasiliev, Y. (2020) Natural Language Processing with Python and spaCy: A Practical Introduction.
No Starch Press.
Chapter 14
Workshop#5 Sentiment Analysis and Text
Classification with LSTM Using spaCy
(Hour 9–10)

14.1 Introduction

NLTK and spaCy are two major NLP Python implementation tools for basic text
processing, N-gram modeling, POS tagging, and semantic analysis introduced in
last four workshops. Workshop 5 will explore how to position these NLP implemen-
tation techniques into two important NLP applications: text classification and senti-
ment analysis. TensorFlow and Kera are two vital components to implement
Long-Short Term Memory networks (LSTM networks), a commonly used Recurrent
Neural Networks (RNN) on machine learning especially in NLP applications.
This workshop will:
1. study text classification concepts in NLP and how spaCy NLP pipeline works on
text classifier training.
2. use movie reviews as a problem domain to demonstrate how to implement senti-
ment analysis with spaCy.
3. introduce Artificial Neural Networks (ANN) concepts, TensorFlow, and Kera
technologies.
4. introduce sequential modeling scheme with LSTM technology using movie
reviews domain as example to integrate these technologies for text classification
and movie sentiment analysis.

14.2 Text Classification with spaCy and LSTM Technology

Text classification is a vital component in sentiment analysis application.

TextCategorizer is a spaCy’s text classifier component applied in dataset for sen-
timent analysis to perform text classification with two vital Python frameworks: (1)
TensorFlow Keras API and (2) spaCy technology.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 335
Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_14
336 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

Sequential data modelling with LSTM technology is used to process text for
machine learning tasks with Keras’s text preprocessing module and implement a
neural network with tf.keras.
This workshop will cover the following key topics:
• Basic concept and knowledge of text classification.
• Model training of spaCy text classifier.
• Sentiment Analysis with spaCy.
• Sequential modeling with LSTM Technology.

14.3 Technical Requirements

Codes for training spaCy text classifier and sentiment analysis are spaCy v3.0 com-
patible. Text classification with spaCy and Keras requires Python libraries as
follows:
• TensorFlow (version 2.3 or above)
• NumPy
• pandas
• Matplotlib
If these packages are not installed into PC/notebook, use pip install xxx command.

14.4 Text Classification in a Nutshell

14.4.1 What Is Text Classification?

Text Classification (Albrecht et al. 2020; Bird et al. 2009; George 2022; Sarkar
2019; Siahaan and Sianipar 2022; Srinivasa-Desikan 2018) is the task of assigning
a set of predefined labels to text.
They are classified by manual tagging, but machine learning techniques are
applied progressively to train classification system with known examples, or train
samples to classify unseen cases. It is a fundamental task of NLP (Perkins 2014;
Sarkar 2019) using various machine learning method such as LSTM technology
(Arumugam and Shanmugamani 2018; Géron 2019; Kedia and Rasu 2020).
Text classification types are (Agarwal 2020; George 2022; Pozzi et al. 2016):
• Language detection is the first step of many NLP systems, i.e. machine translation.
• Topic generation and detection are the process of summarization, or classifica-
tion of a batch of sentences, paragraphs, or texts into certain Topic of Interest
(TOI) or topic titles, e.g. customers’ email request refund or complaints about
products or services.
14.4 Text Classification in a Nutshell 337

• Sentiment analysis to classify or analyze users’ responses, comments, and mes-

sages on a particular topic attribute to positive, neutral, or negative sentiments. It
is an essential task in e-commerce and social media platforms.
Text classifiers can emphasize on overall text sentiments, text language detec-
tion, and words levels, i.e. verbs. A text classifier of a customer service automation
system is shown in Fig. 14.1.

14.4.2 Text Classification as AI Applications

Text classification is considered as Supervised Learning (SL) task in AI which

means that the classifier can predict class label of a text based on sample input text-
class label pairs. It must require sufficient input (text)-output (classified labels) pairs
databank for network training, testing, and validation. Hence, a labeled dataset is a
list of text-label pairs required to train a text classifier. An example dataset of five
training sentences with sentiment labels is shown in Fig. 14.2.
When a classifier encounters a new text which is not in the training text, it pre-
dicts a class label of this unseen text based on examples during training phase to
induce a text classifier output is always a class label.

Fig. 14.1 Example of top detection for customer complaint in CSAS (Customer Service
Automation System)

Fig. 14.2 Sample input texts and their corresponding output class labels
338 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

Text classification can also be divided into (1) binary, (2) multi-class, and (3)
multi-label categories:
1. Binary text classification refers to categorize text with two classes.
2. Multi-class text classification refers to categorize texts with more than two
classes. Each class is mutually exclusive where one text is associated with single
class, e.g. rating customer reviews are represented by 1–5 stars category single
class label.
3. Multi-label text classification system is to generalize its multi-class counterpart
assigned to each example text e.g. toxic, severe toxic, insult, threat, obscenity
levels of negative sentiment. What are Labels in Text Classification?
Labels are class names for output. A class label can be categorical (string) or
numerical (a number).
Text classification has the following class labels:
• Sentiment analysis has positive and negative class labels abbreviated by pos and
neg where 0 represents negative sentiment and 1 represents positive sentiment.
Binary class labels are popular as well.
• The identical numeric representation applies to binary classification problems,
i.e. use 0–1 for class labels.
• Class labeled with a meaningful name for multi-class and multi-label problems,
e.g. movie genre classifier has labels action, scifi, weekend, Sunday movie, etc.
Numbers are labels for a five-class classification problem, i.e. 1–5.

14.5 Text Classifier with spaCy NLP Pipeline

TextCategorizer (tCategorizer) is spaCy’s text classifier component (Altinok 2021;

SpaCy 2022; Vasiliev 2020). It required class labels and examples in NLP pipeline
to perform training procedure as shown in Fig. 14.3.
TextCategorizer provides user-friendly and end-to-end approaches to train clas-
sifier so that it does not need to deal with neural network architecture directly.

Fig. 14.3 TextCategorizer in the spaCy NLP pipeline

14.5 Text Classifier with spaCy NLP Pipeline 339

14.5.1 TextCategorizer Class

Import spaCy and load nlp component from "en_core_web_md":

In[1] # Load and import spacy package

import spacy
# Load the en_core_web_md module
nlp = spacy.load( "en_core_web_md" )

Import TextCategorizer from spaCy pipeline components:

In[2] # Import the Single Text Categorizer Model

from spacy.pipeline.textcat import
DEFAULT_SINGLE_TEXTCAT_MODEL

TextCategorizer consists of (1) single-label and (2) multi-label classifiers.

A multi-label classifier can predict more than single class. A single-label classi-
fier predicts a single class for each example and classes are mutually exclusive.
The preceding import line imports single-label classifier, and the following code
imports multi-label classifier:

In[3] # Import the Multiple Text Categorizer Model

from spacy.pipeline.textcat_multilabel import
DEFAULT_MULTI_TEXTCAT_MODEL

There are two parameters (1) a threshold value and (2) a model name (either
Single or Multi depends on classification task) required for a TextCategorizer com-
ponent configuration.
TextCategorizer generates a probability for each class and a class is assigned to
text if the probability of this class is higher than the threshold value.
A traditional threshold value for text classification is 0.5; however, if prediction
is required for a higher confidence, it can adjust threshold to 0.6–0.8.
A single-label TextCategorizer (tCategorizer) component is added to nlp pipe-
line as follows:

In[4] # Import the Single Text Categorizer Model

# Define the model parameters: threshold and model
from spacy.pipeline.textcat import
DEFAULT_SINGLE_TEXTCAT_MODEL
config = {
"threshold": 0.5,
"model": DEFAULT_SINGLE_TEXTCAT_MODEL
}
340 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

In[5] # Define the Text Categorizer object (tCategorizer)

tCategorizer = nlp.add_pipe("textcat", config=config)

Let’s look at Text Categorizer object (tCategorizer):

In[6] tCategorizer
Out[6] <spacy.pipeline.textcat.TextCategorizer at 0x1bf406cedc0>

Add a multilabel component to nlp pipeline:

In[7] # Import the Multiple Text Categorizer Model

# Define the model parameters: threshold and model
from spacy.pipeline.textcat_multilabel import
DEFAULT_MULTI_TEXTCAT_MODEL
config = {
"threshold": 0.5,
"model": DEFAULT_MULTI_TEXTCAT_MODEL
}

In[8] tCategorizer = nlp.add_pipe( "textcat_multilabel" , config=config)

In[9] tCategorizer
Out[9]

Add a TextCategorizer pipeline component to nlp pipeline object at the last line of
each preceding code blocks. The newly created TextCategorizer component is
captured by textcat variable and set for training

14.5.2 Formatting Training Data for the TextCategorizer

Let us prepare a customer sentiment dataset for binary text classification.

The label (category) will be called sentiment to obtain two possible values, 0 and
1 corresponding to negative and positive sentiments.
There are six examples from IMDB with three each of positive and negative
as below:
14.5 Text Classifier with spaCy NLP Pipeline 341

In[10] movie_comment1 = [
("This movie is perfect and worth watching. ",
{"cats": {"Positive Sentiment": 1}}),
("This movie is great, the performance of Al Pacino is brilliant.",
{"cats": {"Positive Sentiment": 1}}),
("A very good and funny movie. It should be the best this year!",
{"cats": {"Positive Sentiment": 1}}),
("This movie is so bad that I really want to leave after the first hour
watching.", {"cats": {"Positive Sentiment": 0}}),
("Even free I won't see this movie again. Totally failure!",
{"cats": {"Positive Sentiment": 0}}),
("I think it is the worst movie I saw so far this year.",
{"cats": {"Positive Sentiment": 0}})
]]

Check on any movie comment1 element:

In[11] movie_comment1 [1]

Out[11] ('This movie is great, the performance of Al Pacino is brilliant.',
{'cats': {'Positive Sentiment': 1}})

• Each training example (movie_coment1) is a tuple object consists of a text and a

nested dictionary
• The dictionary contains a class category in a format recognized by spaCy
• The cats field means categories
• Include class category sentiment and its value. The value should always be a
floating-point number

The code will introduce a class category selected for TextCategorizer component.

In[12] import random

import spacy
from spacy.training import Example
from spacy.pipeline.textcat import
DEFAULT_SINGLE_TEXTCAT_MODEL

• Import a built-in library random to shuffle dataset

• Import spaCy as usual, then import Example to prepare training samples in spaCy
format
• Import TextCategorizer model in final statement
342 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

Initialize pipeline and TextCategorizer component.

When a new TextCategorizer component tCategorizer is created, use calling
add_label method to introduce category sentiment to TextCategorizer component
with examples.
The following code adds label to TextCategorizer component and initialize
TextCategorizer model’s weights with training samples:

In[13] import random

import spacy
from spacy.training import Example
from spacy.pipeline.textcat import
DEFAULT_SINGLE_TEXTCAT_MODEL

# Load the spaCy NLP model

nlp = spacy.load('en_core_web_md')

# Set the threshold and model

config = {
"threshold": 0.5,
"model": DEFAULT_SINGLE_TEXTCAT_MODEL
}

# Define TextCategorizer object (tCategorizer)

tCategorizer = nlp.add_pipe("textcat", config=config)

Let’s look at pipe_names:

In[14] nlp.pipe_names
Out[14] ['tok2vec', 'tagger', 'parser', 'attribute_ruler', 'lemmatizer',
'ner', 'textcat']

When a new TextCategorizer component textcat is created, use calling add_label

method to introduce label sentiment to the TextCategorizer component and initialize
this component with examples.
The following code adds label to TextCategorizer component and initializes
TextCategorizer model’s weights with training samples (movie_comment_exp):
14.5 Text Classifier with spaCy NLP Pipeline 343

In[15] # Create the two sentiment categories

tCategorizer.add_label("Positive Sentiment")
tCategorizer.add_label("Negative Sentiment")

# Create the movie comment samples

movie_comment_exp = [Example.from_dict(nlp.make_doc(comments),
category) for comments,category in movie_comment1]
tCategorizer.initialize(lambda: movie_comment_exp, nlp=nlp)

Let’s look at movie_comment_exp:

In[16] movie_comment_exp
Out[16]
344 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

14.5.3 System Training

Training loop is all set to be defined.

First, disable other pipe components to allow only textcat can be trained.
Second, create an optimizer object by calling resume_training to keep the
weights of existing statistical models.
Examine each epoch training example one by one and update the weights of
textcat. Examine data for 20 epochs.
Try the whole program with training loop:

In[17] movie_comment1
Out[17] [('This movie is perfect and worth watching. ',
{'cats': {'Positive Sentiment': 1}}),
('This movie is great, the performance of Al Pacino is brilliant.',
{'cats': {'Positive Sentiment': 1}}),
('A very good and funny movie. It should be the best this year!',
{'cats': {'Positive Sentiment': 1}}),
('This movie is so bad that I really want to leave after the first hour watch-
ing.',
{'cats': {'Positive Sentiment': 0}}),
("Even free I won't see this movie again. Totally failure!",
{'cats': {'Positive Sentiment': 0}}),
('I think it is the worst movie I saw so far this year.',
{'cats': {'Positive Sentiment': 0}})]
14.5 Text Classifier with spaCy NLP Pipeline 345

In[18] # Full implementation of the Movie Sentiment Analysis System

import random
import spacy
from spacy.training import Example
from spacy.pipeline.textcat import
DEFAULT_SINGLE_TEXTCAT_MODEL

# Load the spaCy NLP model

nlp = spacy.load('en_core_web_md')

# Set the threshold and model

config = {
"threshold": 0.5,
"model": DEFAULT_SINGLE_TEXTCAT_MODEL
}

# Create the TextCategorizer object (tCategorizer)

tCategorizer = nlp.add_pipe("textcat", config=config)

# Add the two movie sentiment categories

tCategorizer.add_label("Positive Sentiment")
tCategorizer.add_label("Negative Sentiment")

# Create the movie sample comments

movie_comment_exp = [Example.from_dict(nlp.make_doc(comments),
category) for comments,category in movie_comment1]
tCategorizer.initialize(lambda: movie_comment_exp, nlp=nlp)

# Set the training epochs and loss values

epochs=20
losses = {}

# Main program loop

with nlp.select_pipes(enable="textcat"):
optimizer = nlp.resume_training()
for i in range(epochs):
random.shuffle(movie_comment1)
for comments, category in movie_comment1:
mdoc = nlp.make_doc(comments)
exp = Example.from_dict(mdoc, category)
nlp.update([exp], sgd=optimizer, losses=losses)
print("Epoch #",i, "Losses: ",losses)
346 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

Out[18] Epoch # 0 Losses: {'textcat': 1.5183179080486298}

Epoch # 1 Losses: {'textcat': 2.7817106544971466}
Epoch # 2 Losses: {'textcat': 3.5506987050175667}
Epoch # 3 Losses: {'textcat': 3.8420143127441406}
Epoch # 4 Losses: {'textcat': 3.9003750435076654}
Epoch # 5 Losses: {'textcat': 3.9074860664550215}
Epoch # 6 Losses: {'textcat': 3.908426207563025}
Epoch # 7 Losses: {'textcat': 3.908603171435061}
Epoch # 8 Losses: {'textcat': 3.9086502377413126}
Epoch # 9 Losses: {'textcat': 3.908672676368724}
Epoch # 10 Losses: {'textcat': 3.9086846519847427}
Epoch # 11 Losses: {'textcat': 3.908692961549093}
Epoch # 12 Losses: {'textcat': 3.9086987949742706}
Epoch # 13 Losses: {'textcat': 3.9087034759107553}
Epoch # 14 Losses: {'textcat': 3.908707513363254}
Epoch # 15 Losses: {'textcat': 3.90871090364098}
Epoch # 16 Losses: {'textcat': 3.9087138364217537}
Epoch # 17 Losses: {'textcat': 3.9087164432144874}
Epoch # 18 Losses: {'textcat': 3.9087187692100116}
Epoch # 19 Losses: {'textcat': 3.908720835553922}

14.5.4 System Testing

Let us test a new textcategorizer component, doc.cats property holds the class labels:

In[19] # Test 1: This movie sucks

test1 = nlp("This movie sucks and the worst I ever saw.")
test1.cats

Out[19] {'Positive Sentiment': 0.05381184443831444,

'Negative Sentiment': 0.9461881518363953}

In[20] # Test 2: I'll watch it again, how amazing.

test2 = nlp("This movie really very great!")
test2.cats

Out[20] {'Positive Sentiment': 0.8159973621368408,

'Negative Sentiment': 0.1840025931596756}

The small dataset trained spaCy text classifier successfully for a binary text
classification problem to perform correct sentiment analysis. Now, let us perform
multi-label classification
14.5 Text Classifier with spaCy NLP Pipeline 347

14.5.5 Training TextCategorizer for Multi-Label Classification

Multi-label classification means the classifier can predict more than single label for
an example text. Naturally, the classes are not mutually exclusive.
Provide training samples with at least more than 2 categories to train a multiple
labelled classifier.
Construct a small training set to train spaCy‘s TextCategorizer for multi-label
classification. This time will form a set of movie reviews, where the multi-
category is:
• ACTION.
• SCIFI.
• WEEKEND.
Here is a small sample dataset (movie_comment2):

In[21] movie_comment2 = [
("This movie is great for weekend watching.",
{"cats": {"WEEKEND": True}}),
("This a 100% action movie, I enjoy it.",
{"cats": {"ACTION": True}}),
("Avatar is the best Scifi movie I ever seen!" ,
{"cats": {"SCIFI": True}}),
("Such a good Scifi movie to watch during the weekend!",
{"cats": {"WEEKEND": True, "SCIFI": True}}),
("Matrix a great Scifi movie with a lot of action. Pure action, great!",
{"cats": {"SCIFI": True, "ACTION": True}})
]

Check dataset first:

In[22] movie_comment2
Out[22] [('This movie is great for weekend watching.', {'cats': {'WEEKEND':
True}}),
('This a 100% action movie, I enjoy it.', {'cats': {'ACTION': True}}),
('Avatar is the best Scifi movie I ever seen!', {'cats': {'SCIFI': True}}),
('Such a good Scifi movie to watch during the weekend!',
{'cats': {'WEEKEND': True, 'SCIFI': True}}),
('Matrix a great Scifi movie with a lot of action. Pure action, great!',
{'cats': {'SCIFI': True, 'ACTION': True}})]
348 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

In[23] movie_comment2[1]
Out[23] ('This a 100% action movie, I enjoy it.', {'cats': {'ACTION': True}})

Provide examples with a single label, such as first example (the first sentence of
movie_comment2, the second line of preceding code block), and examples with
more than single label, such as fourth example of movie_comment2.
Import after the training set is formed.

In[24] import random

import spacy
from spacy.training import Example
from spacy.pipeline.textcat_multilabel import
DEFAULT_MULTI_TEXTCAT_MODEL

# Load spaCy NLP model

nlp = spacy.load( 'en_core_web_md' )

Note that the last line has different code than previous section. Import multi-label
model instead of single-label model.
Next, add multi-label classifier component to nlp pipeline.
Also note that pipeline component name is textcat_multilabel as compared with
previous section’s textcat:

In[25] # Set the threshold and model

config = {
"threshold": 0.5,
"model": DEFAULT_MULTI_TEXTCAT_MODEL
}

# Create the TextCategorizer object (tCategorizer)

tCategorizer = nlp.add_pipe( "textcat_multilabel" , config=config)

Add categories to TextCategorizer component and initialize model like previous

text classifier section.
14.5 Text Classifier with spaCy NLP Pipeline 349

Add three labels instead of one:

In[26] # Create the categorizer object with 3 categories

categories = ["SCIFI", "ACTION", "WEEKEND"]

# Using For Loop to add the 3 categories

for category in categories:
tCategorizer.add_label(category)

# Create the movie comment sample for training

movie_comment_exp = [Example.from_dict(nlp.make_doc(comments), cat-
egory) for comments,category in movie_comment2]

# Initializer the tCategorizer

tCategorizer.initialize(lambda: movie_comment_exp, nlp=nlp)

Training loop is all set to be defined.

Code functions are like previous section’s code, the only difference is component
name textcat_multilabel in the first line:

In[27] # Set the training epochs and loss values

epochs=20
losses = {}

# Main Loop of the program

with nlp.select_pipes(enable="textcat_multilabel"):
optimizer = nlp.resume_training()
for i in range(epochs):
random.shuffle(movie_comment2)
for comments, category in movie_comment2:
mdoc = nlp.make_doc(comments)
exp = Example.from_dict(mdoc, category)
nlp.update([exp], sgd=optimizer, losses=losses)
print(losses)
350 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

Out[27] {'textcat_multilabel': 6.85278207868123e-06}

{'textcat_multilabel': 1.3591816482971808e-05}
{'textcat_multilabel': 2.003331736943892e-05}
{'textcat_multilabel': 2.6209833507095937e-05}
{'textcat_multilabel': 3.211208475306648e-05}
{'textcat_multilabel': 3.780755992011109e-05}
{'textcat_multilabel': 4.3277938615915446e-05}
{'textcat_multilabel': 4.857392603696553e-05}
{'textcat_multilabel': 5.372697206951216e-05}
{'textcat_multilabel': 5.867910277856936e-05}
{'textcat_multilabel': 6.350277087108225e-05}
{'textcat_multilabel': 6.81268817857017e-05}
{'textcat_multilabel': 7.271952523524305e-05}
{'textcat_multilabel': 7.709734516936351e-05}
{'textcat_multilabel': 8.136703193883932e-05}
{'textcat_multilabel': 8.552625510560574e-05}
{'textcat_multilabel': 8.953699784797209e-05}
{'textcat_multilabel': 9.34374557175488e-05}
{'textcat_multilabel': 9.724928190735227e-05}
{'textcat_multilabel': 0.00010092528287941605}

The output should look like output of previous section but use multiple category
for system training. Let’s test the new multi-label classifier:

In[28] test3 = nlp("Definitely in my weekend scifi movie night list")

test3.cats
Out[28] {'SCIFI': 0.9995421171188354,
'ACTION': 0.5897207260131836,
'WEEKEND': 0.9992324113845825}

In[29] test4 = nlp("Go to watch action scifi movie this weekend.")

test4.cats
Out[29] {'SCIFI': 0.994023859500885,
'ACTION': 0.9914324283599854,
'WEEKEND': 0.9998989105224609}

Although sample size is small, but the multiple textcategorizer can classify two
IMDB user comments correctly into three categories: SCIFI, ACTION, and
WEEKEND. Note that over thousands of IMDB user comments are required to
perform a satisfactory sentiment analysis in real situations
14.6 Sentiment Analysis with spaCy 351

This section has learnt how to train a spaCy‘s TextCategorizer component for
binary and multilabel text classifications.
Now, TextCategorizer will be trained on a real-world dataset for a sentiment
analysis using IMDB user comments dataset.

Workshop 5.1 Movie comments from IMDB.com

Movie comments is a significant classification in social media. This workshop
constructs a simple movie comment classification with millions of user
comments from IMDB.com, the world biggest movie social media platform.
1. Try to collect 900 comments with 300 Good, 300 Average and 300 Bad
comments to train system. Make sure they make sense or system won’t
function
2. Construct a Multi-label Classification System to create three movie
comments: Good, Average or Bad
3. Train system with at least 100 epochs
4. Use 10 examples to test and see whether it works

14.6 Sentiment Analysis with spaCy

14.6.1 IMDB Large Movie Review Dataset

This section will work on a real-world dataset using IMDB Large Movie Reviews
Dataset from Kaggle (2022).
The original imdb_sup.csv dataset has 50,000 rows. They need to down-size and
select first 5000 records into datafile imdb_5000.csv to speed up training. This
movie reviews dataset consists of movie reviews, reviews sizes, IMDB Ratings
(1–10), and Sentiment Ratings (0 or 1).
The dataset can be downloaded from workshop directory namely: imdb_sup.csv
(complete dataset) or imdb_5000.csv (5000 records).

14.6.2 Explore the Dataset

Let us have some understanding from dataset prior sentiment analysis.

1. First, import to read and visualize dataset:

In[30] import pandas as pd

import matplotlib.pyplot as plt
%matplotlib inline
352 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

2. Read imdb_5000.csv datafile into a pandas DataFrame (mcommentDF) and out-

put the shape of DataFrame:

In[31] mcommentDF=pd.read_csv( 'imdb_5000.csv' )

In[32] mcommentDF.shape
Out[32] (5000, 3)

Note: This IMDB movie reviews dataset contains 5000 records, each record has 3
fields attributes: Review, Rating, and Sentiment
3. Examine rows and columns of dataset by printing the first few rows using
head() method:

In[33] mcommentDF.head()
Out[33]

4. Use Review and Sentiment columns only in this workshop. Hence, drop other
columns that won't use, and call dropna() method to drop the rows with miss-
ing values:

In[34] mcommentDF_clean = mcommentDF[[ 'Review', 'Sentiment' ]].dropna()

14.6 Sentiment Analysis with spaCy 353

In[35] mcommentDF_clean.head()
Out[35]

5. Let us look at how review scores are distributed:

In[36] axplot=mcommentDF.Rating.value_counts().plot
(kind='bar', colormap='Paired')
plt.show()
Out[36] 1400
1200

1000

800

600

400

200

0
10

1. Users prefer to give high rating, i.e. 8 or above, and 10 is the highest as shown
2. It is better to select sample set with even distribution to balance sample data rating
3. Check system performance first. If it is not as good as predicted, can use fine-tune
sampling method to improve system performance

Here use the sentiments already labeled.

354 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

6. Plot ratings distribution:

In[37] axplot=mcommentDF.Sentiment.value_counts().plot (kind= 'bar',

colormap='Paired')
plt.show()

Out[37]
2500

2000

1500

1000

500

0
0
1

Note that rating distribution has better results than the previous one, it has higher
number of positive reviews, but negative reviews are also significant as shown

After the dataset is processed, it can be reduced to a two-column dataset with

negative and positive ratings. So, call mcommentDF.head() again and the following
result is obtained:

In[38] mcommentDF.head()
Out[38]

Complete dataset exploration and display review scores with class categories
distribution. The dataset is ready to be processed. Drop unused columns and convert
review scores to binary class labels. Let us begin with the training procedure
14.6 Sentiment Analysis with spaCy 355

14.6.3 Training the TextClassfier

Use multi-label classifier to train a binary text classifier this time.

1. Import spaCy classes as follows:

In[39] import spacy

import random
from spacy.training import Example
from spacy.pipeline.textcat_multilabel import
DEFAULT_MULTI_TEXTCAT_MODEL

2. Create a pipeline object nlp, define classifier configuration, and add TextCategorizer
component to nlp with the following configuration:

In[40] # Load the spaCy NLP model

nlp = spacy.load( "en_core_web_md" )

# Set the threshold and model

config = {
"threshold": 0.5,
"model": DEFAULT_MULTI_TEXTCAT_MODEL
}

# Create the TextCategorizer object (tCategorizer)

tCategorizer = nlp.add_pipe("textcat_multilabel", config=config)

3. When TextCategorizer object is available, create movie comment sample object

as a list and load all user comments and categories into it.

In[41] # Create the IMDB movie comment sample object

movie_comment_exp = []

# Load all the IMDB user comments and categories

for idx, rw in mcommentDF.iterrows():
comments = rw["Review"]
rating = rw["Sentiment"]
category = {"POS": True, "NEG": False} if rating == 1 else
{"NEG": True, "POS": False}
movie_comment_exp.append(Example.from_dict(nlp.make_doc
(comments), {"cats": category}))
356 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

4. Let’s check movie_comment_exp:

In[42] movie_comment_exp[0]
Out[42]

5. Use POS and NEG labels for positive and negative sentiment respectively.
Introduce these labels to the new component and initialize it with examples.

In[43] # Add the two sentiment categories into tCategorizer

tCategorizer.add_label("POS")
tCategorizer.add_label("NEG")
tCategorizer.initialize(lambda: movie_comment_exp, nlp=nlp)

In[44] tCategorizer
Out[44] <spacy.pipeline.textcat_multilabel.MultiLabel_TextCategorizer at 0x2626f-
9c4ee0>

6. Define training loop by examining the training set for two epochs but can exam-
ine further if necessary. The following code snippet will train the new text cate-
gorizer component:

In[45] # Set the training epochs to 2 to save time

epochs = 2

# Main program loop

with nlp.select_pipes(enable="textcat_multilabel"):
optimizer = nlp.resume_training()
for i in range(epochs):
random.shuffle(movie_comment_exp)
for exp in movie_comment_exp:
nlp.update([exp], sgd=optimizer)
14.7 Artificial Neural Network in a Nutshell 357

7. Test how text classifier component works for two example sentences:

In[46] test5 = nlp("This is the best movie that I have ever watched")

In[47] test5.cats
Out[47] {'POS': 0.9857660531997681, 'NEG': 0.018266398459672928}

In[48] test6 = nlp("This movie is so bad")

In[49] test6.cats
Out[49] {'POS': 0.1364014744758606, 'NEG': 0.8908849358558655}

Note both NEG and POS labels appeared in prediction results because it used
multi-label classifier. The results are satisfactory, but it can improve if the numbers
for training epochs are increased. The first sentence has a high positive probability
output, and the second sentence has predicted as negative with a high probability

SpaCy’s text classifier component training is completed.

The next section will explore Kera, a popular deep leaning library and how to
write Keras code for text classification with another machine learning library—
TensorFlow‘s Keras API.

14.7 Artificial Neural Network in a Nutshell

This workshop section will learn how to incorporate spaCy technology with ANN
(Artificial Neural Networks) technology using TensorFlow and its Keras package
(Géron 2019; Kedia and Rasu 2020; TensorFlow 2022).
A typical ANN has:
1. Input layer consists of input neurons, or nodes.
2. Hidden layer consists of hidden neurons, or nodes.
3. Output layer consists of output neurons, or nodes
ANN will learn knowledge by its network weights update through network train-
ing with sufficient sample inputs and target outputs pairs. The network can predict
or match unseen inputs to corresponding output result after it had sufficient training
to a predefined accuracy. A typical ANN architecture is shown in Fig. 14.4.
358 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

Fig. 14.4 System

architecture of ANN

14.8 An Overview of TensorFlow and Keras

TensorFlow (Géron 2019; TensorFlow 2022) is a popular Python tool widely used
for machine learning. It has huge community support and great documentation
available at TensorFlow official site (TensorFlow 2022), while Keras is a Python
based deep learning tool that can be integrated with Python platforms such as
TensorFlow, Theano, and CNTK.
TensorFlow 1 was disagreeable to symbolic graph computations and other low-
level computations, but TensorFlow 2 initiated great changes in machine learning
methods allowing developers to use Keras with TensorFlow’s low-level methods.
Keras is popular in R&D because it supports rapid prototyping and user-friendly
API to neural network architectures (Kedia and Rasu 2020; Srinivasa-Desikan 2018).
Neural networks are commonly used for computer vision and NLP tasks includ-
ing object detection, image classification, scene understanding, text classification,
POS tagging, text summarization, and natural language generation.
TensorFlow 2 will be used to study the details of a neural network architecture
for text classification with tf.keras implementation throughout this section.

14.9 Sequential Modeling with LSTM Technology

Long-Short Term Memory network (LSTM network) is one of the significant recur-
rent networks used in various machine learning applications such as NLP applica-
tions nowadays (Ekman 2021; Korstanje 2021).
RNNs are special neural networks that can process sequential data in steps.
All inputs and outputs are independent but not for text data in neural networks.
Every word’s presence depends on neighboring words, e.g. a word is predicted by
considering all preceding predicted words and stored the past sequence token of
words within a LTSM cell in a machine translation task. A LSTM is showed in
Fig. 14.5.
14.10 Keras Tokenizer in NLP 359

Fig. 14.5 RNN with LSTM technology

Fig. 14.6 Architecture of LSTM cell

An LSTM cell is moderately complex than an RNN cell, but computation logic
is identical. A diagram of a LSTM cell is shown in Fig. 14.6. Note that input and
output steps are identical to RNN counterparts:
Keras has extensive support for RNN variations GRU, LSTM, and simple API
for training RNNs. RNN variations are crucial for NLP tasks as language data’s
nature is sequential, i.e. text is a sequence of words, speech is a sequence of sounds,
and so on.
Since the type of statistical model has identified in the design, it can transform a
sequence of words into a word IDs sequence and build vocabulary with Keras pre-
processing module simultaneously.

14.10 Keras Tokenizer in NLP

Text is a sequence of words or characters data. A sentence can be fed by a tokens

sequence. Hence, tokens are to be vectorized first by the following steps:
1. Tokenize each utterance and turn these utterances into a sequence of tokens.
2. Build a vocabulary from set of tokens presented in step 1. These are tokens to be
recognized by neural network design.
3. Create a vocabulary and assign ID to each token.
360 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

4. Map token vectors with corresponding token IDs.

Let us look at a short example of a corpus for three sentences:

In[50] testD = [ "I am going to buy a gift for Christmas tomorrow morning.",
"Yesterday my mom cooked a wonderful meal.",
"Jack promised he would remember to turn off the lights." ]

In[51] testD
Out[51] ['I am going to buy a gift for Christmas tomorrow morning.',
'Yesterday my mom cooked a wonderful meal.',
'Jack promised he would remember to turn off the lights.']

Let’s tokenize words into utterances:

In[52] import spacy

# Load the NLP model

nlp = spacy.load("en_core_web_md")

# Create the utterances object

utterances = [[token.text for token in nlp(utterance)] for utterance in testD]
for utterance in utterances:
utterance

All tokens of Doc object generated by calling nlp(sentence) are iterated in the
preceding code. Note that punctuation marks have not filtered as this filtering
depends on the task e.g. punctuation marks such as ‘!’, correlate to the result in
sentiment analysis, they are preserved in this example.

Build vocabulary and token sequences into token-ID sequences using Tokenizer
as shown:

In[53] # Import Tokenizer

from tensorflow.keras.preprocessing.text import Tokenizer

# Create tokenizer object (ktoken)

ktoken = Tokenizer(lower=True)
14.10 Keras Tokenizer in NLP 361

In[54] ktoken.fit_on_texts(testD)
ktoken

Out[54] <keras_preprocessing.text.Tokenizer at 0x2322f3ae970>

In[55] ktoken.word_index
Out[55]

The following are performed in the above codes:

1. Import Tokenizer from Keras text preprocessing module
2. Create a tokenizer object (ktoken) with parameter lower=True, which means
tokenizer should lower all words for vocabulary formation
3. Call ktoken.fit_on_texts on data to form vocabulary. fit_on_text work on a tokens
sequence; input should always be a list of tokens
4. Examine vocabulary by printing ktoken.word_index. Word_index is a dictionary
where keys are vocabulary tokens and values are token-IDs

Call ktoken.texts_to_sequences() method to retrieve a token ID.

Notice that the input to this method should always be a list, even if single
token is fed.
Feed one-word input as a list (notice list brackets) in the following code segment:

In[56] ktoken.texts_to_sequences(["Christmas"])
Out[56] [[9]]

In[57] ktoken.texts_to_sequences(["cooked", "meal"])

Out[57] [[15], [18]]
362 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

1. Note token-IDs start from 1 and not 0. 0 is a reserved value, which means a
padding value with specific meaning
2. Keras cannot process utterances of different lengths, hence need to pad all
utterances
3. Pad each sentence of dataset to a maximum length by adding padding utterances
either at the start or end of utterances
4. Keras inserts 0 for the padding which means it is a padding value without a token

Let’s understand how padding works with a simple example.

In[58] # Import the pad_sequences package

from tensorflow.keras.preprocessing.sequence import pad_sequences

# Create the utterance sequences

seq_utterance = [[7], [8,1], [9,11,12,14]]

# Define Maximum Length (MLEN)

MLEN=4

# Pad the utterance sequences.

pad_sequences(seq_utterance, MLEN, padding="post")
Out[58]

In[59] pad_sequences(seq_utterance, MLEN, padding="pre")

Out[59]

Call pad_sequences on this sequences list and every sequence is padded with
zeros so that its length reaches MAX_LEN=4 which is the length of the longest
sequence. Then pad sequences from the right or left with post and pre options.
Sentences with post option were padded in the preceding code, hence the sentences
were padded from the right.
14.10 Keras Tokenizer in NLP 363

When these sequences are organized, the complete text preprocessing steps are
as follows:

In[60] # Import the Tokenizer and pad sequences package

from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences

# Create the token object

ktoken = Tokenizer(lower=True)
ktoken.fit_on_texts(testD)

# Create the sequence utterance object

sutterance = ktoken.texts_to_sequences(testD)
MLEN=7

# Pad the utterance sequences

pseq_utterance = pad_sequences(sutterance, MLEN, padding="post")
pseq_utterance

Out[60]

Transform utterances into a token-IDs sequence for tokens vectorization so that

utterances will be ready to feed into neural network.

14.10.1 Embedding Words

Tokens can be transformed into token vectors. Embedding tokens into vectors
occurred via a lookup embedding table. Each row holds a token vector indexed by
token-IDs, hence the flow of obtaining a token vector is as follows:
1. token->token-ID: A token-ID is assigned with each token with Keras’ Tokenizer
in previous section. Tokenizer holds all vocabularies and maps each vocabulary
token to an ID.
2. token-ID->token vector: A token-ID is an integer that can be used as an index to
embed table's rows. Each token-ID corresponds to one row and when a token
vector is required, first obtain its token-ID and lookup in the embedding table
rows with this token-ID.
364 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

Embedding
Token Token-ID
Table
food 11 row 0
row 1 Token-vector
row 2 “food”

row 11

Fig. 14.7 A sample of embedding words into token vectors

A sample of embedding words into token vectors is shown in Fig. 14.7.

Remember when a list of utterances began in the previous section:
1. Each utterance is divided into tokens and built a vocabulary with Keras’
Tokenizer.
2. The Tokenizer object held a token index with a token->token-ID mapping.
3. When a token-ID is obtained, lookup to embedding table rows with this token-
ID to acquire a token vector.
4. This token vector is fed to neural network.
There are several steps to transform sentences into vectors as training a neural
network is complex.
A LSTM neural network architecture can be designed to perform model training
after these preliminary steps.

14.11 Movie Sentiment Analysis with LTSM Using Keras

and spaCy

This section will demonstrate the design of LSTM-based RNN text classifier for
sentiment analysis with steps below:
1. Data retrieve and preprocessing.
2. Tokenize review utterances with padding.
3. Create utterances pad sequence and put into Input Layer.
4. Vectorize each token and verified by token-ID in Embedding Layer.
5. Input token vectors into LSTM.
6. Train LSTM network.
Let us start by recalling the dataset again.
14.11 Movie Sentiment Analysis with LTSM Using Keras and spaCy 365

14.11.1 Step 1: Dataset

IMDB movie reviews identical dataset from sentiment analysis with spaCy section
will be used. They had already processed with pandas and condensed into two col-
umns with binary labels.
Reload reviews table and perform data preprocessing as done in previous section
to ensure the data is up to date:

In[61] import pandas as pd

import matplotlib.pyplot as plt
%matplotlib inline

# Create the movie comment DataFrame and display the statistics

mcommentDF=pd.read_csv('imdb_5000.csv')
mcommentDF = mcommentDF[['Review','Sentiment']].dropna()
axplot=mcommentDF.Sentiment.value_counts().
plot(kind='bar', colormap='Paired')
plt.show()

Out[61]
2500

2000

1500

1000

500

0
0
1

Here is how mcommentDF dataset should look:

In[62] mcommentDF.head()
Out[62]
366 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

Next, extract review text and review label from each dataset row and add them
into Python lists:

In[63] # Import spaCy

import spacy

# Load the spaCy NLP model

nlp = spacy.load("en_core_web_md")

In[64] # Create movie comment sample and categories objects

movie_comment_exp = []
categories = []

# Perform Tokenization
for idx, rw in mcommentDF.iterrows():
comments = rw["Review"]
rating = rw["Sentiment"]
categories.append(rating)
mtoks = [token.text for token in nlp(comments)]
movie_comment_exp.append(mtoks)

In[65] movie_comment_exp[0]
Out[65] ['*', '*', 'Possible', 'Spoilers', '*', '*']

Note that a list of words to movie_comment_exp has added, hence each element of
this list is a list of tokens. Next, invoke Keras' Tokenizer on this tokens list to build
vocabulary

14.11.2 Step 2: Data and Vocabulary Preparation

Since the dataset had already processed, tokenize dataset sentences and build a
vocabulary.

1. Import necessary Python packages.

In[66] # Import Tokenizer, pad_sequences

from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
import numpy as np
14.11 Movie Sentiment Analysis with LTSM Using Keras and spaCy 367

2. Feed ktoken into token list and convert them into IDs by calling
texts_to_sequences:

In[67] # Create ktoken and perform tokenization

ktoken = Tokenizer(lower=True)
ktoken.fit_on_texts(movie_comment_exp)

# Create utterance sequences object

seq_utterance = ktoken.texts_to_sequences(movie_comment_exp)

3. Pad short utterance sequences to a maximum length of 50. This will truncate long
reviews to 50 words:

In[68] # Set the max length to 50

MLEN = 50

# Create pad utterance sequence object

ps_utterance = pad_sequences(seq_utterance, MLEN, padding="post")

4. Convert this list of reviews and labels to numpy arrays:

In[69] # Convert the ps_utterance into numpy arrays

ps_utterance = np.array(ps_utterance)

# Create the category list (catlist)

catlist = np.array(categories)

In[70] catlist = catlist.reshape(catlist.shape[0] , 1)

In[71] catlist.shape
Out[71] (5000, 1)

All basic preparation works are completed at present to create a LSTM network
and input data.
Load TensorFlow Keras related modules:

In[72] # Import the LSTM model and the optimizers

from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, LSTM, Dense, Embedding
from tensorflow.keras import optimizers
368 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

14.11.3 Step 3: Implement the Input Layer

In[73] utterance_input = Input(shape=(None,))

Do not confuse None as input shape. Here, None means that dimension can be any
scalar number. So, use this expression when Keras infers the input shape

14.11.4 Step 4: Implement the Embedding Layer

Create an Embedding Layer as follows:

In[74] # Create the Embedding_Layer

embedding = Embedding(input_dim = len(ktoken.word_index)+1,
output_dim = 100)(utterance_input)

1. When defining embedding layer, input dimension should always be tokens

number in the vocabulary (+1 because the indices start from 1 and not 0. Index 0
is reserved for padding value)
2. Here, 100 is selected as the output shape, hence token vectors for vocabulary
tokens will be 100-dimensional. Popular numbers for token vector dimensions
are 50, 100, and 200 depending on task complexity

14.11.5 Step 5: Implement the LSTM Layer

Create LSTM_Layer:

In[75] # Create the LSTM_Layer

LSTM_layer = LSTM(units=256)(embedding)

Here, units = 256 is the dimension of hidden state. LSTM output shape and hidden
state shape are identical due to LSTM architecture.
14.11 Movie Sentiment Analysis with LTSM Using Keras and spaCy 369

14.11.6 Step 6: Implement the Output Layer

When a 256-dimensional vector from LSTM layer has obtained, it will be con-
densed into a 1-dimensional vector (possible values of this vector are 0 and 1, which
are class labels):

In[76] # Create the Output Layer

outlayer = Dense(1, activation='sigmoid')(LSTM_layer)

A sigmoid function is an S-shaped function used as an activation function to map its

input to a [0-1] range in output layer. It is commonly used in many neural networks

14.11.7 Step 7: System Compilation

After the model has defined, it required to compile with an optimizer, a loss func-
tion, and an evaluation metric:

In[77] # Create the IMDB User Review LSTM Model (imdb_mdl)

imdb_mdl = Model(inputs=[utterance_input],outputs=[outlayer])

Let's look at an imdb_mdl model setup:

In[78] imdb_mdl.summary()
Out[78]
370 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

Next, invoke model compilation:

In[79] imdb_mdl.compile(optimizer="adam", loss="binary_crossentropy",

metrics=["accuracy"])

1. Use ADAM (adaptive moment estimation) as optimizer for LSTM training for
imdb_mdl LSTM model
2. Use binary cross-entropy as loss function
3. A list of supported performance metrics can be found in Keras official site (Keras
2022)

14.11.8 Step 8: Model Fitting and Experiment Evaluation

Feed imdb_mdl model to data with 5 epochs to reduce time:

In[80] # Model fitting by using 5 epochs

imdb_mdl.fit(x=ps_utterance,
y=catlist,
batch_size=64,
epochs=5,
validation_split=0.3)

Out[80] Epoch 1/5

55/55 [=====] - 30s 528ms/step - loss: 0.6062 - accuracy: 0.6591 - val_
loss: 0.5078 - val_accuracy: 0.7427
Epoch 2/5
55/55 [=====] - 28s 501ms/step - loss: 0.2577 - accuracy: 0.9054 - val_
loss: 0.4609 - val_accuracy: 0.7893
Epoch 3/5
55/55 [=====] - 26s 465ms/step - loss: 0.0940 - accuracy: 0.9737 - val_
loss: 0.8353 - val_accuracy: 0.7580
Epoch 4/5
55/55 [=====] - 24s 432ms/step - loss: 0.0357 - accuracy: 0.9911 - val_
loss: 0.9209 - val_accuracy: 0.7633
Epoch 5/5
55/55 [=====] - 24s 438ms/step - loss: 0.0314 - accuracy: 0.9917 - val_
loss: 0.5480 - val_accuracy: 0.7793

<keras.callbacks.History at 0x26289a53f10>
References 371

1. x is a list of ps_utterance for network training and y is the list of categories

(catlist). The epochs parameter is set to 5 to process 5 passes over the data
2. Data has processed 5 times in parameter batch_size = 64 means a batch of 64
training utterances are fed into the memory at a time due to memory
limitations
3. The validation_split = 0.3 means 70% of dataset are used for training and 30%
are used for system validation
4. An experiment validation accuracy rate 0.7793 is acceptable for a basic LSTM
network training for 5 epochs only

Workshop 5.2 Further Exploration of LSTM model on Movie Sentiment

Analysis
1. Follow Workshop 14.1 logic and use rating (0–10) field of IMDB movie
reviews dataset to reconstruct a LSTM for sentiment analysis into 3
categories: Positive, Neutral and Negative
2. Verify training performance
3. Experiment with the code further by placing dropout layers at different
locations such as after embedding layer or, after LSTM layer
4. Try different values for embedding dimensions such as 50, 150, and 200 to
observe change in accuracy
5. Experiment with different values instead of 256 at LSTM layer's hidden
dimension. Try different parameters for each to perform simulations and see
whether the best configuration can be found

References

Agarwal, B. (2020) Deep Learning-Based Approaches for Sentiment Analysis (Algorithms for
Intelligent Systems). Springer.
Albrecht, J., Ramachandran, S. and Winkler, C. (2020) Blueprints for Text Analytics Using
Python: Machine Learning-Based Solutions for Common Real World (NLP) Applications.
O’Reilly Media.
Altinok, D. (2021) Mastering spaCy: An end-to-end practical guide to implementing NLP applica-
tions using the Python ecosystem. Packt Publishing.
Arumugam, R., & Shanmugamani, R. (2018). Hands-on natural language processing with python.
Packt Publishing.
Bird, S., Klein, E., and Loper, E. (2009). Natural language processing with python. O'Reilly.
Ekman, M. (2021) Learning Deep Learning: Theory and Practice of Neural Networks, Computer
Vision, Natural Language Processing, and Transformers Using TensorFlow. Addison-Wesley
Professional.
George, A. (2022) Python Text Mining: Perform Text Processing, Word Embedding, Text
Classification and Machine Translation. BPB Publications.
Géron, A. (2019) Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow:
Concepts, Tools, and Techniques to Build Intelligent Systems. O’Reilly Media.
Kaggle (2022) IMDB Large Movie Review Dataset from Kaggle. https://www.kaggle.com/code/
nisargchodavadiya/movie-review-analytics-sentiment-ratings/data. Accessed 23 June 2022.
372 14 Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy…

Kedia, A. and Rasu, M. (2020) Hands-On Python Natural Language Processing: Explore tools and
techniques to analyze and process text with a view to building real-world NLP applications.
Packt Publishing.
Keras (2022) Keras official site performance metrics. https://keras.io/api/metrics. Accessed 23
June 2022.
Korstanje, J. (2021) Advanced Forecasting with Python: With State-of-the-Art-Models Including
LSTMs, Facebook’s Prophet, and Amazon’s DeepAR. Apress.
Perkins, J. (2014). Python 3 text processing with NLTK 3 cookbook. Packt Publishing Ltd.
Pozzi, F., Fersini, E., Messina, E. and Liu, B. (2016) Sentiment Analysis in Social Networks.
Morgan Kaufmann.
SpaCy (2022) spaCy official site. https://spacy.io/. Accessed 16 June 2022.
Sarkar, D. (2019) Text Analytics with Python: A Practitioner’s Guide to Natural Language
Processing. Apress.
Siahaan, V. and Sianipar, R. H. (2022) Text Processing and Sentiment Analysis using Machine
Learning and Deep Learning with Python GUI. Balige Publishing.
Srinivasa-Desikan, B. (2018). Natural language processing and computational linguistics: A prac-
tical guide to text analysis with python, gensim, SpaCy, and keras. Packt Publishing Limited.
TensorFlow (2022) TensorFlow official site. https://tensorflow.org/. Accessed 22 June 2022.
Vasiliev, Y. (2020) Natural Language Processing with Python and spaCy: A Practical Introduction.
No Starch Press.
Chapter 15
Workshop#6 Transformers with spaCy
and TensorFlow (Hour 11–12)

15.1 Introduction

In Chap. 8, the basic concept about Transfer Learning, its motivation and related
background knowledge such as Recurrent Neural Networks (RNN) with Transformer
Technology and BERT model are introduced.
This workshop will learn about the latest topic Transformers in NLP, how to use
them with TensorFlow and spaCy. First, we will learn about Transformers and
Transfer learning. Second, we will learn about a commonly used Transformer archi-
tecture—Bidirectional Encoder Representations from Transformers (BERT) as well
as to how BERT Tokenizer and WordPiece algorithms work.
Further, we will learn how to start with pre-trained transformer models of
HuggingFace library (HuggingFace 2022) and practice to fine-tune HuggingFace
Transformers with TensorFlow and Keras (TensorFlow 2022; Keras 2022) followed
by how spaCy v3.0 (spaCy 2022) integrates transformer models as pre-trained pipe-
lines. These techniques and tools will be used in the last workshop for building a
Q&A chatbot.
Hence, this workshop will cover the following topics:
• Transformers and Transfer Learning.
• Understanding BERT.
• Transformers and TensorFlow.
• Transformers and spaCy.

15.2 Technical Requirements

Transformers, TensorFlow and spaCy (TensorFlow 2022; spaCy 2022) are to be

installed in own PC/notebook computer.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 373
Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_15
374 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

All source codes for this workshop can be downloaded from GitHub archive on
NLPWorkshop6 (NLPWorkshop6 2022).
Use pip install commands to install the following packages:
• pip install spacy.
• pip install TensorFlow (note: version 2.2 or above).
• pip install transformers.

15.3 Transformers and Transfer Learning in a Nutshell

Transformers in NLP is an innovative idea which aims to solve sequential modelling

tasks and targets problems introduced by Long-Short-Term-Memory (LSTM) archi-
tecture (Ekman 2021; Korstanje 2021).
It is a contemporary machine learning concept and architecture introduced by
Vaswani et al. (2017) in research paper Attention Is All You Need. It explained that
“The Transformer is the first transduction model relying entirely on self-attention to
compute representations of its input and output without using sequence-aligned
RNNs or convolution.”
Transduction in this context means transforming input words to output words by
transforming input words and sentences into vectors. A transformer is trained on a
large corpus such as Wiki or news. These vectors will be used to convey information
regarding word semantics, sentence structures, and sentence semantics for down-
stream tasks.
Word vectors like Glove and FastText are already trained on Wikipedia corpus
that can be used in semantic similarity calculations, hence, Transfer Learning
means to import knowledge from pre-trained word vectors or pre-trained statisti-
cal models.
Transformers offer many pre-trained models to perform NLP tasks such as text
classification, text summarization, question answering, machine translation, and
natural language generation in over 100 languages. It aims to make state-of-the-art
NLP accessible to everyone (Bansal 2021; Rothman 2022; Tunstall et al. 2022;
Yıldırım and Asgari-Chenaghlu 2021).
A list of Transformer models provided by HuggingFace ( 2022) is shown in
Fig. 15.1. Each model is named with a combination of architecture names such
as BERT or DistilBert, possibly a language code, i.e. en, de, multilingual, which
is located at the left side of the figure, and information regarding whether the
model is cased or uncased i.e. distinguish between uppercase and lowercase
characters.
Task names are also listed at the left-hand side. Each model is labeled with a task
name such as text classification or machine translation for Q&A chatbot.
15.4 Why Transformers? 375

Fig. 15.1 Sample Input Texts and their corresponding Output Class Labels

15.4 Why Transformers?

Let us review text classification with spaCy in LSTM architecture.

LSTMs work for modelling text effectively, but there are shortcomings:
• LSTM architecture has difficulties in learning long texts sometimes. Statistical
dependencies in a long text have problems represented by LSTM because it can
fail to recall words processed earlier as time steps progress.
• LSTMs are sequential which means that a single word can process at each time
step but is impossible to parallelizing learning process causing bottleneck.
Transformers address these problems by not using recurrent layers at all, their
architecture is different from LSTM architecture (Bansal 2021; Rothman 2022;
Tunstall et al. 2022; Yıldırım and Asgari-Chenaghlu 2021). A Transformer architec-
ture has an input encoder block at the left, called Encoder, and an output decoder at
the right called Decoder as shown in Fig. 15.2.
The architecture is catered for a machine translation task, input is a sequence of
words from source language, and output is a sequence of words in target language.
376 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

Fig. 15.2 Transformer Output

architecture Probabilities

Softmax

Linear

Add & Norm

Feed
Forward

Add & Norm

Add & Norm
Multi-Head
Feed Attention
Forward Nx

Add & Norm

Nx Add & Norm
Masked
Multi-Head Multi-Head
Attention Attention

Positional Positional
Encoding Encoding
Input Output
Embedding Embedding

Inputs Outputs
(Shifted right)

Encoder generates a vector representation of input words and passes them to decoder
where the word vector transfer is represented by an arrow from encoder block to
decoder block direction. Decoder extracts input word vectors, transforms output
words into word vectors, and generates the probability of each output word.
There are feedforward layers, which are dense layers in encoder and decoder
blocks used for text classification with spaCy. The innovative transformers can
place in a Multi-Head Attention block to create a dense representation for each word
with self-attention mechanism. This mechanism relates each word in input sentence
to other words in the input sentence. Word embedding is calculated by taking a
weighted average of other words’ embeddings, and each word significance can be
calculated in input sentence to enable the architecture focus on each input word
sequentially.
A self-attention mechanism of how input words at the left-hand side attend input
word it at the right-hand side is shown in Fig. 15.3. Dark colors represented rele-
vance, phrase the animal are related to it than other words in the sentence. This
signified transformer can resolve many semantic dependencies in a sentence and
15.5 An Overview of BERT Technology 377

Fig. 15.3 Illustration of

the self-attention
mechanism

used in different tasks such as text classification and machine translation since they
have several architectures depending on tasks. BERT is a popular architecture to
be used.

15.5 An Overview of BERT Technology

15.5.1 What Is BERT?

BERT is introduced in a Google’s original research paper published by Devlin et al.

(2019), the complete Google BERT model can be downloaded from Google’s
GitHub archive (GoogleBert 2022).
It has the following output features (Bansal 2021; Rothman 2022; Tunstall et al.
2022; Yıldırım and Asgari-Chenaghlu 2021):
• Bidirectional: Each input sentence text data training is processed from left to
right and from right to left.
• Encoder: An encoder encodes input sentence.
• Representations: A representation is a word vector.
• Transformers: A transformer-based architecture.
BERT is a trained transformer encoder stack. The input is a sentence, and the
output is a sequence of word vectors. Word vectors are contextual which means that
a word vector is assigned to a word based on input sentence. In short, BERT outputs
contextual word representations as shown in Fig. 15.4.
378 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

Fig. 15.4 Word vector for

the word “bank”

Fig. 15.5 Two distinct word vectors generated by BERT for the same word bank in two different
contexts

It is noted that word bank has different meanings in these two sentences, word
vectors are the same because Glove and FastText are static. Each word has only one
vector and vectors are saved to a file after training. Then, these pre-trained vectors
can be downloaded to our application. BERT word vectors are dynamic on the con-
trary. It can generate different word vectors for the same word depending on input
sentence. Word vectors generated by BERT are shown in Fig. 15.5 against the coun-
terpart shown in Fig. 15.4:

15.5.2 BERT Architecture

BERT is a transformer encoder stack, which means several encoder layers are
stacked on top of each other. The first layer initializes word vectors randomly, and
then each encoder layer transforms output of the previous encoder layer. Figure 15.6
illustrates two BERT model sizes: BERT Base and BERT Large.
BERT Base and BERT Large have 12 and 24 encoder layers to generate word
vectors sizes of 768 and 1024 comparatively.
BERT outputs word vectors for each input word. A high-level overview of BERT
inputs and outputs is illustrated in Fig. 15.7. It showed that BERT input should be
in a special format to include special tokens such as CLS.

15.5.3 BERT Input Format

After learning BERT basic architecture, let us look at how to generate output vec-
tors using BERT.
BERT input format can represent a single sentence and a pair of sentences in a
single sequence of tokens (for tasks such as question answering and semantic simi-
larity, we input two sentences to the model).
15.5 An Overview of BERT Technology 379

Fig. 15.6 BERT Base and

Large architectures (having
12 and 24 encoder layers,
respectively)

Fig. 15.7 BERT model input word and output word vectors
380 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

BERT works with a special tokens class and a special tokenization algorithm
called WordPiece.
There are several types of special tokens [CLS], [SEP], and [PAD]:
• [CLS] is the first special token type for every input sequence. This token is a
quantity of input sentence for classification tasks but disregard non-
classification tasks.
• [SEP] is a sentence separator. If the input is a single sentence, this token will be
placed at the end of sentence, i.e. [CLS] sentence [SEP], or to separate two sen-
tences, i.e. [CLS] sentence1 [SEP] sentence2 [SEP].
• [PAD] is a special token for padding. The padding values can generate sentences
from dataset with equal length. BERT receives sentences with fixed length only,
hence, padding short sentences is required prior feeding to BERT. The tokens
maximum length can feed to BERT is 512.
It was learnt that a sentence can feed to Keras model one word at a time, input
sentences can be tokenized into words using spaCy tokenizer, but BERT works dif-
ferently as it uses WordPiece tokenization. A word piece is literally a piece of a word.
WordPiece algorithm breaks down words into several subwords, its logic behind
is to break down complex/long tokens into tokens, e.g. the word playing is tokenized
as play + ##ing. A ## character is placed before every word piece to indicate that
this token is not a word from language’s vocabulary but is a word piece.
Let us look at some examples:

playing play, ##ing

played play, ##ed
going go, ##ing
vocabulary = [play, go, ##ing, ##ed]

It can concise language vocabulary as WordPiece groups common subwords.

WordPiece tokenization can divide rare/unseen words into their subwords.
After input sentence is tokenized and special tokens are added, each token is
converted to its ID and feed token-ID sequences to BERT.
An input sentence transformed into BERT input format is illustrated in Fig. 15.8.
BERT Tokenizer has several methods to perform above tasks but it has an encod-
ing method that combines these steps into a single step.

15.5.4 How to Train BERT?

BERT originators stated that “We then train a large model (12-layers to 24-layers
Transformer) on a large corpus (Wikipedia + BookCorpus) for a long time (1 M
update steps), and that is BERT” in Google Research’s BERT GitHub repository
(GoogleBert 2022).
15.5 An Overview of BERT Technology 381

Fig. 15.8 Transforming an input sentence into BERT input format

BERT is trained by masked language model (MLM) and next sentence predic-
tion (NSP).
Language modelling is the task of predicting the next token given the sequence
of previous tokens. For example, given the sequence of words Yesterday I visited, a
language model can predict the next token as one of the tokens church, hospital,
school, and so on.
MLM is different. A percentage of tokens are masked randomly to replace a
[MASK] token and presume MLM predicts the masked words.
BERT’s masked language model is implemented as follows:
1. select 15 input tokens randomly.
2. 80% of selected tokens are replaced by [MASK].
3. 10% of selected tokens are replaced by another token from vocabulary.
4. 10% remain unchanged.
A training sentence to LMM example is as follows:
382 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

NSP is the task of predicting the next sentence given by an input sentence. There
are two sentences fed to BERT and presume BERT predicts sentences order if sec-
ond sentence is followed by first sentence.
An input of two sentences separated by a [SEP] token to NSP example is as
follows:

It showed that second sentence can follow first sentence, hence, the predicted
label is IsNext.
Here is another example:

This example showed that the pair of sentences generate a NotNext label without
contextual or semantical relevance.

15.6 Transformers with TensorFlow

Pre-trained transformer models are provided to program developers in open sources

by many organizations including Google (GoogleBert 2022), Facebook (Facebook-
transformer 2022), and HuggingFace (HuggingFace_transformer 2022).
HuggingFace is an AI company focuses on NLP apportioned to open source.
These pre-trained models and agreeable interfaces can integrate transformers
into Python code, as interfaces are compatible with either PyTorch or TensorFlow
or both. HuggingFace‘s pre-trained transformers and their TensorFlow interface to
transformer models will be used in this workshop.

15.6.1 HuggingFace Transformers

This section will explore HuggingFace‘s pre-trained models, TensorFlow interface

and its conventions. HuggingFace offers several models as in Fig. 15.1. Each model
is dedicated to a task such as text classification, question answering, and sequence-
to-sequence modelling.
15.6 Transformers with TensorFlow 383

Fig. 15.9 Documentation of the distilbert-base-uncased-distilled-squad model

A HuggingFace documentation of a distilbert-base-uncased-distilled-squad

model is shown in Fig. 15.9. A Question-Answering task tag is assigned to the
upper-left corner in the documentation followed by supporting deep learning librar-
ies PyTorch, TensorFlow, TFLite, TFSavedModel, training dataset, e.g. squad,
model language, e.g. en for English; the license and base model’s name, e.g.
DistilBERT.
Some models are trained with similar algorithms that belong to identical model
family. For example, the DistilBERT family has many models such as distilbert-
base-uncased and distilbert-multilingual-cased. Each model name includes infor-
mation such as casing to distinguish uppercase/lowercase or model language such
as en, de, or multilingual.
HuggingFace documentation provides information about each model family
with individual model’s API in detail. Lists of available models and BERT model
architecture variations are shown in Fig. 15.10.
BERT model has many tasks variations such as text classification, question
answering, and next sentence prediction.
Each of these models is obtained by placing extra layers atop of BERT output as
these outputs are a sequence of word vectors for each word of input sentences.
For example, a BERTForSequenceClassification model is obtained by placing a
dense layer atop of BERT word vectors.

15.6.2 Using the BERT Tokenizer

BERT uses WordPiece algorithm for tokenization to ensure that each input word is
divided into subwords.
384 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

Fig. 15.10 Lists of the available models (left-hand side) and BERT model variations (right-
hand side)

Let’s look at how to prepare input data with HuggingFace library.

In[1] # Import transformer package

from transformers import BertTokenizer

# Create bert_tokenizer and sample utterance (utt1) and tokens (tok1)

btokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
utt1 = "He lived characteristically idle and romantic."
utt1 = "[CLS] " + utt1 + " [SEP]"
tok1 = btokenizer.tokenize(utt1)

In[2] # Display bert tokens

tok1
Out[2] ['[CLS]', 'he', 'lived', 'characteristic', '##ally', 'idle', 'and', 'romantic',
'.', '[SEP]']

In[3] # Convert bert tokens to ids (id1)

id1 = btokenizer.convert_tokens_to_ids(tok1)
id1
Out[3] [101, 2002, 2973, 8281, 3973, 18373, 1998, 6298, 1012, 102]
15.6 Transformers with TensorFlow 385

1. Import BertTokenizer. Note that different models have different tokenizers, e.g.
XLNet model’s tokenizer is called XLNetTokenizer.
2. Call from_pretrained method on tokenizer object and provide model’s name.
Needless to download pre-trained bert-base-uncased (or model) as this method
downloads model by itself.
3. Call tokenize method. It tokenizes sentence by dividing all words into
subwords.
4. Print tokens to examine subwords. The words he, lived, idle, that exist in
Tokenizer’s vocabulary are to be remained. Characteristically is a rare word
does not exist in Tokenizer’s vocabulary. Thus, tokenizer splits this word into
subwords characteristic and ##ally. Notice that ##ally starts with characters ##
to emphasize that this is a piece of word.
5. Call convert_tokens_to_ids.

Since [CLS] and [SEP] tokens must add to the beginning and end of input sen-
tence, it required to add them manually for the preceding code, but these prepro-
cessing steps can perform in a single step.
BERT provides a method called encode that can:
• add CLS and SEP tokens to input sentence
• tokenize sentence by dividing tokens into subwords
• converts tokens to their token-IDs
Call encode method on input sentence directly as follows:

In[4] from transformers import BertTokenizer

btokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
utt2 = "He lived characteristically idle and romantic."
id2 = btokenizer.encode(utt2)
print(id2)
Out[4] [101, 2002, 2973, 8281, 3973, 18373, 1998, 6298, 1012, 102]

This code segment outputs token-IDs in a single step instead of step-by-step. The
result is a Python list.

Since all input sentences in a dataset must have equal length because BERT can-
not process variable-length sentences, padding the longest sentence from dataset
into short sentences is required using the parameter "padding='longest'".
Writeup conversion codes are also required if a TensorFlow tensor is used instead
of a plain list. HuggingFace library provides encode_plus to combine all these steps
into single method as follows:
386 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

In[5] from transformers import BertTokenizer

btokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
utt3 = "He lived characteristically idle and romantic."

encoded = btokenizer.encode_plus(
text=utt3,
add_special_tokens=True,
padding='longest',
return_tensors="tf"
)

id3 = encoded["input_ids"]

print(id3)
Out[5]

Call encode_plus to input sentence directly. It is padded to a length of 10 includ-

ing special tokens [CLS] and [SEP]. The output is a direct TensorFlow tensor with
token IDs.
Verify parameter list of encode_plus() by:

In[6] btokenizer.encode_plus?

BERT Tokenizer provides several methods on input sentences. Data preparation is

not straightforward, but practice makes perfect. Try out code examples with own
text.

It is ready to process transformed input sentences to BERT model and obtain

BERT word vectors.

15.6.3 Word Vectors in BERT

This section will examine BERT model output as they are a sequence of word vec-
tors assigned by one vector per input word. BERT has a special output format. Let’
us look at the code first.
15.6 Transformers with TensorFlow 387

In[7] from transformers import BertTokenizer, TFBertModel

btokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
bmodel = TFBertModel.from_pretrained("bert-base-uncased")

utt4 = "He was idle."

encoded = btokenizer.encode_plus(
text=utt4,
add_special_tokens=True,
padding='longest',
max_length=10,
return_attention_mask=True,
return_tensors="tf"
)

id4 = encoded["input_ids"]

outputs = bmodel(id4)

• Import TFBertModel
• Initialize out BERT model with a bert-base-uncased pre-trained model
• Transform input sentence to BERT input format with encode_plus, and capture
result tf.tensor in the input variable
• Feed sentence to BERT model and capture output with the output’s variables

BERT model output is a tuple of two elements. Let us print the shapes of out-
put pair:

In[8] print(outputs[0].shape)
Out[8] (1, 6, 768)

In[9] print(outputs[1].shape)
Out[9] (1, 768)
388 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

1. Shape, i.e. batch size, sequence length, hidden size is the first element of
output. A batch size is the numbers of sentences that can feed to model
instantly. When one sentence is fed, the batch size is 1. Sequence length is 10
because sentence is fed max_length = 10 to the tokenizer and padded to length
of 10. Hidden_size is a BERT parameter. BERT architecture has 768 hidden
layers size to produce word vectors with 768 dimensions. Hence, the first
output element contains 768-dimensional vectors per word means it contains 10
words × 768-dimensional vectors
2. The second output is only one vector of 768-dimension. This vector is the word
embedding of [CLS] token. Since [CLS] token is an aggregate of the whole
sentence, this token embedding is regarded as embeddings pooled version of all
words in the sentence. The shape of output tuple is always the batch size,
hidden_size. It is to collect [CLS] token’s embedding per input sentence
basically

When BERT embeddings are extracted, they can be used to train text classifica-
tion model with TensorFlow and tf.keras.

15.7 Revisit Text Classification Using BERT

Some of the codes will be used from previous workshop, but this time the code is
shorter because the embedding and LSTM layers will be replaced by BERT to train
a binary text classifier and tf.keras.
This section will use an email log dataset emails.csv for spam mail classification
found in NLP Workshop6 GitHub repository (NLPWorkshop6 2022).

15.7.1 Data Preparation

Before Text Classification model using BERT is created, let us prepare the data first
just like being learnt in the previous workshop:

15.7.1.1 Import Related Modules

In[10] import pandas as pd

import numpy as np
import tensorflow
from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.models import Model
15.7 Revisit Text Classification Using BERT 389

15.7.1.2 Read emails.csv Datafile

In[11] emails=pd.read_csv("emails.csv",encoding='ISO-8859-1')
emails.head()
Out[11]

15.7.1.3 Use dropna() to Remove Record with Missing Contents

In[12] emails=emails.dropna()
emails=emails.reset_index(drop=True)
emails.columns = ['text','label']
emails.head()
Out[12]

15.7.2 Start the BERT Model Construction

15.7.2.1 Import BERT Models and Tokenizer

In[13] from transformers import BertTokenizer, TFBertModel, BertConfig, TF-

BertForSequenceClassification
bert_tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
bmodel = TFBertModel.from_pretrained("bert-base-uncased")
390 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

1. Import BertTokenizer and BERT model, TFBertModel

2. Initialize both tokenizer and BERT model with a pre-trained bert-base-uncased
model. Note that model’s name starts with TF as names of all HuggingFace
pre-trained models for TensorFlow start with TF. Please pay attention to this
when using other transformer models

15.7.2.2 Process Input Data with BertTokenizer

In[14] emails.head()
Out[14]

15.7.2.3 Double Check Databank to See Whether Data Has

In[15] messages=emails['text']
labels=emails['label']
len(messages),len(labels)
Out[15] (5728, 5728)

15.7.2.4 Use BERT Tokenizer

In[16] input_ids=[]
attention_masks=[]

for msg in messages:

bert_inp=bert_tokenizer.encode_plus(msg,add_special_tokens = True,
max_length =64,pad_to_max_length = True,
return_attention_mask = True)
input_ids.append(bert_inp['input_ids'])
attention_masks.append(bert_inp['attention_mask'])

input_ids=np.asarray(input_ids)
attention_masks=np.array(attention_masks)
labels=np.array(labels)
15.7 Revisit Text Classification Using BERT 391

This code segment will generate token-IDs for each input sentence of the dataset and
append to a list. They are list of class labels consist of 0 and 1 s. convert python lists,
input_ids, label to numpy arrays and feed them to Keras model

15.7.2.5 Define Keras Model Using the Following Lines

In[17] inputs = Input(shape=(64,), dtype="int32")

bert = bmodel(inputs)
bert = bert[1]
outputs = Dense(units=1, activation="sigmoid")(bert)
model = Model(inputs, outputs)

adam = tensorflow.keras.optimizers.Adam (learning_rate=2e-5,

epsilon=1e-08)
model.compile(loss="binary_crossentropy", metrics=["accuracy"],
optimizer=adam)

15.7.2.6 Perform Model Fitting and Use 1 Epoch to Save Time

In[18] model.fit?

In[19] history=model.fit(input_ids,labels,batch_size=1,epochs=1)
Out[19] 5728/5728 [==============================] - 8675s
2s/step - loss: 0.0950 - accuracy: 0.9663

15.7.2.7 Review Model Summary

In[20] bmodel.summary()
Out[20]
392 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

A BERT-based text classifier using less than 10 lines of code is to:

1. Define input layer to inputs sentences to model. The shape is 64 because each
input sentence has 64 tokens in length. Pad each sentence to 64 tokens when
encode_plus method is called
2. Feed input sentences to BERT model
3. Extract second output of BERT output at the third line. Since BERT model’s
output is a tuple, the first element of output tuple is a sequence of word vectors,
and the second element is a single vector that represents the whole sentence
called pooled output vector. bert[1] extracts pooled output vector which is a
vector of shape (1, 768)
4. Squash pooled output vector to a vector of shape 1 by a sigmoid function which
is the class label
5. Define Keras model with inputs and outputs
6. Compile model
7. Fit Keras model

BERT model accepts one line only but can transfer enormous knowledge of Wiki
corpus to model. This model obtains an accuracy of 0.96 at the end of the training.
A single epoch is usually fitted to the model due to BERT overfits a moderate
size corpus.
The rest of the code handles compiling and fitting Keras model as BERT has a
huge memory requirement as can be seen by RAM requirements of Google
Research’s GitHub archive (GoogleBert-Memory 2022).
The training code operates for about an hour in local machine, where bigger
datasets require more time even for one epoch.
This section will learn how to train a Keras model with BERT from scratch.

15.8 Transformer Pipeline Technology

HuggingFace Transformers library provide pipelines to assist program developers

and benefit from transformer code immediately without custom training. A pipeline
is a combination of a tokenizer and a pre-trained model.
HuggingFace provides models for various NLP tasks, its HuggingFace pipe-
lines offer:
• Sentiment analysis (Agarwal 2020; Siahaan and Sianipar 2022)
• Question answering (Rothman 2022; Tunstall et al. 2022)
• Text summarization (Albrecht et al. 2020; Kedia and Rasu 2020)
• Translation (Arumugam and Shanmugamani 2018; Géron 2019)
This section will explore pipelines for sentiment analysis and question answering.
15.8 Transformer Pipeline Technology 393

15.8.1 Transformer Pipeline for Sentiment Analysis

Let us start examples on sentiment analysis:

In[21] from transformers import pipeline

nlp = pipeline("sentiment-analysis")

utt5 = "I hate I am being a worker in the desert."

utt6 = "I like you who are beautiful and kind."

result1 = nlp(utt5)
result2 = nlp(utt6)

The following steps are taken in the preceding code snippet:

1. Import pipeline function from transformers’ library. This function creates
pipeline objects with task name given as a parameter. Hence, a sentiment
analysis pipeline object nlp is created by calling this function on the second
line
2. Define two example sentences with negative and positive sentiments. Then
feed these sentences to the pipeline object nlp.

Check outputs:

In[22] result1
Out[22] [{'label': 'NEGATIVE', 'score': 0.9276903867721558}]

In[23] result2
Out[23] [{'label': 'POSITIVE', 'score': 0.9998767375946045}]

15.8.2 Transformer Pipeline for QA System

Next, we will perform on question answering. Let us see the code:

394 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

In[24] from transformers import pipeline

nlp = pipeline("question-answering")

res = nlp({
'question': 'What is the name of this book ?',
'context': "I'll publish my new book Natural Language Processing
soon."
})

print(res)
Out[24] {'score': 0.9857430458068848, 'start': 25, 'end': 52, 'answer': 'Natural
Language Processing'}

Again, import pipeline function to create a pipeline object nlp. A context which
has identical background information for the model is required for question-
answering tasks to the model
• Request the model about this book’s name after giving information of this
new publication will be available soon
• The answer is natural language processing, as expected
• Try your own examples as simple exercise

HuggingFace transformers studies are completed. Let us move on to final section

to see what spaCy offers on transformers.

Workshop 6.1 revisit sentiment analysis using Transformer technology

1. Use either previous workshop databank or another to import databank
for sentiment analysis
2. Try to implement sentiment analysis using previous and Transformer
technology learnt in this workshop
3. Compare performances and analysis (bonus)

15.9 Transformer and spaCy

SpaCy v3.0 had released new features and components. It has integrated transform-
ers into spaCy NLP pipeline to introduce one more pipeline component called
Transformer. This component allows users to use all HuggingFace models with
spaCy pipelines. A spaCy NLP pipeline without transformers is illustrated in
Fig. 15.11.
With the release of v3.0, v2 style spaCy models are still supported and
transformer-based models introduced. A transformer-based pipeline component
looks like the following as illustrated in Fig. 15.12:
15.9 Transformer and spaCy 395

Fig. 15.11 Vector-based spaCy pipeline components

Fig. 15.12 Transformed-based spaCy pipeline components

Fig. 15.13 spaCy English transformer-based language models

Transformer-based models and v2 style models are listed under Models page of
the documentation (spaCy-model 2022) in English model for each supported lan-
guage. Transformer-based models have various sizes and pipeline components like
v2 style models. Also, each model has corpus and genre information like v2 style
models. An example of an English transformer-based language model from Models
page is shown in Fig. 15.13.
It showed that the first pipeline component is a transformer that generates word
representations and deals with WordPiece algorithm to tokenize words into sub-
words. Word vectors are fed to the rest of the pipeline.
396 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

Downloading, loading, and using transformer-based models are identical to v2

style models.
English has two pre-trained transformer-based models, en_core_web_trf and
en_core_web_lg currently. Let us start by downloading the en_core_web_trf model:

python3 -m spacy download en_core_web_trf

Import spaCy module and transformer-based model:

In[25] import spacy

import torch
import spacy_transformers
nlp = spacy.load("en_core_web_trf")

After loading model and initializing pipeline, use this model the same way as in
v2 style models:

In[26] utt7 = nlp("I visited my friend Betty at her house.")

utt7.ents
Out[26] (Betty,)

In[27] for word in utt7:

print(word.pos_, word.lemma_)
Out[27]

These features related to the transformer component can be accessed by ._.trf_

data which contain word pieces, input ids, and vectors generated by the transformer.
Let’s examine the features one by one:

In[28] utt8 = nlp("It went there unwillingly.")

15.9 Transformer and spaCy 397

In[29] utt8._.trf_data.wordpieces
Out[29]

There are five elements: word pieces, input IDs, attention masks, lengths, and
token type IDs in the preceding output.
Word pieces are subwords generated by WordPiece algorithm. The word pieces
of this sentence are as follows:

The first and last tokens are special tokens used at the beginning and end of the
sentence. The word unwillingly is divided into three subwords—unw, ill, and ingly.
A G character is used to mark word boundaries. Tokens without G are subwords,
such as ill and ingly in the preceding word piece list, except first word in the sen-
tence marked by <'s'>.
Input IDs have identical meanings which are subword IDs assigned by the trans-
former’s tokenizer.
The attention mask is a list of 0 s and 1 s for pointing the transformer to tokens
it should notice. 0 corresponds to PAD tokens, while all other tokens should have a
corresponding 1.
Lengths refer to the length of sentence after dividing into subwords. Here is 9 but
notice that len(doc) outputs is 5, while spaCy always operates on linguistic words.
token_type_ids are used by transformer tokenizers to mark sentence boundaries
of two sentences input tasks such as question and answering. Since there is only one
text provided, this feature is inapplicable.
Token vectors are generated by transformer, doc._.trf_data.tensors which contain
transformer output, a sequence of word vectors per word, and the pooled output
vector. Please refer to Obtaining BERT word vectors section if necessary.
398 15 Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)

In[30] utt8._.trf_data.tensors[0].shape
Out[30] (1, 9, 768)

In[31] utt8._.trf_data.tensors[1].shape
Out[31] (1, 768)

The first element of tuple is the vectors for tokens. Each vector is 768-dimensional,
hence 9 words produce 9 x 768-dimensional vectors. The second element of tuple is
the pooled output vector which is an aggregate representation for input sentence, and
the shape is 1 x 768

spaCy provides user-friendly API and packaging for complicated models such as
transformers. Transformer integration is a validation of using spaCy for NLP.

References

Agarwal, B. (2020) Deep Learning-Based Approaches for Sentiment Analysis (Algorithms for
Intelligent Systems). Springer.
Albrecht, J., Ramachandran, S. and Winkler, C. (2020) Blueprints for Text Analytics Using
Python: Machine Learning-Based Solutions for Common Real World (NLP) Applications.
O’Reilly Media.
Arumugam, R., & Shanmugamani, R. (2018). Hands-on natural language processing with python.
Packt Publishing.
Bansal, A. (2021) Advanced Natural Language Processing with TensorFlow 2: Build effective
real-world NLP applications using NER, RNNs, seq2seq models, Transformers, and more.
Packt Publishing.
Devlin, J., Chang, M. W., Lee, K. and Toutanova, K. (2019). BERT: Pre-training of Deep
Bidirectional Transformers for Language Understanding. Archive: https://arxiv.org/
pdf/1810.04805.pdf.
Ekman, M. (2021) Learning Deep Learning: Theory and Practice of Neural Networks, Computer
Vision, Natural Language Processing, and Transformers Using TensorFlow. Addison-Wesley
Professional.
Facebook-transformer (2022) Facebook Transformer Model archive. https://github.com/pytorch/
fairseq/blob/master/examples/language_model/README.md. Accessed 24 June 2022.
Géron, A. (2019) Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow:
Concepts, Tools, and Techniques to Build Intelligent Systems. O’Reilly Media.
GoogleBert (2022) Google Bert Model Github archive. https://github.com/google-research/bert.
Accessed 24 June 2022. Accessed 24 June 2022.
GoogleBert-Memory (2022) GoogleBert Memory Requirement. https://github.com/
google-research/bert#out-of-memory-issues.
HuggingFace (2022) Hugging Face official site. https://huggingface.co/. Accessed 24 June 2022.
HuggingFace_transformer (2022) HuggingFace Transformer Model archive. https://github.com/
huggingface/transformers. Accessed 24 June 2022.
References 399

Kedia, A. and Rasu, M. (2020) Hands-On Python Natural Language Processing: Explore tools and
techniques to analyze and process text with a view to building real-world NLP applications.
Packt Publishing.
Keras (2022) Keras official site. https://keras.io/. Accessed 24 June 2022.
Korstanje, J. (2021) Advanced Forecasting with Python: With State-of-the-Art-Models Including
LSTMs, Facebook’s Prophet, and Amazon’s DeepAR. Apress.
NLPWorkshop6 (2022) NLP Workshop 6 GitHub archive. https://github.com/raymondshtlee/
NLP/tree/main/NLPWorkshop6. Accessed 24 June 2022.
Rothman, D. (2022) Transformers for Natural Language Processing: Build, train, and fine-tune
deep neural network architectures for NLP with Python, PyTorch, TensorFlow, BERT, and
GPT-3. Packt Publishing.
SpaCy (2022) spaCy official site. https://spacy.io/. Accessed 24 June 2022.
SpaCy-model (2022) spaCy English Pipeline Model. https://spacy.io/models/en. Accessed 24
June 2022.
Siahaan, V. and Sianipar, R. H. (2022) Text Processing and Sentiment Analysis using Machine
Learning and Deep Learning with Python GUI. Balige Publishing.
TensorFlow (2022) TensorFlow official site> https://tensorflow.org /. Accessed 24 June 2022.
Tunstall, L, Werra, L. and Wolf, T. (2022) Natural Language Processing with Transformers:
Building Language Applications with Hugging Face. O’Reilly Media.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin,
I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
https://arxiv.org/abs/1706.03762.
Yıldırım, S, Asgari-Chenaghlu, M. (2021) Mastering Transformers: Build state-of-the-art models
from scratch with advanced natural language processing techniques. Packt Publishing.
Chapter 16
Workshop#7 Building Chatbot
with TensorFlow and Transformer
Technology (Hour 13–14)

16.1 Introduction

In previous 6 NLP workshops, we studied NLP implementation tools and tech-

niques ranging from tokenization, N-gram generation to semantic and sentiment
analysis with various key NLP Python enabling technologies: NLTK, spaCy,
TensorFlow, and contemporary Transformer Technology. This final workshop will
explore how to integrate them for the design and implementation of a live domain-
based chatbot system on a movie domain.
This workshop will explore:
1. Technical requirements for chatbot system.
2. Knowledge domain—the Cornell Large Movie Conversation Dataset is a well-
known conversation dataset with over 200,000 movie dialogs of 10,000+ movie
characters (Cornell 2022; Cornell_Movie_Corpus 2022)
3. A step-by-step Movie Chatbot system implementation which involves movie
dialog preprocessing, model construction, attention learning, system integration
with spaCy, TensorFlow, Keras and Transformer Technology, an important tool
in NLP system implementation (Bansal 2021; Devlin et al. 2019; Géron 2019;
Rothman 2022; Tunstall et al. 2022; Yıldırım and Asgari-Chenaghlu 2021).
4. Evaluation metrics with live chat examples.

16.2 Technical Requirements

Transformers, Tensorflow, and spaCy and Python modules include numpy and
scikit-learn that are to be installed in machine.

© The Author(s), under exclusive license to Springer Nature Singapore Pte 401
Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4_16
402 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

Use pip install commands to:

• pip install spacy
• pip install tensorflow (note: version 2.2 or above)
• pip install transformers
• pip install scikit-learn
• pip install numpy
The data files used in this workshop can be found in DATA sub-directory of
NLPWorkshop7 directory of JupyterHub Server (NLPWorkshop7 2022).

16.3 AI Chatbot in a Nutshell

16.3.1 What Is a Chatbot?

Conversational artificial intelligence (conversational AI) is a field of machine learn-

ing that aims to create technology and enables users to have text or speech-based
interactions with machines. Chatbots, virtual assistants, and voice assistants are
typical conversational AI products (Batish 2018; Freed 2021; Janarthanam 2017;
Raj 2018).
A chatbot is a software application designed to make conversations with humans.
Chatbots are widely used in human resources, marketing and sales, banking,
healthcare, and non-commercial areas such as personal conversations. They include:
–– Amazon Alexa is a voice-based virtual assistant to perform tasks per user requests
or inquiries, i.e. play music, podcasts, set alarms, read audiobooks, provide real-
time weather, traffic, and other information. Alexa Home can connect smart
home devices to oversee premises and electrical appliances.
–– Facebook Messenger and Telegram instant messaging services provide inter-
faces and API documentations (Facebook 2022; Telegram 2022) for developers
to connect bots.
–– Google Assistant provides real-time weather, flight, traffic information, send and
receive text messages, email services, device information, set alarms and inte-
grate with smart home devices, etc. available on Google Maps, Google Search,
and standalone Android and iOS applications.
–– IKEA provides customer service chatbot called Anna, AccuWeather, and FAQ
chatbots.
–– Sephora has virtual make-up artist and customer service chatbots at Facebook
messenger.
–– Siri integrates with iPhone, iPad, iPod, and macOS to initiate, answer calls, send,
receive text messages and WhatsApp messages at iPhone.
Other virtual assistants include AllGenie, Bixby, Celia, Cortana, Duer, and
Xiaowei.
16.3 AI Chatbot in a Nutshell 403

16.3.2 What Is a Wake Word in Chatbot?

A wake word is the gateway between user and user’s digital assistant/Chatbot. Voice
assistants such as Alexa and Siri are powered by AI with word detection abilities to
queries response and commands.
Common wake words include Hey, Google, Alexa, and Hey Siri.
Today’s wake word performance and speech recognition are operated by machine
learning or AI with cloud processing.
Sensory’s wake word and phrase recognition engines use deep neural networks
to provide an embedded or on-device wake word and phrase recognition engine
(Fig. 16.1).

16.3.2.1 Tailor-Made Wake Word

Wake words like Alexa, Siri, and Google are associated with highly valued and
technical products experiences, other companies had created tailor-made wake
word and uniqueness to their products, i.e. Hi Toyota had opened a doorway to
voice user interface to strengthen the relationship between customers and the brand.

16.3.2.2 Why Embedded Word Detection?

Wake word technology has been used in cases beyond mobile applications. Some
battery powered devices like Bluetooth headphones, smart watches, cameras, and
emergency alert devices.

Fig. 16.1 Wake word to invoke Chatbot (Tuchong 2022)

404 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

Chatbot allow users to utter commands naturally. Queries like what time is it? or
how many steps have I taken? are phrases examples that a chatbot can process zero
latency with high accuracy.
Wake word technology can integrate with voice recognition applications like
touch screen food ordering, voice-control microwaves, or user identification set-
tings at televisions or vehicles.

16.3.3 NLP Components in a Chatbot

A typical chatbot consists of major components:

1. Speech-to-text converts user speech into text. The input is a wav/mp3 file and the
output is a text file containing user’s utterance.
2. Conversational NLU performs intent recognition and entity extraction on user’s
utterance text. The output is the user’s intent with a list of entities. Resolving
references in the current to previous utterances is processed by this component.
3. Dialog manager retains conversation memory to generate a meaningful and
coherent chat. This component is regarded as dialog memory in conversational
state hitherto entities and intents appeared. Hence, the input is the previous dia-
log state for current user to parse intent and entities to a new dialog state output.
4. Answer generator gives all inputs from previous stages to generate answers to
user’s utterance.
5. Text-to-speech generates a speech file (WAV or mp3) from system’s answers
Each of these components is trained and evaluated separately, e.g. speech-to-text
training is performed by speech files and corresponding transcriptions on an anno-
tated speech corpus.

16.4 Building Movie Chatbot by Using TensorFlow

and Transformer Technology

This workshop will integrate the learnt technologies including: TensorFlow (Bansal
2021; Ekman 2021; TensorFlow 2022), Keras (Géron 2019; Keras 2022a),
Transformer technology with Attention Learning Scheme (Ekman 2021; Kedia and
Rasu 2020; Rothman 2022; Tunstall et al. 2022; Vaswani et al. 2017; Yıldırım and
Asgari-Chenaghlu 2021) to build a live domain-based chatbot system. The Cornell
Large Movie Dialog Corpus (Cornell 2022) will be used as conversation dataset for
system training. The movie dataset can be downloaded either from Cornell data-
bank (2022) or Kaggle’s Cornell Movie Corpus archive (2022).
16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 405

Use pip install command to invoke TensorFlow package and install its dataset:

In[1] import tensorflow as tflow

tflow.random.set_seed(1234)

# !pip install tensorflow-datasets==1.2.0

import tensorflow_datasets as tflowDS

import re
import matplotlib.pyplot as pyplt

1. Install and import TensorFlow-datasets in addition to TensorFlow package. Please

use pip install command as script if not installed already
2. Use random.set_seed() method to set all random seeds required to replicate
TensorFlow codes

16.4.1 The Chatbot Dataset

The Cornell Movie Dialogs corpus is used in this project. This dataset, movie_con-
versations.txt contains lists of conversation IDs and movie_lines.txt associative con-
versation ID. It has generated 220,579 conversations and 10,292 movie characters
amongst movies.

16.4.2 Movie Dialog Preprocessing

The maximum numbers of conversations (MAX_CONV) and the maximum length

of utterance (MLEN) are set for 50,000 and 40 for system training, respectively.
Preprocessing data procedure (PP) involves the following steps:
1. Obtain 50,000 movie dialog pairs from dataset.
2. PP each utterance by special and control characters removal.
3. Construct tokenizer.
4. Tokenize each utterance.
5. Cap the max utterance length to MLEN.
6. Filter and pad utterances.
406 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

In[2] # Set the maximum number of training conversation

MAX_CONV = 50000

# Preprocess all utterances

def pp_utterance(utterance):
utterance = utterance.lower().strip()
# Add a space to the following special characters
utterance = re.sub(r"([?.!,])", r" \1 ", utterance)
# Delete extrac spaces
utterance = re.sub(r'[" "]+', " ", utterance)
# Other than below characters, the other character replace by spaces
utterance = re.sub(r"[^a-zA-Z?.,!]+", " ", utterance)
utterance = utterance.strip()
return utterance

def get_dialogs():
# Create the dialog object (dlogs)
id2dlogs = {}
# Open the movie_lines text file
with open('data/movie_lines.txt', encoding = 'utf-8', errors = 'ignore') as
f_dlogs:
dlogs = f_dlogs.readlines()
for dlog in dlogs:
sections = dlog.replace('\n', '').split(' +++$+++ ')
id2dlogs[sections[0]] = sections[4]

query, ans = [], []

with open('data/movie_conversations.txt',
encoding = 'utf-8', errors = 'ignore') as f_conv:
convs = f_conv.readlines()
for conv in convs:
sections = conv.replace('\n', '').split(' +++$+++ ')
# Create movie conservation object m_conv as a list
m_conv = [conv[1:-1] for conv in sections[3][1:-1].split(', ')]
for i in range(len(m_conv) - 1):
query.append(pp_utterance(id2dlogs[m_conv[i]]))
ans.append(pp_utterance(id2dlogs[m_conv[i + 1]]))
if len(query) >= MAX_CONV:
return query, ans
return query, ans

queries, responses = get_dialogs()

16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 407

Select query 13 and verify response:

In[3] print('Query 13: {}'.format(queries[13]))

print('Response 13: {}'.format(responses[13]))
Out[3] Query 13: that s because it s such a nice one .
Response 13: forget french .

Select query 100 and verify response:

In[4] print('Query 100: {}'.format(queries[100]))

print('Response 100: {}'.format(responses[100]))
Out[4] Query 100: you set me up .
Response 100: i just wanted

Verify queries (responses) size to see whether it situates within MAX_CONV:

In[5] len(queries)
Out[5] 50000

In[6] Len(responses)
Out[6] 50000

1. After max 50,000 movie conversations had obtained to perform basic

preprocessing, it is sufficient for model training
2. Perform tokenization procedure to add START and END tokens using commands
below

16.4.3 Tokenization of Movie Conversation

In[7] # Define the Movie Token object

m_token =
tflowDS.deprecated.text.SubwordTextEncoder.build_from_corpus
(queries + responses, target_vocab_size = 2**13)

# Define the Start and End tokens

START_TOKEN, END_TOKEN =
[m_token.vocab_size], [m_token.vocab_size + 1]

# Define the size of Vocab (SVCAB)

SVCAB = m_token.vocab_size + 2
408 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

Verify movie token lists for conv 13 and 100:

In[8] print('The movie token of conv 13: {}'.format(m_token.encode

(queries[13])))
Out[8] The movie token of conv 13: [15, 8, 151, 12, 8, 354, 10, 347, 188, 1]

In[9] print('The movie token of conv 100: {}'.format(m_token.encode

(queries[100])))
Out[9] The movie token of conv 100: [5, 539, 36, 119, 1]

16.4.4 Filtering and Padding Process

Cap utterance max length (MLEN) to 40, perform filtering and padding:

In[10] # Set the maximum length of each utterance MLEN to 40

MLEN = 40

# Performs the filtering and padding of each utterance

def filter_pad (qq, aa):
m_token_qq, m_token_aa = [], []

for (utterance1, utterance2) in zip(qq, aa):

utterance1 = START_TOKEN + m_token.encode(utterance1) +
END_TOKEN
utterance2 = START_TOKEN + m_token.encode(utterance2) +
END_TOKEN
if len(utterance1) <= MLEN and len(utterance2) <= MLEN:
m_token_qq.append(utterance1)
m_token_aa.append(utterance2)

# pad tokenized sentences

m_token_qq = tflow.keras.preprocessing.sequence.pad_sequences
(m_token_qq, maxlen=MLEN, padding = 'post')

m_token_aa = tflow.keras.preprocessing.sequence.pad_sequences
(m_token_aa, maxlen=MLEN, padding = 'post')

return m_token_qq, m_token_aa

queries, responses = filter_pad (queries, responses)

16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 409

Review the size of movie vocab (SVCAB) and total number of conversation
(conv):

In[11] print('Size of vocab: {}'.format(SVCAB))

print('Total number of conv: {}'.format(len(queries)))
Out[11] Size of vocab: 8333
Total number of conv: 44095

1. Note that the total number of conversations after filtering and padding process is
44,095 which is less than previous max conv size 50,000 as some conversations
are filtered out
2. SVCAB size is around 8000 which makes sense as the total numbers of
conversation are around 44,000 lines, the number of vocabulary used is between
5000 and 10,000

16.4.5 Creation of TensorFlow Movie Dataset Object (mDS)

TensorFlow dataset object is created by using Dataset.from_tensor_slices() method

of TensorFlow Data class as below:

In[12] tflow.data.Dataset.from_tensor_slices?

In[13] # Define the Batch and Buffer size

sBatch = 64
sBuffer = 20000

# Create mDS object from TensorFlow class

mDS = tflow.data.Dataset.from_tensor_slices(({'inNodes':queries,
'decNodes':responses[:, :-1]},{'outNodes':responses[:, 1:]}))

mDS = mDS.cache()
mDS = mDS.shuffle(sBuffer)
mDS = mDS.batch(sBatch)
mDS = mDS.prefetch(tflow.data.experimental.AUTOTUNE)

1. Create a TensorFlow Dataset object first to define Batch and Buffer size
2. Define three layers of Transformer Model: a. Input node layer (inNodes) –
Queries b. Decoder input node layer (decNodes) – Responses c. Output node
layer (outNodes) - Responses
3. Define prefetch scheme—AUTOTUNE in our project
410 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

16.4.6 Calculate Attention Learning Weights

The main concept of Transformer Technology is Attention Learning technique,

which aimed at network capability to focus attention to various parts of training
sequence during recurrent network learning. AI chatbot corresponds to self-attention
learning on movie dialogs, in which the network has attention ability to different
positions of dialog token sequences to compute utterances representation. A system
architecture of Attention Learning model with Transformer Technology is shown in
Fig. 16.2. Implement Attention Equation to calculate the attention weight is given by

QK T
Attention Q,K ,V soft max k (16.1)
d
k

Attention Equation is a typical scaled-dot-product attention function in trans-

former object Query (Q), K (Key), and V (Value) Value and Python implementation
is given below:

In[14] # Calculate the Attention Weight, Query (q), Key(k), Value(v), Mask(m)
def calc_attention(q, k, v, m):
qk = tflow.matmul(q, k, transpose_b = True)
dep = tflow.cast(tflow.shape(k)[-1], tflow.float32)
mlogs = qk / tflow.math.sqrt(dep)

# Use the masking for padding

if m is not None:
mlogs += (m * -1e9)

# Apply softmax on the final axis of the utterance sequence

att_wts = tflow.nn.softmax(mlogs, axis = -1)

# Apply matmul() operation

out_wts = tflow.matmul(att_wts, v)

return out_wts

Fig. 16.2 Attention Learning with Transformer Technology

16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 411

16.4.7 Multi-Head-Attention (MHAttention)

Multi-Head-Attention (MHAttention) consists of the following steps:

1. Construct linear layers.
2. Perform head-splitting.
3. Calculate attention weights.
4. Combine heads.
5. Condense layers
MHAttention is implemented as follows:

In[15] class MHAttention(tflow.keras.layers.Layer):

def init(self, dm, nhd, name="MHAttention"):

super(MHAttention, self).__init__(name=name)
self.nhd = nhd
self.dm = dm

assert dm % self.nhd == 0

self.dep = dm // self.nhd

self.qdes = tflow.keras.layers.Dense(units=dm)
self.kdes = tflow.keras.layers.Dense(units=dm)
self.vdes = tflow.keras.layers.Dense(units=dm)

self.des = tflow.keras.layers.Dense(units=dm)

def sheads(self, inNodes, bsize):

inNodes = tflow.reshape(
inNodes, shape=(bsize, -1, self.nhd, self.dep))
return tflow.transpose(inNodes, perm=[0, 2, 1, 3])

def call(self, inNodes):

q, k, v, m = inNodes['q'], inNodes['k'], inNodes['v'], inNodes['m']
bsize = tflow.shape(q)[0]

# 1. Construct Linear-layers
q = self.qdes(q)
k = self.kdes(k)
v = self.vdes(v)
412 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

# 2. Perform Head-splitting
q = self.sheads(q, bsize)
k = self.sheads(k, bsize)
v = self.sheads(v, bsize)

# 3. Calculate Attention Weights

sattention = calc_attention(q, k, v, m)

sattention = tflow.transpose(sattention, perm=[0, 2, 1, 3])

# 4. Head Combining
cattention = tflow.reshape(sattention,
(bsize, -1, self.dm))

# 5. Layer Condensation
outNodes = self.des(cattention)

return outNodes

16.4.8 System Implementation

16.4.8.1 Step 1. Implement Masking

Implement (1) Padding Mask and (2) Look_ahead Mask to mask token sequences.

In[16] # Generate Padding Mask (gen_pmask)

def gen_pmask(p):

pmask = tflow.cast(tflow.math.equal(p, 0), tflow.float32)

return pmask[:, tflow.newaxis, tflow.newaxis, :]

In[17] # Generate Look_Ahead Mask (gen_lamask)

def gen_lamask(x):
slen = tflow.shape(x)[1]
lamask = 1- tflow.linalg.band_part(tflow.ones((slen, slen)), -1, 0)
pmask = gen_pmask(x)

return tflow.maximum(lamask, pmask)

16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 413

Review lamask with a sample matrix:

In[18] print(gen_lamask(tflow.constant([[1, 2, 0, 4, 5]])))

Out[18] tf.Tensor(
[[[[0. 1. 1. 1. 1.]
[0. 0. 1. 1. 1.]
[0. 0. 1. 1. 1.]
[0. 0. 1. 0. 1.]
[0. 0. 1. 0. 0.]]]], shape=(1, 1, 5, 5), dtype=float32)

16.4.8.2 Step 2. Implement Positional Encoding

The main function of positional encoding is to provide model with information

about the relative position of word tokens within utterance for attention learning
given by the following formula:

PE pos, 2i sin pos / 100002i / dmodel
(16.2)

PE pos, 2i 1 cos pos / 10000 2i / d model

In[19] # Implementation of Positional Encoding Class (PEncoding)
class PEncoding(tflow.keras.layers.Layer):

def init(self, pos, dm):

super(PEncoding, self).__init__()
self.pencode = self.pencods(pos, dm)

def gdeg(self, pos, i, dm):

deg = 1 / tflow.pow(10000,(2 * (i // 2)) /
tflow.cast(dm, tflow.float32))
return pos * deg

def pencods(self, pos, dm):

deg_rads = self.gdeg(pos = tflow.range(pos, dtype=tflow.float32)
[:, tflow.newaxis], i=tflow.range(dm, dtype=tflow.float32)
[tflow.newaxis, :], dm = dm)
m_sin = tflow.math.sin(deg_rads[:, 1::2])
m_cos = tflow.math.cos(deg_rads[:, 1::2])

pencode = tflow.concat([m_sin, m_cos], axis = -1)

pencode = pencode[tflow.newaxis, ...]
return tflow.cast(pencode, tflow.float32)

def call(self, inNodes):

return inNodes + self.pencode[:, :tflow.shape(inNodes)[1], :]
414 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

Try to plot PositionalEncoding diagram:

In[20] # Create PositionalEncoding Sample

pencoding_sample = PEncoding(50, 512)

pyplt.pcolormesh(pencoding_sample.pencode.numpy()[0], cmap = 'RdBu')

pyplt.xlabel('Depth')
pyplt.xlim((0, 512))
pyplt.ylabel('Position')
pyplt.colorbar()
pyplt.show()

Out[20]
50 1.00

0.75
40
0.50

30 0.25
Position

0.00
20 –0.25

–0.50
10
–0.75

0 –1.00
0 100 200 300 400 500
Depth

16.4.8.3 Step 3. Implement Encoder Layer

Encoder Layer (enclayer) implementation involves:

1. Create MHAttention object.
2. Two dense layers.
16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 415

Details as shown below:

In[21] # Implementation of Encoder Layer (enclayer)

def enclayer(i, dm, nhd, drop, name="enclayer"):
inNodes = tflow.keras.Input(shape=(None, dm), name="inNodes")
pmask = tflow.keras.Input(shape=(1, 1, None), name="pmask")

att = MHAttention(
dm, nhd, name="att")({
'q': inNodes,
'k': inNodes,
'v': inNodes,
'm': pmask
})
att = tflow.keras.layers.Dropout(rate=drop)(att)
att = tflow.keras.layers.LayerNormalization(
epsilon=1e-6)(inNodes + att)

outNodes = tflow.keras.layers.Dense(units=i, activation='relu')(att)

outNodes = tflow.keras.layers.Dense(units=dm)(outNodes)
outNodes = tflow.keras.layers.Dropout(rate=drop)(outNodes)
outNodes = tflow.keras.layers.LayerNormalization(
epsilon=1e-6)(att + outNodes)

return tflow.keras.Model(
inputs=[inNodes, pmask], outputs=outNodes, name=name)

1. An Attention Learning object is defined and used at Encoder Layer

implementation class
2. relu function is used as default setting for Encoder Layer Activation Function.
Current research includes the modification (or change) of activation function for
system enhancement

Try to display a sample Encoder Layer using Keras plot model():

In[22] # Create a sample Encoder Layer and display object diagram

enclayer_sample = enclayer(i = 512, dm = 128, nhd = 4, drop = 0.3, name =
"enclayer_sample")
tflow.keras.utils.plot_model(enclayer_sample, to_file = 'enclayer.png',
show_shapes = True)
416 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

Out[22]

16.4.8.4 Step 4. Implement Encoder

Encoder implementation involves the following processes:

1. Embed inputs.
2. Perform positional encoding scheme.
3. Encode Num Layers
16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 417

In[23] # Implementation of Encoder Class (encoder)

def encoder(svcab,
nlayers,
x,
dm,
nhd,
drop,
name="encoder"):
inNodes = tflow.keras.Input(shape=(None,), name="inNodes")
pmask = tflow.keras.Input(shape=(1, 1, None), name="pmask")

embeddings = tflow.keras.layers.Embedding(svcab, dm)(inNodes)

embeddings *= tflow.math.sqrt(tflow.cast(dm, tflow.float32))
embeddings = PEncoding(svcab, dm)(embeddings)

outNodes = tflow.keras.layers.Dropout(rate=drop)(embeddings)

for i in range(nlayers):
outNodes = enclayer(
i=x,
dm=dm,
nhd=nhd,
drop=drop,
name="enclayer_{}".format(i),
)([outNodes, pmask])

return tflow.keras.Model(
inputs=[inNodes, pmask], outputs=outNodes, name=name)
418 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

Display a sample Encoder using Keras Plot model:

In[24] # Create a sample Encoder Sample and display object diagram

encoder_sample = encoder(svcab = 8192,
nlayers = 2,
x = 512,
dm = 128,
nhd = 4,
drop = 0.3,
name = "encoder_sample")

tflow.keras.utils.plot_model
(encoder_sample, to_file='encoder_sample.png', show_shapes = True)
Out[24]
16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 419

16.4.8.5 Step 5. Implement Decoder Layer

Decoder Layer implementation involves the following steps:

1. MHAttention.
2. 2 Dense Decoder Layers with dropout.

In[25] # Implementation of Decoder Layer (declayer)

def declayer(i, dm, nhd, drop, name = "declayer"):
inNodes = tflow.keras.Input(shape=(None, dm), name ="inNodes")
encouts = tflow.keras.Input(shape=(None, dm), name="encouts")
lamask = tflow.keras.Input(shape=(1, None, None), name = "lamask")
pmask = tflow.keras.Input(shape=(1, 1, None), name = "pmask")

att1 = MHAttention(dm, nhd, name="att1")(inNodes={'q':inNodes,

'k':inNodes,
'v':inNodes,
'm':lamask})

att1 = tflow.keras.layers.LayerNormalization(epsilon=1e-6)
(att1 + inNodes)

att2 = MHAttention(dm,nhd, name = "att2")(inNodes={'q':att1,

'k':encouts,
'v':encouts,
'm':pmask})
att2 = tflow.keras.layers.Dropout(rate=drop)(att2)
att2 = tflow.keras.layers.LayerNormalization(epsilon = 1e-6)(att2 + att1)

outNodes = tflow.keras.layers.Dense(units=i, activation='relu')(att2)

outNodes = tflow.keras.layers.Dense(units=dm)(outNodes)
outNodes = tflow.keras.layers.Dropout(rate=drop)(outNodes)
outNodes = tflow.keras.layers.LayerNormalization(epsilon=1e-6)(out-
Nodes + att2)

return tflow.keras.Model(inputs=[inNodes, encouts, lamask, pmask],

outputs = outNodes,
name = name)
420 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

1. Encoder Layer implements single Attention Learning object, and Decoder Layer
implements two Attention Learning objects att1 and att2 according to Transformer
Learning model
2. Again, relu function is used as Activation Function. It can modify or adopt different
Activation Function to improve network performance as studied in Sect. 16.1

Display sample Decoder Layer using Keras plot_model():

In[26] # Create a decoder layer sample and show object association diagram
declayer_sample = declayer(i = 512, dm = 128, nhd = 4, drop = 0.3,
name = "declayer_sample")

tflow.keras.utils.plot_model
(declayer_sample, to_file='declayer_sample.png', show_shapes=True)

Out[26]
16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 421

16.4.8.6 Step 6. Implement Decoder

Decoder implementation involves the following processes:

1. Embed network outputs.
2. Look ahead and pad masking.
3. Positional encoding scheme.
4. Perform N-decoder layers.

In[27] # Implementation of Decoder class (decoder)

def decoder(svcab,
nlayers,
x,
dm,
nhd,
drop,
name='decoder'):
inNodes = tflow.keras.Input(shape=(None,), name="inNodes")
encouts = tflow.keras.Input(shape=(None, dm), name="encouts")
lamask = tflow.keras.Input(shape=(1, None, None), name="lamask")
pmask = tflow.keras.Input(shape=(1, 1, None), name="pmask")

embeddings = tflow.keras.layers.Embedding(svcab, dm)(inNodes)

embeddings *= tflow.math.sqrt(tflow.cast(dm, tflow.float32))
embeddings = PEncoding(svcab, dm)(embeddings)

outNodes = tflow.keras.layers.Dropout(rate=drop)(embeddings)

for i in range(nlayers):
outNodes = declayer(i = x,
dm=dm,
nhd=nhd,
drop=drop,
name = 'declayer_{}'.format(i),)(inputs=[outNodes, en-
couts, lamask, pmask])

return tflow.keras.Model(inputs=[inNodes, encouts, lamask, pmask],

outputs = outNodes,
name = name)
422 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

Display sample Decoder using Keras Plot_model:

In[28] # Create a decoder sample and show object association diagram

decoder_sample = decoder(svcab=8192,
nlayers=2,
x = 512,
dm = 128,
nhd = 4,
drop = 0.3,
name = "decoder_sample")
tflow.keras.utils.plot_model(decoder_sample,
to_file='decoder_sample.png', show_shapes = True)

Out[28]

16.4.8.7 Step 7. Implement Transformer

Transformer involves implementing Encoder, Decoder, and the final Linear Layer.
Transformer Decoder output is input to Linear Layer as a Recurrent Neural
Network (RNN) and output model is returned.
16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 423

In[29] # Implementation of Transformer Class

def transformer(svcab, nlayers, x, dm, nhd, drop, name="transformer"):
queries = tflow.keras.Input(shape=(None,), name="inNodes")
dec_queries = tflow.keras.Input(shape=(None,), name="decNodes")

enc_pmask = tflow.keras.layers.Lambda(
gen_pmask, output_shape=(1, 1, None),
name="enc_pmask")(queries)

# Perform Look Ahead Masking for Decoder Input for the Att1
lamask = tflow.keras.layers.Lambda(gen_lamask,
output_shape=(1, None, None),
name = "lamask")(dec_queries)

# Perform Padding Masking for Encoder Output for the Att2

dec_pmask = tflow.keras.layers.Lambda(gen_pmask,
output_shape=(1, 1, None),
name="dec_pmask")(queries)

encouts = encoder(svcab=svcab,
nlayers = nlayers,
x = x,
dm = dm,
nhd = nhd,
drop = drop,)(inputs = [queries, enc_pmask])

decouts = decoder(svcab=svcab,
nlayers = nlayers,
x = x,
dm = dm,
nhd = nhd,
drop=drop,)(inputs=[dec_queries, encouts, lamask, dec_
pmask])

responses =
tflow.keras.layers.Dense(units=svcab, name="outNodes")(decouts)

return tflow.keras.Model(inputs=[queries, dec_queries],

outputs=responses, name=name)
424 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

Display sample transformer object using Keras Plot_model:

In[30] # Create a transformer sample and display object diagram

transformer_sample = transformer(svcab=8192, nlayers=4, x=512,
dm=128, nhd = 4, drop=0.3, name="transformer_sample")

tflow.keras.utils.plot_model(transformer_sample,
to_file="transformer_sample.png", show_shapes=True)

Out[30]

16.4.8.8 Step 8. Model Training

Parameters for nLayers, dm, and units (x) had reduced to speed up training process.

In[31] # Create Transformer Model

tflow.keras.backend.clear_session()

model = transformer(svcab = SVCAB,

nlayers=2,
x=512,
dm=256,
nhd=8,
drop=0.1)

1. A Movie Chatbot Transformer Model consists of two layers with 512 units,
data-model size 256, head number 8 and dropout rate 0.1 according to Transformer
Model as in Fig. 16.2
2. It is recommended to modify these parameter settings to improve network
performance as discussed in Sect. 16.1
16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 425

16.4.8.9 Step 9. Implement Model Evaluation Function

A loss function is implemented for system evaluation. It is important to apply a pad-

ding mask when calculating the loss since target sequences are padded.

In[32] # Implementation of Evaluation Function (Loss Function)

def Eval_function(xtrue, xpred):
xtrue = tflow.reshape(xtrue, shape=(-1, MLEN - 1))

loss_val = tflow.keras.losses.SparseCategoricalCrossentropy(
from_logits= True, reduction='none')(xtrue, xpred)

mask_val = tflow.cast(tflow.not_equal(xtrue, 0), tflow.float32)

loss_val = tflow.multiply(loss_val, mask_val)

return tflow.reduce_mean(loss_val)

16.4.8.10 Step 10. Implement Customized Learning Rate

Adam_Optimizer with customized learning rate is used with formula below:

5
I rate d model min step _ num 0.5 ,step _ num warmup _ steps 1.5 (16.3)

In[33] # Implementation of Customized Learning Rate

class CLearning(tflow.keras.optimizers.schedules.LearningRateSchedule):

def init(self, dm, warmup_steps=4000):

super(CLearning, self).__init__()
self.dm = dm
self.dm = tflow.cast(self.dm, tflow.float32)

self.warmup_steps = warmup_steps

def call(self, step):

# arg1 = tflow.math.rsqrt(step)
arg1 = tflow.math.rsqrt(tflow.cast(step, tflow.float32))
arg2 = tflow.cast(step, tflow.float32) * (tflow.cast(self.warmup_steps,
tflow.float32)**-1.5)

return tflow.math.rsqrt(self.dm) * tflow.math.minimum(arg1, arg2)

426 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

Plot customized Learning Rate:

In[34] # Create customized learning rate object and display performance

CLearning_sample = CLearning(dm=128)

pyplt.plot(CLearning_sample(tflow.range(200000, dtype=tflow.float32)))
pyplt.ylabel("Learning Rate")
pyplt.xlabel("Train Step")

Out[34]

0.0014

0.0012

0.0010
Learning Rate

0.0008

0.0006

0.0004

0.0002

0.0000
0 25000 50000 75000 100000 125000 150000 175000 200000
Train step

16.4.8.11 Step 11. Compile Chatbot Model

In[35] # Compile Movie Chatbot Model

# Set the Customized Learning Rate
cLRate = CLearning(256)

# Set Adam Optimizers

optimizer = tflow.keras.optimizers.Adam
(learning_rate=cLRate, beta_1=0.9, beta_2=0.98, epsilon=1e-9)

# Implement Accuracy Evaluation Scheme

def accuracy(xtrue, xpred):
xtrue = tflow.reshape(xtrue, shape=(-1, MLEN - 1))
return tflow.keras.metrics.sparse_categorical_accuracy(xtrue, xpred)

# Compile Chatbot Model

model.compile(optimizer=optimizer, loss=Eval_function,
metrics=[accuracy])
16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 427

16.4.8.12 Step 12. System Training (Model Fitting)

Train Chatbot transformer model by calling model.fit() for 20 epochs to save time.

In[36] EPOCHS = 20

model.fit(mDS, epochs = EPOCHS)

Out[36]
428 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

16.4.8.13 Step 13. System Evaluation and Live Chatting

System evaluation and live chatting implementation involve following steps:

1. Create Mining() method by performing data preprocessing of all utterances.
2. Perform tokenization of utterances and padded with START and END tokens.
3. Perform LookAhead and Padding Masks.
4. Construct Transformer model with Attention Learning.
5. Implement chatting() method by decoder scheme.
6. Combine chatted word sequences to decoder input.
7. Use Transformer Model for system to predict responses based on previous train-
ing epochs.

In[37] # Implementation of Movie Chatting class - mchat

def mchat(utterance):
# Utterance Preprocessing and add START AND END TOKENS
utterance = pp_utterance(utterance)
utterance = tflow.expand_dims(START_TOKEN +
m_token.encode(utterance) + END_TOKEN, axis = 0)

# Create response object

response = tflow.expand_dims(START_TOKEN, 0)

for i in range(MLEN):
chatting = model(inputs = [utterance, response], training = False)

# Choose last_word from token sequence

chatting = chatting[:, -1:, :]
chatted_id = tflow.cast(tflow.argmax(chatting, axis=-1), tflow.int32)

# Return with chattedID with ENDTOKEN

if tflow.equal(chatted_id, END_TOKEN[0]):
break

# Combine CHATTEDID with utterance response

response = tflow.concat([response, chatted_id], axis=-1)

return tflow.squeeze(response, axis = 0)

16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology 429

# Implementation of main class for Movie Chatting - mchatting

def mchatting(utterance):
mchatting = mchat(utterance)

chatted_utterance =
m_token.decode([i for i in mchatting if i < m_token.vocab_size])

print('Query: {}'.format(utterance))
print('Response: {}'.format(chatted_utterance))

return chatted_utterance

Try some movie conversations to see whether it works:

In[38] output = mchatting("Where have you been?")

Out[38] Query: Where have you been?
Response: i m going to get my father .

In[39] output = mchatting("It's a trap")

Out[39] Query: It's a trap
Response: i don t know what to do . it s just that way .

In[40] output = mchatting("Do you need help?")

Out[40] Query: Do you need help?
Response: no .

In[41] output = mchatting("What do you think?")

Out[41] Query: What do you think?
Response: i don t know . i don t know . i m not sure . i just had to see what
i m saying .

In[42] output = mchatting("Are you happy?")

Out[42] Query: Are you happy?
Response: no . but you re not . you re not sure ?
430 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

1. Training showed that epochs 1–20 are rather slow but increased in accuracy and
decreased in loss rate
2. Two chatbots experiments with one used 2 epochs and the other used 20 epochs.
Results showed that performance on 20 epochs has satisfactory performance than
the one with 2 epochs
3. Increase epochs, say up to 50 epochs to review whether accuracy has continuous
improvement. It is natural to require more time unless there are sufficient GPUs

Workshop 7.1 Fine-tune Chatbot Model

TensorFlow and Transformer Technology are used to develop a domain-based
Chatbot system
There are rooms to fine-tune model performance like any AI model. It can be
conducted by:
1. Dataset Level
– Enhance preprocessing process
– Improve data record selection scheme, e.g. sample size, utterance max
length, etc.
2. Network Model Level
– Fine-tune system parameters, e.g. Learning Rate and Method, etc.
– Fine-tune Transformer Model by modifying Attention Function etc.
Compare performances (MUST) and analysis (bonus)
Fine-tune Movie Chatbot model and compare with original version

Workshop 7.2 Mini Project - Build a Semantic-Level AI Chatbot System

Extend Character-level and Word-level NLU to a Semantic-Level NLU
1. Modify codes of AI Chatbot learnt in this section to implement a
Semantic-level AI Chatbot system
2. Compare system performance of this revised system with previous
Character-level and Word-level AI Chatbot system

16.5 Related Works

This workshop had integrated all NLP related implementation techniques including
TensorFlow and Keras with Transformer Technology to design an AI-based NLP
application chatbot system. It is a step-by-step implementation consisting of data
preprocessing, model construction, system training, testing evaluation process; and
Attention Learning and Transformer Technology with TensorFlow and Keras imple-
mentation platform easily applied to other chatbot domain and interactive QA sys-
tems using Cornell Large Movie dataset with over 200,000 movie conversations
with 10,000+ movie characters.
Nevertheless, it is only the dawn of journey. There are regular new R&D preva-
lence and usage in NLP applications. Below are lists of renowned domains and
resources related to chatbot systems for reference.
References 431

Datasets for Chatbot Systems:

• Taskmaster from Google Research (GoogleResearch 2022a).
• Simulated Dialogue dataset from Google Research (GoogleResearch 2022b).
• Dialog Challenge dataset from Microsoft (MicrosoftDialog 2022).
• Dialog State Tracking Challenge dataset (DSTC 2022).

Keras Modules and Optimizer:

• Keras layers (Keras 2022a).
• Keras optimizers (Keras 2022b).
• An overview of optimizers (Ruder 2022).
• Adam optimizer (Adam 2022).

Famous Chatbot System:

• Amazon Alexa developer blog (Alexa 2022).
• Apple Siri Developer (AppleSiri 2022).
• Duer from Baidu (Duer 2022).
• Google Assistant (GoogleAssistant 2022).
• Microsoft Cortana Developer (MicrosoftCortana 2022).
• Samsung Bixby Developer (SamsungBixby 2022).
• Xiaowei from Tencent (Xiaowei 2022).

References

Adam (2022) Adam optimizer: https://arxiv.org/abs/1412.6980. Accessed 29 June 2022.

Alexa (2022) Amazon Alexa developer blog: https://developer.amazon.com/blogs/home/tag/
Alexa. Accessed 29 June 2022.
AppleSiri (2022) Apple Siri Developer: https://developer.apple.com/siri/. Accessed 29 June 2022.
Bansal, A. (2021) Advanced Natural Language Processing with TensorFlow 2: Build effective
real-world NLP applications using NER, RNNs, seq2seq models, Transformers, and more.
Packt Publishing.
Batish, R. (2018) Voicebot and Chatbot Design: Flexible conversational interfaces with Amazon
Alexa, Google Home, and Facebook Messenger. Packt Publishing.
Cornell (2022) https://www.cs.cornell.edu/~cristian/Chameleons_in_imagined_conversations.
html. Accessed 29 June 2022.
Cornell_Movie_Corpus (2022) Cornell Movie Corpus archive. https://www.kaggle.com/datasets/
Cornell-University/movie-dialog-corpus. Accessed 29 June 2022.
Devlin, J., Chang, M. W., Lee, K. and Toutanova, K. (2019). BERT: Pre-training of Deep
Bidirectional Transformers for Language Understanding. Archive: https://arxiv.org/
pdf/1810.04805.pdf.
Duer (2022) Duer Baidu AI Chatbot. http://duer.baidu.com/en/index.html. Accessed 29 June 2022.
DSTC (2022) Dialog State Tracking Challenge dataset: https://github.com/matthen/dstc. Accessed
29 June 2022.
Ekman, M. (2021) Learning Deep Learning: Theory and Practice of Neural Networks, Computer
Vision, Natural Language Processing, and Transformers Using TensorFlow. Addison-Wesley
Professional.
432 16 Workshop#7 Building Chatbot with TensorFlow and Transformer Technology…

Facebook (2022) Facebook Messenger API documentation. https://developers.facebook.com/

docs/messenger-platform/getting-started/quick-start/. Accessed 29 June 2022.
Freed, A. (2021) Conversational AI: Chatbots that work. Manning.
Géron, A. (2019) Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow:
Concepts, Tools, and Techniques to Build Intelligent Systems. O’Reilly Media.
GoogleAssistant (2022) Google Assistant: https://assistant.google.com/. Accessed 29 June 2022.
GoogleResearch (2022a) Taskmaster from Google Research. https://github.com/google-research-
datasets/Taskmaster/tree/master/TM-1-2019. Accessed 29 June 2022.
GoogleResearch (2022b) Simulated Dialogue dataset from Google Research. https://github.com/
google-research-datasets/simulated-dialogue. Accessed 29 June 2022.
Janarthanam, S. (2017) Hands-On Chatbots and Conversational UI Development: Build chatbots
and voice user interfaces with Chatfuel, Dialogflow, Microsoft Bot Framework, Twilio, and
Alexa Skills. Packt Publishing.
Kedia, A. and Rasu, M. (2020) Hands-On Python Natural Language Processing: Explore tools and
techniques to analyze and process text with a view to building real-world NLP applications.
Packt Publishing.
Keras (2022a) Keras official sites: https://keras.io. Accessed 29 June 2022.
Keras (2022b) Keras optimizers: https://keras.io/api/optimizers/. Accessed 29 June 2022.
MicrosoftCortana (2022) Microsoft Cortana Developer: https://www.microsoft.com/en-us/cor-
tana/. Accessed 29 June 2022.
MicrosoftDialog (2022) https://github.com/xiul-msr/e2e_dialog_challenge. Accessed 29
June 2022.
NLPWorkshop7 (2022) NLP Workshop 6 GitHub archive. https://github.com/raymondshtlee/
NLP/tree/main/NLPWorkshop7. Accessed 29 June 2022.
Raj, S. (2018) Building Chatbots with Python: Using Natural Language Processing and Machine
Learning. Apress.
Rothman, D. (2022) Transformers for Natural Language Processing: Build, train, and fine-tune
deep neural network architectures for NLP with Python, PyTorch, TensorFlow, BERT, and
GPT-3. Packt Publishing.
Ruder (2022) An overview of optimizers: https://ruder.io/optimizing-gradient-descent/. Accessed
29 June 2022.
SamsungBixby (2022) Samsung Bixby Developer: https://developer.samsung.com/bixby.
Accessed 29 June 2022.
Telegram (2022) Telegram bot API documentation: (https://core.telegram.org/bots. Accessed 29
June 2022.
TensorFlow (2022) TensorFlow official site> https://tensorflow.org /. Accessed 29 June 2022.
Tuchong (2022) Wake word to invoke your Chatbot. https://stock.tuchong.com/image/detail?imag
eId=918495180260638796. Accessed 29 June 2022.
Tunstall, L, Werra, L. and Wolf, T. (2022) Natural Language Processing with Transformers:
Building Language Applications with Hugging Face. O’Reilly Media.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin,
I. (2017). Attention is all you need. Advances in neural information processing systems, 30.
https://arxiv.org/abs/1706.03762.
Xiaowei (2022) Xiaowei chatbot system from Tencent. https://xiaowei.tencent.com/. Accessed 29
June 2022.
Yıldırım, S, Asgari-Chenaghlu, M. (2021) Mastering Transformers: Build state-of-the-art models
from scratch with advanced natural language processing techniques. Packt Publishing.
Index

A Brill Taggers, 61–63

Abstractive Text Summary (ATS), 212 Brown Corpus, 22, 24, 56
Add-k smoothing, 36, 38
Adjectives, 43, 44
Adverbs, 43, 44, 51 C
AI Tutor, 3, 4, 15 Canonical form, 101, 102, 112
Alexa, 402, 403, 431 Carroll, L., 247
AliMe QA System, 231–232 Case roles analysis, 106
A Lite BERT (ALBERT), 195 Cataphor, 151
AllGenie, 402 CELEX, 47, 51–56
Anaphora, 151, 155, 158, 169 Centering algorithm, 166–169
Antecedent, 151, 164–166, 169, 170 Centering theory (CT), 166–169
Antonyms, 121–122 CFG parsing, 67, 79, 93
Artificial Intelligence Markup (AIML), 226 Chain rule, 24–25, 40
Artificial neural networks (ANNs), 180, Chatbots, 3, 4, 9, 10, 14, 401, 402, 404,
335, 357 410, 430
Attention equation, 410 Chomsky, N., 8
Augmented translation network (ATN), 9 COBUILD, 51
Coherence, 151–154, 159–162, 167, 171
Coherence relation, 159
B Cohesion, 154
Backoff and Interpolation (B&I), 39 Collocation, 263–264
BASEBALL system, 8 Common sense, 116, 124
Bidirectional, 373, 377 world knowledge, 21, 29
Bidirectional Encoder Representation from Compositional semantic analysis, 117–118, 145
Transformer (BERT), 192–196, Compositional semantics, 118
201, 204, 206, 227, 228, 243, 373, Computational linguistics (CL), 5, 10, 48
374, 377–392, 397 Computational linguists, 49
Bidirectional Recurrent Neural Network Computer-to-computer interaction (CCI), 149
(BRNN), 186–188 Conceptual dependency (CD), 99
Bigrams, 21, 27–32, 37, 38, 269, Conceptual dependency diagram
271–273, 275 (CDD), 98, 99
Bixby, 402, 431 Conjunctions, 43, 51–54
Bottom-up parser, 82–84 Connectives, 108
Brill, E., 61 Constituent, 70–74, 76, 77, 85, 88, 91

© The Author(s), under exclusive license to Springer Nature Singapore Pte 433
Ltd. 2024
R. S. T. Lee, Natural Language Processing,
https://doi.org/10.1007/978-981-99-1999-4
434 Index

Constituent Likelihood Automatic Word- G

tagging System (CLAWS), 56 Gate recurrent unit (GRU), 185–187, 196
Context-free grammar (CFG), 67, 76, Generic summarization systems, 217
77, 87, 93 Glove vectors, 320
Context-free language (CFL), 76–77 Good Turing (GT) Smoothing, 36, 40
Conversational artificial intelligence, 402 GPT, 226–228, 235
Convolutional neural networks (CNNs), 170 Grammar-based method, 221–223
Coordinating conjunctions, 52
Coreference, 150–153, 159, 163–170
Coreference resolution, 150, 151, 163 H
Cornell Large Movie Conversation Hapaxes, 262, 263
Dataset, 401 Hearst’s TextTiling, 155–157, 171
Cornell Movie Corpus archive, 404, 405 Heterogeneous transfer learning, 177
Corpus, 19, 21–25, 27, 29–31, 33, 36, 38–41, HMM Tagger, 60
44, 45, 51, 56, 57, 59, 60, 63, 244, Hobbs algorithm, 163–166
251, 252, 254–255, 259, 262, 264 Hobbs, J.R., 159, 163
Cortana, 402, 431 Homogeneous transfer learning, 177
Cosine similarity, 327–328 Homographs, 119, 124
Homonymy, 119–120
Homophones, 119
D Hopfield, J., 9
Decoder, 188 Hopfield Network, 9
Deep learning, 10 HuggingFace, 373, 382–385, 390, 393, 394
DeepQA, 9 Human–computer interaction (HCI), 10
Dekang Lin method, 135 Human language ambiguity, 7–8
Determiners, 43, 46, 51 Human-to-computer interaction (HCI), 149
Discourse, 48, 95, 97, 123, 124, 149, 151, 154, Hybrid POS tagging, 45
155, 158–162, 164, 166, 167, 171 Hypernymy, 122, 129
Discourse markers, 211 Hyponymy, 122, 129
Discourse segmentation, 154, 208–212, 235
Dispersion, 253–255
Distributional models, 137 I
Document-likelihood, 207, 208 IMDB, 340, 350–352, 355, 365, 369, 371
Duer, 402, 431 Inference, 102–103, 111, 112
Inflection, 43, 58
Information content similarity, 134–135
E Information extraction (IE), 15, 276, 283
Embedding words, 363–364 Information retrieval (IR), 15, 199–212,
Encoder, 188–192, 373, 375–377 226–228, 235, 276
Extractive Text Summary (ETS), 212 Interjections, 43, 50
Extrinsic evaluation, 35 Inverted document frequency (IDF), 200

F J
FastText, 320 Jupyter Notebook, 244
Feature-based model, 219
Feedforward neural network (FNN), 180
Fillmore’s Case Roles Theory, 153 K
Fillmore’s theory, 103–107, 112 Kaggle, 351
First-order predicate calculus (FOPC), 98, KBQA, 227, 228
100, 102, 103, 107–113 Keras, 336, 357–371, 401, 404, 415, 418, 420,
Frame-based representation, 98–100 422, 424, 430, 431
Index 435

Knowledge acquisition and inferencing Multi-head-attention (MHAttention), 191,

(KAI), 11 376, 411–412
Knowledge-based (KB), 125 Multiple document summarization, 217–224

L N
Language detection, 336 Name entity recognition (NER), 227, 276
Language model, 21, 24, 26, 34, 40 Natural language generation (NLG), 11
Language model evaluation (LME), 34–40 Natural language processing, 3–16
Language modeling, 24–25 Natural language toolkit (NLTK), 243–264,
Laplace (Add-one) Smoothing, 36–38 267–311, 401
Large Movie Reviews Dataset, 351 Natural language understanding (NLU),
Latent Semantic Analysis (LSA), 221 11–13, 48, 65, 70, 276
Latent Semantic Indexing, 207–208 Next sentence prediction (NSP), 193, 195
Lemma, 23 N-gram, 21, 22, 26, 27, 29–33, 35, 36, 38–41,
Lesk Algorithm, 136 46, 47, 58, 60, 86, 87, 90, 92, 211,
Lexical ambiguity, 7, 123 267, 268, 270, 271, 273
Lexical analysis, 75 Noun-phrase (NP), 72
Lexical dispersion plot, 253–255 Nouns, 43, 44, 49, 51, 53, 56
Lexical diversity, 258–259
Lexicalized parsing, 91–93
Lexical probability, 92, 93 P
Lexical semantic analysis, 117 Parsing, 67–94, 268
Lexical semantics, 117, 119, 145 Part-of-Speech (POS), 43–65, 67
Lexicology, 6 Path-based similarity, 132–134, 145
Lexicon, 75, 78, 80, 91 Penn Treebank (PTB), 45, 46
Linguistic levels, 6, 7 Perplexity (PP), 34–35, 41
Log-linear model, 170 Phonetics, 6
Long short-term memory (LSTM), 10, 170, Phonological parsing, 79
183–186, 188, 196, 224, 226, 230, Pointer-Generator Networks, 224
335–371, 374, 375, 388 Point-wise Mutual Information (PMI), 139
Luhn’s Algorithm, 219 Polysemy, 120, 129
Porter Stemmer, 285, 289–292
Positional encoding, 189–190, 413, 416
M Positive Point-wise Mutual Information
Machine learning (ML) method, 169–170 (PPMI), 140–142, 144–146
Machine translation (MT), 8, 14, 19, 22 POS tagger, 285, 304–306, 308–310
Markov chain, 25–27, 40 POS tagging, 44, 45, 47–49, 58, 60, 64, 65,
Maximum entropy Markov model 124, 244, 285–311
(MEMM), 60 POS Tagset, 301, 302
Maximum likelihood estimates (MLE), 30 Pragmatic, 124
Meaning representations, 95–98, 100–103, Pragmatic ambiguity, 8
110, 112, 113 Pragmatic analysis, 12, 13, 16
MeSH, 130–131 Pragmatic meaning, 95
Metaphor, 120 Pragmatics analysis, 149
Metonymy, 120 Predicates, 108, 109
Minsky, M., 99 Prepositions, 51–52
Modal verb, 54 Probabilistic context-free grammar (PCFG),
Morphological parsing, 79 87, 88, 90–91, 94
Morphology, 6 Probabilistic Ranking Principle (PRP), 202–207
Morphology analysis, 96 Pronouns, 43, 51, 53, 55
Movie comments, 351 PTB Tagset, 285, 302
436 Index

Q Stop-words, 292
Q&A chatbots, 19 Subject–Predicate–Object (SPO), 227
QA systems, 16, 224, 225, 227–229, 231, SummaRuNNer, 219, 221
233, 235 Supervised discourse segmentation, 158
Quadrigram, 22, 33, 269, 275 Supervised learning (SL), 125
Quantifiers, 108 SVD model, 208
Query-focused summarization (QFS) Symbolic representations, 118
systems, 215–216 Synonyms, 121
Query-likelihood, 207–208 Synsets, 126, 145
Syntactic ambiguity, 7
Syntactic levels, 6
R Syntactic parsing, 79, 80
Recurrent neural network (RNN), 170, Syntactic rules, 68
180–188, 196, 373 Syntax, 48, 67–94
Referring expression (RE), 159, 161 Syntax analysis, 13, 67
Regular language (RL), 76
Resnik method, 135, 145
Rhetorical structure theory (RST), 158, 209, 211 T
Rule-based POS tagging, 45 Taggers evaluations, 63–64
Rule-based QA systems, 227 Tagging, 43–65, 335, 336, 358
Tag sequence frequency, 60
Tagset, 56–58
S Taskmaster, 430
Selectional restrictions, 107 TensorFlow, 316, 335, 336, 357, 358, 367,
Self-attention, 190, 410 373–398, 401–431
Self-attention mechanism, 227 Term-context matrix, 139, 141, 145
Semantic ambiguity, 7 Term Distribution Models, 207
Semantic analysis, 12, 13, 244, 268, 313–333 Term-document matrix, 137–139
Semantic categorization, 330 Term-frequency (TF), 200
Semantic level, 6 Text analysis, 243, 249, 253, 257,
Semantic meaning, 95 259, 296–299
Semantic networks, 98–99 TextCategorizer, 335, 338–343, 345, 347–351,
Semantic processing, 97 355, 356
Semantics, 48, 95, 107, 108, 111 Text classification, 244, 335–371
Semantic similarity, 320, 323, 326–333 Text coherent, 158–159
Semi-supervised methods, 125 Text processing, 248–249, 252
Sentiment analysis, 15, 337, 338, 401 Text summarization (TS), 212–224, 235
Shakespeare, W., 31 TextTeaser, 219
Shannon’s method, 31, 32 TextTiling, 208
Sherlock Holmes, 27–31, 33, 35, 37, 38, 40, Thesaurus, 130–132
41, 247, 249, 251, 256, 257, 262, TL system, 235
263, 268, 270, 271, 275, 278–281 Tokenization, 243, 244, 255–259,
Single and multiple document 267–288, 301
summarization, 217 Tokens, 23
Smoothing, 141–143 Top-down parser, 80–82
Smoothing techniques, 36 Topic generation, 336
Snowball Stemmer, 285, 289, 291–292 Transfer learning (TL), 175–196
SpaCy, 243, 267, 276–284, 313–333, TransferTransfo Conversational
335–371, 373–398, 401 Agents, 234–235
Speech recognition, 10, 12, 13, 268 Transformation-based learning (TBL),
Stem, 23 61–62, 65
Stemming, 244, 285, 288, 289, 301 Transformers, 170, 175–196, 243,
Stochastic POS tagging, 45 373–398, 401–431
Index 437

Trigrams, 21, 27, 32, 269 Wide&Deep model, 202, 205

Turing, A., 5, 8 WordCloud, 285, 299–301
Turing Test, 5 Word form, 23
Two-tower model, 202 Word frequency, 60
WordNet, 123, 126–130, 145
Word prediction, 19, 20
U Word sense, 118, 123
Unigram, 21, 27, 28, 32 Word sense disambiguation (WSD),
Unsupervised discourse 123–126, 145
segmentation, 154–155 Word tokenization, 244
Utterance, 23 Word usage frequency, 259
Word2vec, 314, 320
Word vectors, 137, 313, 314, 317–320, 326
V World knowledge, 116, 124
Vagueness, 100, 101
Vectorial representation, 118
Vector space model, 200–204 X
Verb-phrase (VP), 72–73 Xiao Ice QA System, 232–234
Verbs, 43, 44, 47, 53–56, 59, 64 Xiaowei, 402, 431
Verifiability, 100
VP subcategorization, 74–75
Z
Zeugma test, 120–121, 145
W Zipf’s law, 36
Wake word, 403

NLP Front Matter
No ratings yet
NLP Front Matter
28 pages
Topic 2: Introduction To Natural Language Processing (NLP)
No ratings yet
Topic 2: Introduction To Natural Language Processing (NLP)
16 pages
Natural Language Processin1
No ratings yet
Natural Language Processin1
86 pages
Unit 3
No ratings yet
Unit 3
14 pages
Chapter - 6 Communicating, Perceiving, and Acting
No ratings yet
Chapter - 6 Communicating, Perceiving, and Acting
30 pages
Introducing Natural Language Processing
No ratings yet
Introducing Natural Language Processing
13 pages
Introduction To Natural Language Processing
No ratings yet
Introduction To Natural Language Processing
211 pages
NLP LectureNotes UNIT 1
No ratings yet
NLP LectureNotes UNIT 1
55 pages
Unit 1
No ratings yet
Unit 1
99 pages
SCO409 Lecture Notes
No ratings yet
SCO409 Lecture Notes
64 pages
NLP PPT1
No ratings yet
NLP PPT1
29 pages
Natural Language Processing 101
No ratings yet
Natural Language Processing 101
26 pages
Harmonizing Humanity and Technology
No ratings yet
Harmonizing Humanity and Technology
10 pages
Unit 4
No ratings yet
Unit 4
39 pages
NLP Lect Unit I
100% (1)
NLP Lect Unit I
140 pages
What Is Natural Language Processing?
No ratings yet
What Is Natural Language Processing?
5 pages
Nlp-Unit-I Final
No ratings yet
Nlp-Unit-I Final
31 pages
NLP 833
No ratings yet
NLP 833
26 pages
NLP 1
No ratings yet
NLP 1
29 pages
An In-Depth Exploration of Natural Language Processing: Evolution, Applications, and Future Directions
100% (8)
An In-Depth Exploration of Natural Language Processing: Evolution, Applications, and Future Directions
5 pages
Natural Language Processing New
No ratings yet
Natural Language Processing New
25 pages
Natural Language Processing
No ratings yet
Natural Language Processing
21 pages
NLP Notes
No ratings yet
NLP Notes
90 pages
Chapter 6.
No ratings yet
Chapter 6.
31 pages
(IJETA-V11I3P43) :ravi Joshi, Sheikh Mohammed Farhan, Udeshya Sharma, Sahil Bhatt
No ratings yet
(IJETA-V11I3P43) :ravi Joshi, Sheikh Mohammed Farhan, Udeshya Sharma, Sahil Bhatt
6 pages
Introduction To Natural Language Processing
No ratings yet
Introduction To Natural Language Processing
32 pages
NLP Materia
No ratings yet
NLP Materia
29 pages
Natural Language Processing Inside Pages 2
No ratings yet
Natural Language Processing Inside Pages 2
159 pages
NLP Lab Manual-1
No ratings yet
NLP Lab Manual-1
18 pages
Ai 2
No ratings yet
Ai 2
7 pages
NLP MODULE 1 Chapter1 &2
No ratings yet
NLP MODULE 1 Chapter1 &2
83 pages
Natural Language Processing Unit1
No ratings yet
Natural Language Processing Unit1
23 pages
Module I NLP
No ratings yet
Module I NLP
65 pages
Natural Language Processing (NLP) : April 2024
No ratings yet
Natural Language Processing (NLP) : April 2024
88 pages
Unit 1
No ratings yet
Unit 1
20 pages
Introduction To Data Science - Week 7 - LAQ's
No ratings yet
Introduction To Data Science - Week 7 - LAQ's
4 pages
Notes MSC NLP
No ratings yet
Notes MSC NLP
36 pages
Natural Language Processing - Bridging The Gap Between Humans and Machines
No ratings yet
Natural Language Processing - Bridging The Gap Between Humans and Machines
6 pages
Sha 10
No ratings yet
Sha 10
6 pages
Natural Language Processing State of The Art, Current Trends and Challenges - s11042-022-13428-4 PDF
No ratings yet
Natural Language Processing State of The Art, Current Trends and Challenges - s11042-022-13428-4 PDF
32 pages
Natural Language Processing: Zhao Hai 赵海 Department of Computer Science and Engineering Shanghai Jiao Tong University
No ratings yet
Natural Language Processing: Zhao Hai 赵海 Department of Computer Science and Engineering Shanghai Jiao Tong University
61 pages
NLP 1
No ratings yet
NLP 1
37 pages
CC S 339 NLP Basics &TSA
No ratings yet
CC S 339 NLP Basics &TSA
68 pages
Deep Learning For Natural Language Processing: July 2021
No ratings yet
Deep Learning For Natural Language Processing: July 2021
10 pages
NLP Sheets
No ratings yet
NLP Sheets
23 pages
Eco 36
No ratings yet
Eco 36
6 pages
Introduction To Natural Language Processing
No ratings yet
Introduction To Natural Language Processing
31 pages
NLP Notes Unit-1
No ratings yet
NLP Notes Unit-1
20 pages
Bhawini NLP Practical
No ratings yet
Bhawini NLP Practical
98 pages
Natural Language Processing Tools and Approaches
No ratings yet
Natural Language Processing Tools and Approaches
106 pages
Natural Language Processing (NLP) With Python - Tutorial
No ratings yet
Natural Language Processing (NLP) With Python - Tutorial
72 pages
Natural Language Processing State of The Art Curre
No ratings yet
Natural Language Processing State of The Art Curre
33 pages
ML Module A7707 - Part1
No ratings yet
ML Module A7707 - Part1
48 pages
Nlp-Unit-I Final
No ratings yet
Nlp-Unit-I Final
31 pages
Wisdom Natural Language Processing
No ratings yet
Wisdom Natural Language Processing
4 pages
Module-I NLP
No ratings yet
Module-I NLP
35 pages
Unit 1a
No ratings yet
Unit 1a
53 pages
Introduction To NLP - Part 1
No ratings yet
Introduction To NLP - Part 1
23 pages
Akchukwu Wisdom Chidi Seminar Corrected Version
No ratings yet
Akchukwu Wisdom Chidi Seminar Corrected Version
17 pages
Mastering Natural Language Processing with Python and NLTK
From Everand
Mastering Natural Language Processing with Python and NLTK
Pedro Martins
No ratings yet
Artificial Intelligence (Subject Code - 417)
No ratings yet
Artificial Intelligence (Subject Code - 417)
9 pages
Analysis and Representation of Igbo Text Document For A Text-Based System
No ratings yet
Analysis and Representation of Igbo Text Document For A Text-Based System
7 pages
Artificial Intelligence and Engineering Design
No ratings yet
Artificial Intelligence and Engineering Design
4 pages
Ai Important Questions For Viva
No ratings yet
Ai Important Questions For Viva
4 pages
The Medical Chatbot Practical 01 To 05
No ratings yet
The Medical Chatbot Practical 01 To 05
44 pages
Spacy Io Usage Spacy 101
No ratings yet
Spacy Io Usage Spacy 101
10 pages
Computational Linguistics
No ratings yet
Computational Linguistics
4 pages
Large Language Models Ryan Cotterell, Mrinmaya Sachan, Florian Tramèr, Ce ZhangLecture 1
No ratings yet
Large Language Models Ryan Cotterell, Mrinmaya Sachan, Florian Tramèr, Ce ZhangLecture 1
37 pages
Subtitle
No ratings yet
Subtitle
2 pages
10 1016@j Copsyc 2020 04 005
No ratings yet
10 1016@j Copsyc 2020 04 005
6 pages
Prateek Kumar Tiwari: Internship Skills
No ratings yet
Prateek Kumar Tiwari: Internship Skills
1 page
Full Download Deep Learning in Natural Language Processing Deng PDF
100% (4)
Full Download Deep Learning in Natural Language Processing Deng PDF
55 pages
The Generative Pre-Trained Transformer: GPT-3
No ratings yet
The Generative Pre-Trained Transformer: GPT-3
1 page
ASEAN AWS Academy Quickstart
No ratings yet
ASEAN AWS Academy Quickstart
12 pages
ACWR 106 Assignment For Style Manuals
No ratings yet
ACWR 106 Assignment For Style Manuals
2 pages
Amity School of Engineering and Technology: In-House Practical (Etpt100)
No ratings yet
Amity School of Engineering and Technology: In-House Practical (Etpt100)
25 pages
Digital Marketing Brochure 2024
No ratings yet
Digital Marketing Brochure 2024
19 pages
Bachelor in Artificial Intelligence and Data Science La Salle Campus Barcelona
No ratings yet
Bachelor in Artificial Intelligence and Data Science La Salle Campus Barcelona
2 pages
Research On The Application of Deep Learning-Based BERT Model in Sentiment Analysis
No ratings yet
Research On The Application of Deep Learning-Based BERT Model in Sentiment Analysis
10 pages
Web Application For Screening Resume: January 2019
No ratings yet
Web Application For Screening Resume: January 2019
10 pages
Ai Unit 1 - Compressed
No ratings yet
Ai Unit 1 - Compressed
142 pages
AI in Software An Overview
No ratings yet
AI in Software An Overview
10 pages
Aiet Brochure
No ratings yet
Aiet Brochure
14 pages
6CS4 AI Unit-5
No ratings yet
6CS4 AI Unit-5
65 pages
NLP-Intro
No ratings yet
NLP-Intro
61 pages
Covibot-An Intelligent Whatsapp Based Advising Bot For Covid-19
No ratings yet
Covibot-An Intelligent Whatsapp Based Advising Bot For Covid-19
5 pages
Moot Proposition For National Moot and Mediation Competition
No ratings yet
Moot Proposition For National Moot and Mediation Competition
9 pages
Natural Language Understanding
No ratings yet
Natural Language Understanding
3 pages
Unit - 1 Artificial Intelligence: History of AI
No ratings yet
Unit - 1 Artificial Intelligence: History of AI
19 pages
Data Science Lab-KTU
No ratings yet
Data Science Lab-KTU
5 pages