NLP Front Matter
NLP Front Matter
net/publication/375744928
CITATIONS READS
8 4,274
1 author:
Raymond Lee
BNU-HKBU United International College
125 PUBLICATIONS 940 CITATIONS
SEE PROFILE
All content following this page was uploaded by Raymond Lee on 19 November 2023.
© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore
Pte Ltd. 2024
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmission or information storage and retrieval, electronic adaptation, computer software, or by similar
or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors, and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.
This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
This book is dedicated to all readers and
students taking my undergraduate and
postgraduate courses in Natural Language
Processing, your enthusiasm in seeking
knowledge incited me to write this book.
Preface
Natural Language Processing (NLP) and its related applications become part of
daily life with exponential growth of Artificial Intelligence (AI) in past decades.
NLP applications including Information Retrieval (IR) systems, Text Summarization
System, and Question-and-Answering (Chatbot) System became one of the preva-
lent topics in both industry and academia that had evolved routines and benefited
immensely to a wide array of day-to-day services.
The objective of this book is to provide NLP concepts and knowledge to readers
with a 14-h 7 step-by-step workshops to practice various core Python-based NLP
tools: NLTK, spaCy, TensorFlow Keras, Transformer, and BERT Technology to
construct NLP applications.
vii
viii Preface
This introductory chapter begins with human language and intelligence con-
stituting six levels of linguistics followed by a brief history of NLP with major
components and applications. It serves as the cornerstone to the NLP concepts
and technology discussed in the following chapters. This chapter also serves as
the conceptual basis for Workshop#1: Basics of Natural Language Toolkit
(NLTK) in Chap. 10.
• Chapter 2: N-gram Language Model
Language model is the foundation of NLP. This chapter introduces N-gram
language model and Markov Chains using classical literature The Adventures of
Sherlock Holmes by Sir Conan Doyle (1859–1930) to illustrate how N-gram
model works that form NLP basics in text analysis followed by Shannon’s model
and text generation with evaluation schemes. This chapter also serves as the con-
ceptual basis for Workshop#2 on N-gram modelling with NLTK in Chap. 11.
• Chapter 3: Part-of-Speech Tagging
Part-of-Speech (POS) Tagging is the foundation of text processing in
NLP. This chapter describes how it relates to NLP and Natural Language
Understanding (NLU). There are types and algorithms for POS Tagging includ-
ing Rule-based POS Tagging, Stochastic POS Tagging, and Hybrid POS Tagging
with Brill Tagger and evaluation schemes. This chapter also serves as the concep-
tual basis for Workshop#3: Part-of-Speech using Natural Language Toolkit in
Chap. 12.
• Chapter 4—Syntax and Parsing
As another major component of Natural Language Understanding (NLU),
this chapter explores syntax analysis and introduces different types of constitu-
ents in English language followed by the main concept of context-free grammar
(CFG) and CFG parsing. It also studies different major parsing techniques,
including lexical and probabilistic parsing with live examples for illustration.
• Chapter 5: Meaning Representation
Before the study of Semantic Analysis, this chapter explores meaning repre-
sentation, a vital component in NLP. It studies four major meaning representa-
tion techniques which include: first-order predicate calculus (FOPC), semantic
net, conceptual dependency diagram (CDD), and frame-based representation.
After that it explores canonical form and introduces Fillmore’s theory of univer-
sal cases followed by predicate logic and inference work using FOPC with live
examples.
• Chapter 6: Semantic Analysis
This chapter studies Semantic Analysis, one of the core concepts for learning
NLP. First, it studies the two basic schemes of semantic analysis: lexical and
compositional semantic analysis. After that it explores word senses and six com-
monly used lexical semantics followed by word sense disambiguation (WSD)
and various WSD schemes. Further, it also studies WordNet and online thesauri
for word similarity and various distributed similarity measurement including
Point-wise Mutual Information (PMI) and Positive Point-wise Mutual informa-
tion (PPMI) models with live examples for illustration. Chapters 4 and 5 also
Preface ix
serve as the conceptual basis for Workshop#4: Semantic Analysis and Word
Vectors using spaCy in Chap. 13.
• Chapter 7: Pragmatic Analysis
After the discussion of semantic meaning and analysis, this chapter explores
pragmatic analysis in linguistics and discourse phenomena. It also studies coher-
ence and coreference as the key components of pragmatics and discourse critical
to NLP, followed by discourse segmentation with different algorithms on Co-
reference Resolution including Hobbs Algorithm, Centering Algorithm, Log-
Linear Model, the latest machine learning methods, and evaluation schemes.
This chapter also serves as the conceptual basis for Workshop#5: Sentiment
Analysis and Text Classification in Chap. 14.
• Chapter 8: Transfer Learning and Transformer Technology
Transfer learning is a commonly used deep learning model to minimize com-
putational resources. This chapter explores: (1) Transfer Learning (TL) against
traditional Machine Learning (ML); (2) Recurrent Neural Networks (RNN), a
significant component of transfer learning with core technologies such as Long
Short-Term Memory (LSTM) Network and Bidirectional Recurrent Neural
Networks (BRNNs) in NLP applications, and (3) Transformer technology archi-
tecture, Bidirectional Encoder Representation from Transformers (BERT)
Model, and related technologies including Transformer-XL and ALBERT tech-
nologies. This chapter also serves as the conceptual basis for Workshop#6:
Transformers with spaCy and Tensorflow in Chap. 15.
• Chapter 9: Major Natural Language Processing Applications
This is a summary of Part I with three core NLP applications: Information
Retrieval (IR) systems, Text Summarization (TS) systems, and Question-and-
Answering (Q&A) chatbot systems, how they work and related R&D in building
NLP applications. This chapter also serves as the conceptual basis for
Workshop#7: Building Chatbot with TensorFlow and Transformer Technology
in Chap. 16.
provide a foundation technique for text analysis, parsing and semantic analysis
in subsequent workshops. Part II introduces spaCy, the second important NLP
Python implementation tools not only for teaching and learning (like NLTK) but
also widely used for NLP applications including text summarization, informa-
tion extraction, and Q&A chatbot. It is a critical mass to integrate with
Transformer Technology in subsequent workshops.
• Chapter 12: Workshop#3 Part-of-Speech Tagging with Natural Language Toolkit
(Hour 5–6)
In Chap. 3, we studied basic concepts and theories related to Part-of-Speech
(POS) and various POS tagging techniques. This workshop explores how to
implement POS tagging by using NLTK starting from a simple recap on tokeni-
zation techniques and two fundamental processes in word-level progressing:
stemming and stop-word removal, which will introduce two types of stemming
techniques: Porter Stemmer and Snowball Stemmer that can be integrated with
WordCloud commonly used in data visualization followed by the main theme of
this workshop with the introduction of PENN Treebank Tagset and to create your
own POS tagger.
• Chapter 13: Workshop#4 Semantic Analysis and Word Vectors using spaCy
(Hour 7–8)
In Chaps. 5 and 6, we studied the basic concepts and theories related to mean-
ing representation and semantic analysis. This workshop explores how to use
spaCy technology to perform semantic analysis starting from a revisit on word
vectors concept, implement and pre-train them followed by the study of similar-
ity method and other advanced semantic analysis.
• Chapter 14: Workshop#5 Sentiment Analysis and Text Classification (Hour 9–10)
This is a coherent workshop of Chap. 7, this workshop explores how to posi-
tion NLP implementation techniques into two important NLP applications: text
classification and sentiment analysis. TensorFlow and Kera are two vital compo-
nents to implement Long Short-Term Memory networks (LSTM networks), a
commonly used Recurrent Neural Networks (RNN) on machine learning espe-
cially in NLP applications.
• Chapter 15: Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)
In Chap. 8, the basic concept about Transfer Learning, its motivation and
related background knowledge such as Recurrent Neural Networks (RNN) with
Transformer Technology and BERT model are introduced. This workshop
explores how to put these concepts and theories into practice. More importantly,
is to implement Transformers, BERT Technology with the integration of spaCy’s
Transformer Pipeline Technology and TensorFlow. First, it gives an overview
and summation on Transformer and BERT Technology. Second, it explores
Transformer implementation with TensorFlow by revisiting Text Classification
using BERT model as example. Third, it introduces spaCy’s Transformer Pipeline
Technology and how to implement Sentiment Analysis and Text Classification
system using Transformer Technology.
Preface xi
This book is both an NLP textbook and NLP Python implementation book tai-
lored for:
• Undergraduates and postgraduates of various disciplines including AI, Computer
Science, IT, Data Science, etc.
• Lecturers and tutors teaching NLP or related AI courses.
• NLP, AI scientists and developers who would like to learn NLP basic concepts,
practice and implement via Python workshops.
• Readers who would like to learn NLP concepts, practice Python-based NLP
workshops using various NLP implementation tools such as NLTK, spaCy,
TensorFlow Keras, BERT, and Transformer technology.
This book can be served as a textbook for undergraduates and postgraduate courses
on Natural Language Processing, and a reference book for general readers who
would like to learn key technologies and implement NLP applications with contem-
porary implementation tools such as NLTK, spaCy, TensorFlow, BERT, and
Transformer technology.
Part I (Chaps. 1–9) covers the main course materials of basic concepts and key
technologies which include N-gram Language Model, Part-of-Speech Tagging,
Syntax and Parsing, Meaning Representation, Semantic Analysis, Pragmatic
xii Preface
xiii
About the Book
xv
Contents
xvii
xviii Contents
16
Workshop#7 Building Chatbot with TensorFlow and Transformer
Technology (Hour 13–14)������������������������������������������������������������������������ 401
16.1 Introduction������������������������������������������������������������������������������������ 401
16.2 Technical Requirements������������������������������������������������������������������ 401
16.3 AI Chatbot in a Nutshell ���������������������������������������������������������������� 402
16.3.1 What Is a Chatbot?������������������������������������������������������������ 402
16.3.2 What Is a Wake Word in Chatbot?������������������������������������ 403
16.3.3 NLP Components in a Chatbot ���������������������������������������� 404
16.4 Building Movie Chatbot by Using TensorFlow
and Transformer Technology���������������������������������������������������������� 404
16.4.1 The Chatbot Dataset���������������������������������������������������������� 405
16.4.2 Movie Dialog Preprocessing�������������������������������������������� 405
16.4.3 Tokenization of Movie Conversation�������������������������������� 407
16.4.4 Filtering and Padding Process������������������������������������������ 408
16.4.5 Creation of TensorFlow Movie Dataset
Object (mDS)�������������������������������������������������������������������� 409
16.4.6 Calculate Attention Learning Weights������������������������������ 410
16.4.7 Multi-Head-Attention (MHAttention)������������������������������ 411
16.4.8 System Implementation���������������������������������������������������� 412
16.5 Related Works �������������������������������������������������������������������������������� 430
References�������������������������������������������������������������������������������������������������� 431
Index�������������������������������������������������������������������������������������������������������������������� 433
About the Author
Raymond Lee is the founder of the Quantum Finance Forecast System (QFFC)
(https://qffc.uic.edu.cn) and currently an Associate Professor at United International
College (UIC) with 25+ years’ experience in AI research and consultancy, Chaotic
Neural Networks, NLP, Intelligent Fintech Systems, Quantum Finance, and
Intelligent E-Commerce Systems. He has published over 100 publications and
authored 8 textbooks in the fields of AI, chaotic neural networks, AI-based fintech
systems, intelligent agent technology, chaotic cryptosystems, ontological agents,
neural oscillators, biometrics, and weather simulation and forecasting systems.
Upon completion of the QFFC project, in 2018 he joined United International
College (UIC), China, to pursue further R&D work on AI-Fintech and to share his
expertise in AI-Fintech, chaotic neural networks, and related intelligent systems
with fellow students and the community. His three latest textbooks, Quantum
Finance: Intelligent Forecast and Trading Systems (2019), Artificial Intelligence in
Daily Life (2020), and this NLP book have been adopted as the main textbooks for
various AI courses in UIC.
xxix
Abbreviations
AI Artificial intelligence
ASR Automatic speech recognition
BERT Bidirectional encoder representations from transformers
BRNN Bidirectional recurrent neural networks
CDD Conceptual dependency diagram
CFG Context-free grammar
CFL Context-free language
CNN Convolutional neural networks
CR Coreference resolution
DNN Deep neural networks
DT Determiner
FOPC First-order predicate calculus
GRU Gate recurrent unit
HMM Hidden Markov model
IE Information extraction
IR Information retrieval
KAI Knowledge acquisition and inferencing
LSTM Long short-term memory
MEMM Maximum entropy Markov model
MeSH Medical subject thesaurus
ML Machine learning
NER Named entity recognition
NLP Natural language processing
NLTK Natural language toolkit
NLU Natural language understanding
NN Noun
NNP Proper noun
Nom Nominal
NP Noun phrase
PCFG Probabilistic context-free grammar
PMI Pointwise mutual information
xxxi
xxxii Abbreviations
POS Part-of-speech
POST Part-of-speech tagging
PPMI Positive pointwise mutual information
Q&A Question-and-answering
RNN Recurrent neural networks
TBL Transformation-based learning
VB Verb
VP Verb phrase
WSD Word sense disambiguation