Chapter – 8

Applications of NLP – Part I

Department of Computer Science

School of Computing
Dire Dawa Institute of Technology
Dire Dawa University

Te ssf u Ge t e ye (Ph D)

2020/2021-Se me st er- II
 Information Retrieval
The Retrieval Process
Information
Classic IR Models
Extraction Machine
IR Performance Evaluation
Translation
NLP in IR
Question-Answering and Dialogue
Sy st em s
The Retrieval Process Text Summarization

 Information Retrieval (IR) provides a list of potentially relevant documents in response

to users query.

User Interface
user need

Text Operations text text

user feedback logical view logical view

Query Operations Indexing DB Manager Module
query inverted file

Searching Index
Text
ranked docs retrieved
Database
docs
Ranking

The process of retrieving information [Baeza-Yates & Ribeiro-Neto]

Department of Computer Science, SC, DDIT, DDU Applications of NLP 2/59

 Information Retrieval The Retrieval Process
Information Classic IR Models
Extraction Machine Deep Learning approaches
Translation IR Performance Evaluation
Question-Answering and Dialogue NLP in IR
Sy st em s
Classic IR Models Text Summarization

 Classic IR models consider that each document is described by a set of representative

keywords called index terms.

 Index terms have the following characteristics

 (Document) words whose semantics help in remembering the documents’ main

themes.

 Used to index and summarize the documents.

 Mainly nouns because nouns have meaning by themselves.

 Depending on how index terms are treated, there are three classic IR models: Boolean,
Vector and Probabilistic models.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 3/59

 Information Retrieval The Retrieval Process
Information Classic IR Models
Extraction Machine Deep Learning approaches
Translation IR Performance Evaluation
Question-Answering and Dialogue NLP in IR
Sy st em s
Classic IR Models: Boolean Model
Text Summarization

 Based on set theoretic (set theory and Boolean algebra) concepts.

 Documents and queries are represented as a set of index terms.

 Queries are specified as Boolean expressions which have precise semantics.

 Similarity of documents to queries is measured with exact matching.

 Retrieval strategy is based on binary decisions (relevant or non-relevant).

 Considers that index terms are present or absent.

 Index terms are linked by three connectives (AND, OR, NOT).

 Simple but not efficient.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 4/59

 Information Retrieval The Retrieval Process
Information Classic IR Models
Extraction Machine Deep Learning approaches
Translation IR Performance Evaluation
Question-Answering and Dialogue NLP in IR
Sy st em s
Classic IR Models: Vector Model
Text Summarization

 Based on algebraic concepts.

 Documents and queries are represented as vectors in a t-dimensional vector space.

 Recognizes that the use of binary weights is too limiting.

 Assigns non-binary weights to index terms in queries and documents.

 These term weights are used to compute the degree of similarity each
document stored in the systems and the user query.
 Retrieved documents can be sorted in decreasing order to get ranked list of
documents.

 Advantages:

 Its term-weighting scheme improves retrieval performance.

 Its partial matching strategy allows retrieval of documents that approximate
query conditions.
 Ranking of retrieved documents.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 5/59

 Information Retrieval The Retrieval Process
Information Classic IR Models
Extraction Machine Deep Learning approaches
Translation IR Performance Evaluation
Question-Answering and Dialogue NLP in IR
Sy st em s
Classic IR Models: Probabilistic Model
Text Summarization

 Based on probabilistic concepts.

 Captures the IR problem with the assumption that for a given user query there is a set
of documents which contains exactly the relevant documents and no other.
 This set of documents is the ideal answer set.
 Given the description of this ideal answer set, there would be no problem in
retrieving its documents.
 The querying process can then be thought of as a process of specifying the properties
of an ideal answer set.
 Initially guess the properties (we can start by using index terms).
 User feed back is then initiated and taken to improve the probability that the
user will find document d with query q.
 Measure of document similarity to the query:
P(d relevant-to q)
P(d non-relevant-to q)
 The main advantage is that documents are ranked in decreasing order of their
probability of being relevant.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 6/59

 Information Retrieval The Retrieval Process
Information Classic IR Models
Extraction Machine Deep Learning approaches
Translation IR Performance Evaluation
Question-Answering and Dialogue NLP in IR
Sy st em s
Deep Learning to IR Text Summarization

 The journey of an IR process begins with a user query sent to the IR system which
encodes the query, compares the query with the available resources, and returns the
most relevant pieces of information. Thus, the system is equipped with the ability to
store, retrieve and maintain information.
 In the early era of IR, the whole process was completed using handcrafted features and
ad-hoc relevance measures.
 Later, principled frameworks for relevance measure were developed with statistical
learning as a basis.
 Recently, deep learning has proven essential to the introduction of more opportunities
to IR. This is because data-driven features combined with data-driven relevance
measures can effectively eliminate the human bias in either feature or relevance
measure design.
 Deep Learning is used for IR for all components.

 Document Ranking

 Document Indexing

 Query Processing

 Document Searching

Department of Computer Science, SC, DDIT, DDU Applications of NLP 7/59

 Information Retrieval The Retrieval Process
Information Classic IR Models
Extraction Machine Deep Learning approaches
Translation IR Performance Evaluation
Question-Answering and Dialogue NLP in IR
Sy st em s
Deep Learning to IR Text Summarization

 For example: Document Ranking

 Traditional IR models include basic handcrafted retrieval models, semantic-

based models, term dependency-based models, and learning to rank models.

 The deep learning approaches, on the other hand, involve methods of

representation learning, methods of matching function learning, and methods of
relevance learning.

 The capability of neural ranking models to extract features directly from raw
text inputs overcomes many limitations of traditional IR models that rely on
handcrafted features.

 Moreover, the deep learning methods manage to capture complicated matching

patterns for document ranking.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 8/59

 Information Retrieval The Retrieval Process
Information Classic IR Models
Extraction Machine Deep Learning approaches
Translation IR Performance Evaluation
Question-Answering and Dialogue NLP in IR
Sy st em s
IR Performance Evaluation Text Summarization

 The performance of IR systems can be evaluated by using two commonly used metrics:
precision and recall.

 Recall is the fraction of the relevant documents which has been retrieved.
relevant  retrieved
R ecall =
relevant

 Precision is the fraction of the retrieved documents

which is relevant.
relevant  retrieved
Precision =
retrieved

Department of Computer Science, SC, DDIT, DDU Applications of NLP 9/59

 Information Retrieval The Retrieval Process
Information Classic IR Models
Extraction Machine Deep Learning approaches
Translation IR Performance Evaluation
Question-Answering and Dialogue NLP in IR
Sy st em s
NLP in IR Text Summarization

 NLP is widely used to improve the performance of IR systems.

 Most commonly used applications are:

 Lexical analysis: with the objective of treating digits, hyphens, punctuation

marks, and the case of letters.

 Stop word removal: with the objective of filtering out words with very low
discrimination values for retrieval purpose.

 Stemming: with the objective of removing affixes.

 Automatic indexing: with the objective of determining representative words

(or groups of words (usually noun groups).

 Document clustering: with the objective of building the relationships

between
documents.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 10/59

 Information Retrieval The Retrieval Process
Information Classic IR Models
Extraction Machine Deep Learning approaches
Translation IR Performance Evaluation
Question-Answering and Dialogue NLP in IR
Sy st em s
NLP in IR Text Summarization

 Construction of lexical relationships: with the objective of building term

categorization structures such as thesaurus which allows
the expansion of the original query with related terms.

 Improves recall and precision.

 Synonymy: absence of synonymy relationship yields poor recall from

missing synonymous documents.

 Polysemy/homonymy: terms yield poor precision results from search

terms that have multiple meanings leading to retrieval of non-
relevant documents.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 11/59

 Information Retrieval Co mpo nents of Information Extraction
Information N amed Entity Recognition
Extraction Machine Relation Detection and Classification
Translation Temporal and Event Processing
Question-Answering and Dialogue Template Filling
Sy st em s
Components of Information Extraction Text Summarization

 Information Extraction (IE) focuses on the recognition, tagging, and extraction of

certain key elements of information (e.g. persons, companies, locations, organizations,
etc.) from large collections of text into a structured representation.

Example: Firm XYZ is a full service advertising agency specializing in direct and
interactive marketing. Located in Bole, Addis Ababa, Firm XYZ is
Text: looking for an Assistant Account Manager to help manage and
coordinate
interactive marketing initiatives. Experience in online marketing and/or the
advertising field is a plus. Depending on the experiences of the applicants,
the company pays an attractive salary of Birr 3,000- Birr 5,000 per month.

Extracted Information:

INDUSTRY: Advertising
POSITION: Assistant
Account Manager LOCATION: Bole,
Addis Ababa.
COMPANY: Firm XYZ
SALARY: Birr 3,000 -
Birr 5,000 per month
Department of Computer Science, SC, DDIT, DDU Applications of NLP 12/59
 Information Retrieval Co mpo nents of Information Extraction
Information N amed Entity Recognition
Extraction Machine Relation Detection and Classification
Translation Temporal and Event Processing
Question-Answering and Dialogue Template Filling
Sy st em s
Components of Information Extraction Text Summarization

 IE is applied to a narrowly restricted domain.

 It has the following subtasks:

 Named Entity Recognition: recognition of entity names.

 Relation Detection and Classification: identification of relations between entities.

 Coreference and Anaphoric Resolution: resolving links to previously named

entities.

 Temporal and Event Processing: recognizing temporal expressions and analyzing

events.

 Template Filling: filling in the extracted information.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 13/59

 Information Retrieval Co mpo nents of Information Extraction
Information N amed Entity Recognition
Extraction Machine Relation Detection and Classification
Translation Temporal and Event Processing
Question-Answering and Dialogue Template Filling
Sy st em s
Named Entity Recognition Text Summarization

 Named Entity Recognition is the process of recognition of entity names such as:

 People: Abebe, Kebede, አበበ, ከበደ, etc.

 Organization: Ministry of Education, ABC Company, ትምህርት ሚኒስቴር, ሀለመ ኩባንያ,

etc.

 Place: Addis Ababa, Megenagna, አዲስ አበባ, መገናኛ, etc.

 Time expression: Tuesday, February 14, ማክሰኞ, የካቲት 6, etc.

 Quantities: three quintals of teff, 3000 Birr, ሶስት ኩንታል ጤፍ, 3ሺ ብር, etc.

 etc.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 14/59

 Information Retrieval Co mpo nents of Information Extraction
Information N amed Entity Recognition
Extraction Machine Relation Detection and Classification
Translation Temporal and Event Processing
Question-Answering and Dialogue Template Filling
Sy st em s
Relation Detection and Classification Text Summarization

 Relation Detection and Classification involves identification of relations between entities.

 Examples:

 PERSON works for ORGANIZATION

 ORGANIZATION located in PLACE

 PERSON lives in PLACE

 SALARY amounts to QUANTITY

 PERSON is paid SALARY

 PERSON works for ORGANIZATION since DATE

Department of Computer Science, SC, DDIT, DDU Applications of NLP 15/59

 Information Retrieval Co mpo nents of Information Extraction
Information N amed Entity Recognition
Extraction Machine Relation Detection and Classification
Translation Temporal and Event Processing
Question-Answering and Dialogue Template Filling
Sy st em s
Temporal and Event Processing Text Summarization

 Temporal and Event Processing recognizes and normalizes temporal expressions and
analyzes events.

 It has three major components:

 Temporal Expression Recognition

Examples:
He was born on October
2, 1938.
He was born in the
middle of the Second
Italo-Ethiopian War.
He was born two years after the Second Italo-Ethiopian War broke.
The pump circulates the water every 2 hours.
 Temporal Normalization

 Event Detection and Analysis

Department of Computer Science, SC, DDIT, DDU Applications of NLP 16/59

 Information Retrieval Co mpo nents of Information Extraction
Information N amed Entity Recognition
Extraction Machine Relation Detection and Classification
Translation Temporal and Event Processing
Question-Answering and Dialogue Template Filling
Sy st em s
Temporal and Event Processing Text Summarization

 Template Filling is the final task of information extraction systems where structured
data is to be filled in the template slots.

 Example:

Department of Computer Science, SC, DDIT, DDU Applications of NLP 17/59

 Information Retrieval Applications an d Approaches
Information Rule-Based Machine Translation
Statistical Machin e Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Applications and Approaches Text Summarization

 Machine Translation (MT) refers to a translation of texts from one natural language to
another by means of a computerized system.

 MT is one of the earliest studied applications of NLP.

 Most important applications of MT are:

 Web-based translation services

 Spoken language translation services

 Although MT for resource poor/under resourced languages is currently under

intensive research and development, it far from being a solved problem.

 Commonly used approaches to MT are:

 Rule-based (involving direct, transfer, and Interlingua) translations

 Statistical translations
 Example-based translations
 Hybrid approaches
 Deep learning approaches

Department of Computer Science, SC, DDIT, DDU Applications of NLP 18/59

 Information Retrieval Applications and Approaches
Information Rule-Based Machine Translation
Statistical Machin e Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Rule-Based Machine Translation: Text Summarization Direct Translation

 Direct (dictionary-based) translation uses a large bilingual dictionary and translates the
source language text word-by-word.

 The process of direct translation involves morphological analysis, lexical transfer, local
reordering, and morphological generation.

Abebe {3Per+M as c+S i n g } አበበ {3Per+Masc+Sing} አበበ {3Per+Masc+Sing} Aበበ {3Per+Masc+Sing}

PA S T (break) PAST(ስብር) መስኮት {Object} መስኮት[ኡ][ን]
the ኡ/ው ኡ/ው
window መስኮት PAST(ስብር) ሰበር[ኧ]
[ው]

Morphological Lexical Local Morphological

analysis re-ordering generation
transfer

Source text Target text

Abebe broke the window አበበ መስኮቱን ሰበረው

Department of Computer Science, SC, DDIT, DDU Applications of NLP 17/59

 Information Retrieval Applications and Approaches
Information Rule-Based Machine Translation
Statistical Machin e Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Rule-Based Machine Translation: Text Summarization Direct Translation

Pros and Con s

 Pros:

 Fast

 Simple

 Inexpensive

 No translation rules hidden in lexicon

 Cons:

 Unreliable

 Not powerful

 Rule proliferation

 Requires too much context

 Major restructuring after lexical substitution

Department of Computer Science, SC, DDIT, DDU Applications of NLP 20/59

 Information Retrieval Applications and Approaches
Information Rule-Based Machine Translation
Statistical Machin e Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Rule-Based Machine Translation: Text Summarization Transfer-Based Translation

 Transfer-based translation uses an intermediate representation that captures the

structure of the original text in order to generate the correct translation.

 The process of transfer-based translation involves analysis, structural transfer, and

generation.

 The structural transfer can be made at two levels:

 Superficial (syntactic) transfer

 Transfers syntactic structures between the source and target
languages.
 Suitable for translation between closely related languages
 Deep (semantic) transfer
 Transfers semantic structures between the source and target
languages.
 Used for translation between distantly related languages.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 21/59

 Information Retrieval Applications and Approaches
Information Rule-Based Machine Translation
Statistical Machin e Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Rule-Based Machine Translation: Text Summarization Transfer-Based Translation

Superficial (Syntactic) Transfer

NP VP S

N V NP NP VP
Syntactic Syntactic Syntactic
Abebe broke Det N Structure Transfer Structure N N V

the window Abebe the window broke

Word Lexical Word

Structure Transfer Structure

Source text Target text

Abebe broke the window አበበ መስኮቱን ሰበረው

Department of Computer Science, SC, DDIT, DDU Applications of NLP 22/59

 Information Retrieval Applications and Approaches
Information Rule-Based Machine Translation
Statistical Machin e Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Rule-Based Machine Translation: Text Summarization Transfer-Based Translation

Deep (Semantic) Transfer

Semantic Semantic Semantic

broke(Abebe, window) Transfer
broke(Abebe, window)
Structure Structure

Syntactic Syntactic Syntactic

Transfer
Structure Structure

Syntactic Lexical Syntactic

Transfer
Structure Structure

Source text Target text

Abebe broke the window አበበ መስኮቱን ሰበረው

Department of Computer Science, SC, DDIT, DDU Applications of NLP 23/59

 Information Retrieval Applications and Approaches
Information Rule-Based Machine Translation
Statistical Machin e Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Rule-Based Machine Translation: Text Summarization Transfer-Based Translation

Pros and Cons of Transfer-Based Translation

 Pros:

 Offers the ability to deal with more complex source language phenomena than
the direct approach.

 High quality translations can be achieved as compared to direct translation.

 Relatively fast as compared to Interlingual translation.

 Cons:

 O(N2) sets transfer rules in multilingual machine translation.

 Proliferation of language-specific rules in lexicon and syntax.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 24/59

 Information Retrieval Applications and Approaches
Information Rule-Based Machine Translation
Statistical Machin e Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Rule-Based Machine Translation: Text Summarization Interlingual Translation

 Interlingual translation uses a language-independent, 'universal', and abstract

representation of the original text in order to generate the target text.

 The process of Interlingual translation involves analysis and generation.

EVENT:
breaking TENSE:
past
AGENT: Abebe
PATIENT: window
DEFINI
Interlingua
TENESS
Analysis : Generation
definite
Source text Target text
Abebe broke the window አበበ መስኮቱን ሰበረው

 Interlingual translation is suitable for multilingual machine translation, and its main
drawback is that the definition of an Interlingua is difficult and maybe even impossible
for a wider domain.

of Computer Science, Addis Ababa University Applications of NLP 25/59

 Information Retrieval Applications and Approaches
Information Rule-Based Machine Translation
Statistical Machin e Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Rule-Based Machine Translation: Text Summarization The Vauquois Triangle

Analysis Interlingua Generation

Conceptual Analysis Conceptual Generation

Semantic Semantic Semantic
Transfer
Structure Structure
Semantic Analysis Semantic Generation
Syntactic Syntactic Syntactic
Transfer
Structure Structure
Syntactic Analysis Syntactic Generation
Word Direct Word
Structure Structure

Morphological Analysis Morphological Generation

Source text Target text

Department of Computer Science, SC, DDIT, DDU Applications of NLP 26/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation Text Summarization

 Statistical Machine Translation (SMT) finds the most probable target sentence given a
source text sentence.

 Parameters of probabilistic models are derived from the analysis of bilingual text
corpora.

Bilingual Corpus of English and Amharic (Example)

Abebe ate besso. አበበ በሶ በላ።
Abebe bought besso. አበበ በሶ ገዛ።
Abebe threw the stone. አበበ ድንጋዩን ወረወረው።
Abebe went to school. አበበ ወደ ትምህርት ቤት ሄደ።
Kebede bought a car.
ከበደ መኪና ገዛ።
Kebede bought the car.
ከበደ መኪናውን ገዛው።
Kebede bought the car.
ከበደ መኪናዋን ገዛት።
Almaz made tea.
አልማዝ ሻይ አፈላች።
Almaz made tella.
አልማዝ ጠላ ጠነሰሰች።

Department of Computer Science, SC, DDIT, DDU Applications of NLP 27/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation Text Summarization

General Architecture of SMTs

Source text

Language Model Decoding Translation Model

Target text

Department of Computer Science, SC, DDIT, DDU Applications of NLP 28/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation: Text Summarization Lan guage Model

 Language Model tries to ensure that words come in the right order.
 Some notion of grammaticality
 Given an English string e, language model assigns p(e) by formula.
 Good English string  high p(e); and bad English string  low p(e).
 Calculated with:
 A statistical grammar such as a probabilistic context free grammar; or
 An n-gram language model.

Probabilistic context free grammar [example]

Grammar Probability Lexicon

S → NP VP 0.8 Det → the | a | that | this
S → Aux NP 0.1 + 1.0 0.6 0.2 0.1 0.1
VP S → VP 0.1 Noun → book | flight | meal | money
NP → Pronoun 0.2 0.1 0.5 0.2
NP → Proper- 0.2 + 1.0 0.2
Noun
N P → Det .0.6 Verb → book |0.5
include0.2
| prefer 0.3
. . .
Nominal

Department of Computer Science, SC, DDIT, DDU Applications of NLP 29/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation: Text Summarization Lan guage Model

 N-gram models can be computed from monolingual corpus.

 Unigram probabilities
አበበ በሶ በላ።
count (w1) 4 አበበ በሶ ገዛ።
p(w1) = total words observed p(አበበ) = 29 =0.138 አበበ ድንጋዩን ወረወረው።
አበበ ወደ ትምህርት ቤት ሄደ።
 Bigram probabilities ከበደ መኪና ገዛ።
count (w1w2) p(በሶ |አበበ) = count(አበበ በሶ) ከበደ መኪናውን ገዛው።
p(w2|w1) = count (w1) count(አበበ) ከበደ መኪናዋን ገዛት።
አልማዝ ሻይ አፈላች።
= 24 = 0.500 አልማዝ ጠላ ጠነሰሰች።

 Trigram probabilities

count (w1w2w3) p(በላ|አበበ በሶ) count(አበበ በሶ በላ)

p(w3|w1w2) = = count(አበበ በሶ)
count (w1w2)

= 12 = 0.500

Department of Computer Science, SC, DDIT, DDU Applications of NLP 30/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation: Text Summarization Lan guage Model

 Similarly, higher order n-gram models can be computed.

 Problems:
 How can we deal with n-gram models if sentences are too long?
 What is p(አበበ ወደ ትምህርት ቤት ሄደ እና ድንጋዩን ወረወረው)?
 How can we deal with n-gram models if sentences are not seen in the corpus?
 What is p(ወረወረው | አበበ ወደ ትምህርት ቤት ሄደ እና ድንጋዩን)?
 Solutions:
 Smoothing: avoid zero probability
 For example, p(አበበ ወደ ትምህርት ቤት ሄደ እና ድንጋዩን ወረወረው)=0.00001
 Compute higher order n-gram models using lower order models such as bigram
and trigram.
 p(አበበ ወደ ትምህርት ቤት ሄደ እና ድንጋዩን ወረወረው) = p(አበበ|<s>) * p(ወደ|አበበ)
* p(ትምህርት|ወደ) * p(ቤት|ትምህርት) * p(ሄደ|ቤት) * p(እና|ሄደ) * p(ድንጋዩን|እና)
*p(ወረወረው|ድንጋዩን) * p(</s>|ወረወረው)

Department of Computer Science, SC, DDIT, DDU Applications of NLP 31/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation: Text Summarization Translation Model

 The job of the translation model is to assign a probability that a given source language
sentence generates target language sentence.
 We can model the translation from a source language sentence S to a target language
sentence Tˆ as:
best-translation Tˆ = argmax faithfulness(T,S) * fluency(T)
T
 Suppose that we want to build a foreign-to-English machine
translation system.
 Thus, in a probabilistic model, the best English sentence e is the
one whose probability ê = argmax p(e|f)
e
p(e|f) is the highest.
 Bayes’ rule:
p(e|f ) = p(f|e) * p(e) / p(f
)
argmax p(e|f ) = argmax p(f|e) * p(e) /
e
p(f )
e ê = argmax p(f|e) * p(e) [for a given
f] e
 Noisy channel equation: Translation model Language model

Department of Computer Science, SC, DDIT, DDU Applications of NLP 32/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation: Text Summarization Translation Model

 We are looking for the e that maximizes p(f|e) * p(e).

 We need to tell a generative “story” about how English strings become
foreign strings, i.e. we want to compute p(f|e) .
 When we see an actual foreign string f, we want to reason backwards ...
what English string e is:
 likely to be uttered; and
 likely to subsequently translate to f ?
 p(f|e) will be a module in overall foreign-to-English machine translation system.
 How do we assign values to p(f|e)?

count (f,e)
p(f |e) = count (e)
 Impossible because sentences are novel, so we would never have enough
data to find values for all sentences.
 For example:
p(Aበበ ወደ ትምህርት ቤት ሄደ እና ድንጋዩን ወረወረው|Abebe went to school and
threw the stone)=?

Department of Computer Science, SC, DDIT, DDU Applications of NLP 33/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation: Text Summarization Translation Model

 Decompose the sentences into smaller chunks, like in language modeling.

p(f|e) = p(a, f|e)

a

 The variable a represents alignments between the individual chunks in the

sentence pair where the chunks in the sentence pair can be words or phrases.
 In word-based translation, the fundamental unit of translation is a word.
 Phrase-based translations translates whole sequences of words (called blocks or
phrases), where the lengths may differ.
 Blocks are not linguistic phrases but phrases found using statistical
methods from corpora.
 Most commonly used form of translation.
 The alignment probability p(a, f|e) is defined as follows:
m

p(a, f| e) = j=1t(fj|e i) where t(fj|e i) is translation probability.

 The translation probability t(fj|ei) is calculated by counting as follows:

count (fj , ei )
t(fj | ei )
= count (ei )

Department of Computer Science, SC, DDIT, DDU Applications of NLP 34/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation: Text Summarization Translation Model

 Word translation table [example]

አበበ በሶ በላ ።

Abebe

ate

besso

 Unfortunately, it is difficult to get word aligned data to compute word translation

probability, so we can't do this directly.
 Use Expectation-Maximization (EM) algorithm.
 EM algorithm estimates model parameters by
 initiatilizing probabilities (e.g. uniformly);
 iteratively find the maximim likelihood of estimates of parameters.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 35/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation: Text Summarization Decoding

Source text
f

Language Model Decoding Translation Model

p(e) argmax p(f|e) * p(f|e)
e
p(e)

Target text
e

Department of Computer Science, SC, DDIT, DDU Applications of NLP 36/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation: Text Summarization Decoding

 A decoder searches for the best sequence of transformations that translates a source
sentence.
 Look up all translations of every source word or phrase, using word or phrase
translation table.
 Recombine the target language phrases that maximizes the translation model
probability * the language model probability.
 This search over all possible combinations can get very large so we need to find
ways of limiting the search space.
 Decoding is, therefore, a searching problem that can be reformulated as a classic
Artificial Intelligence problem, i.e. searching for the shortest path in an implicit graph.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 37/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation Text Summarization

Pros and Co ns
 Pros:
 Has a way of dealing with lexical ambiguity
 Can deal with idioms that occur in the training data
 Can be built for any language pair that has enough training data (language
independent)
 No need of language experts (requires minimal human effort)

 Cons:
 Does not explicitly deal with syntax

Department of Computer Science, SC, DDIT, DDU Applications of NLP 38/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation Text Summarization

Choosing S M T
 Economic reasons:
 Low cost
 Rapid prototyping
 Practical reasons:
 Many language pairs don't have NLP resources, but do have parallel corpora
 Quality reasons:
 Uses chunks of human translated text as its building blocks
 Produces state of the art results (when very large data sets are available)

Department of Computer Science, SC, DDIT, DDU Applications of NLP 39/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Translation
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Statistical Machine Translation Text Summarization

Materials Needed to Build S M T

 Parallel corpus
 For example, Negarit Gazette (ነጋሪት ጋዜጣ) is a useful resource for English-to-
Amharic machine translation or vice versa.

 Word alignment software

 For example, Giza++ is useful for testing the quality of automatically generated
word alignments.

 Language modeling toolkit

 For example, SRILM toolkit estimates n-gram probabilities.

 Decoder
 For example, Pharaoh (phrase-based decoder that builds phrase tables from
Giza++ word alignments and produces best translation for new input using the
phrase table plus SRILM language model)

Department of Computer Science, SC, DDIT, DDU Applications of NLP 40/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Transl at ion
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Example-Based Machine Translation Text Summarization

 Fundamental idea:
 People do not translate by doing deep linguistics analysis of a sentence.
 They translate by decomposing sentence into fragments, translating each of
those, and then composing those properly.
 Uses the principle of analogy in translation
 Example:
Given the following translations:

የመጽሃፉ ዋጋ ከ500 ብር በላይ  The price of the book is mo re than 5 0 0 Birr

ነው
የቤቱ ዋጋ ርካሽ ነው  The price of the house is cheap

Based on the above examples, the following translation can be made:

የቤቱ ዋጋ ከ500 ብር በላይ  The price of the house is more than 5 0 0 Birr
ነው

Department of Computer Science, SC, DDIT, DDU Applications of NLP 41/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Transl at ion
Extraction Machine Example-Based Machine Translation
Translation Hybrid Approaches to Machin e Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Example-Based Machine Translation Text Summarization

Challenges
 Locating similar sentences
 Aligning sub-sentential fragments
 Combining multiple fragments of example translations into a single sentence
 Determining when it is appropriate to substitute one fragment for another
 Selecting the best translation out of many candidates

Pros and Con s

 Pros:
 Uses fragments of human translations which can result in higher quality

 Cons:
 May have limited coverage depending on the size of the example database, and
flexibility of matching heuristics

Department of Computer Science, SC, DDIT, DDU Applications of NLP 42/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Transl at ion
Extraction Machine Example-Based Machine Transl ation
Translation Hybrid Approaches to Machine Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Hybrid Approaches to Machine Translation
Text Summarization

 Machine translation systems discussed so far have their own pros and cons.
 Hybrid systems take the synergy effect of rule-based, statistical and example-based
machine translations.
 Rules can be post-processed by statistics and/or examples
 Statistics guided by rules and/or examples
 Example:

If you prefer another hotel please let me know

Segment1 Segment2

Example-based translation Statistical translation Rule-based translation

Alternative translations with confidence values

Selection module

Segment1 Segment2
Translated by rule-based Translated by example-based

Department of Computer Science, SC, DDIT, DDU Applications of NLP 43/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Transl at ion
Extraction Machine Example-Based Machine Transl ation
Translation Hybrid Approaches to Machine Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Deep Learning Approaches to Machine Translation
Text Summarization

 The application of deep learning approaches for machine translation is called Neural
Machine translation.

 Neural machine translation (NMT) is an approach to machine translation that uses

an artificial neural network to predict the likelihood of a sequence of words, typically
modeling entire sentences in a single integrated model.

 NMT is not a drastic step beyond what has been traditionally done in statistical
machine translation.

 Its main departure is the use of vector representations ("embeddings", "continuous

space representations") for words and internal states. The structure of the models is
simpler than phrase-based models.

 There is no separate language model, translation model, and reordering model, but
just a single sequence model that predicts one word at a time. However, this sequence
prediction is conditioned on the entire source sentence and the entire already produced
target sequence.

 NMT: it is simple architecture and ability in capturing long dependency in the sentence,
which indicates a huge potential in becoming a new trend of the mainstream.

 NMT needs less linguistic knowledge than other approaches but can produce a
competitive performance.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 44/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Transl at ion
Extraction Machine Example-Based Machine Transl ation
Translation Hybrid Approaches to Machine Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Deep Learning Approaches to Machine Translation
Text Summarization

 Motivation of NMT: The inspiration for neural machine translation comes from two
aspects: the success of Deep Learning in other NLP tasks as we mentioned, and the
unresolved problems in the development of MT itself.

 NMT task is originally designed as an end-to-end learning task. It directly processes a

source sequence to a target sequence. The learning objective is to find the correct
target sequence given the source sequence, which can be seen as a high dimensional
classification problem that tries to map the two sentences in the semantic space.

End-to-End structure in modern NMT model.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 45/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Transl at ion
Extraction Machine Example-Based Machine Transl ation
Translation Hybrid Approaches to Machine Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Deep Learning Approaches to Machine Translation
Text Summarization

 End-to-End means the model processes source data to target data directly, without
explicable intermediate result.

 End-to-End architecture of NMT contains four components.

 Source Sentence: is an input sentence for the models from the source natural
language.
 Encoder - is used to represent the source sentence to semantic vector.
 Decoder - makes prediction from this semantic vector to a target sentence.
 Target Sentence: is an output sentence from the model for the target natural
language.

 The common deep learning algorithm used for encoding and decoding in NMT are:
 Recurrent Neural networks such as Conventional RNN, LSTM, GRU, BRNN,
BLSTM, BGRU, these models with attention mechanism and Full attention
mechanism models.

 All the above models are used to model the word sequence of the source and target
natural languages.

 The encoder model is the first model that is used by the neural network to encode a
source sentence for a second model, known as a decoder.

 The decoder model is used to predict words in the target natural language.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 46/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Transl at ion
Extraction Machine Example-Based Machine Transl ation
Translation Hybrid Approaches to Machine Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Deep Learning Approaches to Machine Translation
Text Summarization

 Recurrent neural networks face difficulties in encoding long inputs into a single vector.

 This can be compensated by an attention mechanism which allows the decoder to focus
on different parts of the input while generating each word of the output.

 The common approaches of NMT are:

 NMT with Deep Learning
 NMT with Deep Learning and Attention Mechanism
 Full Attention based NMT e.g. Transformer

 The benefits of NMT compared with SMT:

 It is the newest method of MT and is said to create much more accurate
translations than SMT.
 It is based on the model of neural networks in the human brain, with
information being sent to different “layers” to be processed before output.
 It uses deep learning techniques to teach itself to translate text based on
existing statistical models. It makes for faster translations than the statistical
method and has the ability to create higher quality output.
 It is able to use algorithms to learn linguistic rules on its own from statistical
models. The biggest benefit to NMT is its speed and quality.
 It is said by many to be the way of the future, and the process will no doubt
continue to advance in its capabilities.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 47/59

 Information Retrieval Applications and Approaches
Information Ru le-Based Machine Translatio n
Statistical Machine Transl at ion
Extraction Machine Example-Based Machine Transl ation
Translation Hybrid Approaches to Machine Translation
Question-Answering and Dialogue Deep Learning approach to Machine Translation
Sy st em s
Deep Learning Approaches to Machine Translation
Text Summarization

Architecture of Full Attention based NMT - Transformer

Department of Computer Science, SC, DDIT, DDU Applications of NLP 48/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
NL Interfaces for Human-Machine Interaction
Text Summarization

 Natural Languages (NLs) are increasingly becoming important interfaces styles in

Human-Computer Interaction (HCI).
 The growing popularity of natural language interfaces is due to the rise of human needs
to interact/communicate with computer systems to:
 get answers for real world questions; or
 make conversation in a coherent way about various topics.
 Two of the most important applications of NLP that deal with such issues are Question
Answering and Dialogue Systems.
 Question Answering (QA) System:
 A system that provides an answer or answer containing text for a given question
formulated using natural language.
 Dialogue System (DS):
 A system that converses with human beings in a coherent way.
 An extension of QA system, i.e., a two-way QA system.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 49/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
NL Interfaces for Human-Machine Interaction
Text Summarization

Question
Answer
Question Question
Answer Answer

HCI in Question Answering HCI in Dialogue Systems

Department of Computer Science, SC, DDIT, DDU Applications of NLP 50/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
Question Answering Systems: Text Summarization Questions and Answers

 QA Systems deal with a wide range of question types such as fact, list, “wh”-questions,
definition, hypothetical, and semantically-constrained questions.
 Search engines do not speak natural language.
 Human beings need to speak the language of search engines.
 QA Systems attempt to let human beings ask their questions in the normal way
using natural languages.
 QA Systems are important NLP applications especially for inexperienced users.
 QA Systems are closer to human beings than search engines are.
 QA Systems are viewed as natural language search engines.
 QA Systems are considered as next step to current search engines.
 Question answering can be approached from one of two existing NLP research areas:
 Information Retrieval: QA can be viewed as short passage retrieval.
 Information Extraction: QA can be viewed as open-domain information
extraction.
 The performance of QA Systems is heavily dependent on good search corpus.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 51/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
Question Answering Systems: Text Summarization Questions and Answers

 Answers are searched in collections from

 Database (e.g. documents in an organization):
 relies on small local document collections
 limited type of questions are allowed
(e.g. factoid questions)
 key words are used to represent the
question
 only shallow analysis is made to compile
answers
 closed-domain question answering
 Corpus data (e.g. web):
 relies on world knowledge
 entertains any type of question
 deep linguistic analysis is required (e.g. named entity recognition, relation
detection, co-reference resolution, word sense disambiguation, etc.
 open-domain question answering

Department of Computer Science, SC, DDIT, DDU Applications of NLP 52/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
Question Answering Systems: Text Summarization General Architecture

Major Components of QA Sy st e ms
 Question Analysis: The natural language question input by the user needs to be
analyzed into whatever form or forms are needed by subsequent parts of the system.
 The user could be asked to clarify his or her question before proceeding.
 Candidate Document Selection: A subset of documents from the total document
collection (typically several orders of magnitude smaller) is selected, comprising those
documents deemed most likely to contain an answer to the question.
 This collection may need to be processed before querying, in order to transform
it into a form which is appropriate for real-time question answering.
 Candidate Document Analysis: If the preprocessing stage has only superficially
analyzed the documents in the document collection, then additional detailed analysis of
the candidates selected at the preceding stage may be carried out.
 Answer Extraction: Using the appropriate representation of the question and of each
candidate document, candidate answers are extracted from the documents and ranked
in terms of probable correctness.
 Response Generation: A response is returned to the user.
 This may be affected by the clarification request, and may in turn lead to the
response being updated.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 53/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
Question Answering Systems: Text Summarization General Architecture

User
Question Response

Clarification Request

Question Analysis Response Generation

Question Representation Ranked Answers

Candidate Document Candidate Candidate Document Analyzed

Documents Answer Extraction
Selection Documents Analysis

Document
Collection

Department of Computer Science, SC, DDIT, DDU Applications of NLP 54/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
Question Answering Systems: Text Summarization Deep Learning Approach

Deep Learning approaches to Question and Answer system

 QA system developed using traditional approach used conventional linguistically-based NLP
techniques, such as parsing, part-of-speech tagging and coreference resolution.
 However, with recent developments in deep learning, neural network models have shown
promise for QA. Although these systems generally involve a smaller learning pipeline, they
require a significant amount of training.
 Deep learning based question and Answer systems uses :

 GRU and LSTM units allow recurrent neural networks to handle the longer texts
required for QA.
 Further improvements – such as attention mechanisms and memory networks –
allow the network to focus on the most relevant facts. Such networks provide the
current state-of-the-art performance for deep-learning-based QA.
 Common models developed for question and answer using deep learning are:
 Sequence-to-sequence model:
 Dynamic memory networks
 End-to-end memory networks

Department of Computer Science, SC, DDIT, DDU Applications of NLP 55/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
Dialogue Systems: Modality of Conversation
Text Summarization

 The modality of Dialogue Systems can be text-based, spoken-dialogue, graphical user

interface, or multi-modal.
 Text-Based:
 The conversation is made by making use of natural language texts.
 For example, ELIZA.
 Spoken Dialogue:
 The conversation is made by making use of voice.
 For example, HMIHY (how may I help you) developed at AT&T for call routing.
 Graphical User Interface:
 The conversation is made by making use of images.
 For example, Dialogue Boxes in Windows applications.
 Multimodal:
 The conversation is made by any combination of the above three modalities.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 56/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
Dialogue Systems: Dialogue Interation M ode s
Text Summarization

 Dialogue Systems differ in the degree with which human or computer takes the
initiative.

Question
Answer
Question
Answer

Computer-Initiative Human-Initiative
 Computer maintains tight control  Human maintains tight control
 Human is highly restricted  Computer is highly restricted
 E.g., Dialogue Boxes  E.g., ELIZA
Mixed-Initiative
 Human and computer have flexibility to specify constraints
 Mainly research prototypes

Department of Computer Science, SC, DDIT, DDU Applications of NLP 57/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
Dialogue Systems: Application Areas
Text Summarization

 Currently, Dialogue Systems are used in specific domains such as:

 Customer service: Responding to customers' general questions about products
and services, e.g., answering questions about applying for a bank loan.
 Help desk: Responding to internal employee questions, e.g., responding to
human resource questions.
 Website navigation: Guiding customers to relevant portions of complex
websites, e.g., helping people determine where information or services reside
on a company's website.
 Guided selling: Providing answers and guidance in the sales process,
particularly for complex products being sold to novice customers.
 Technical support: Responding to technical problems, such as diagnosing a
problem with a device.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 58/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
Dialogue Systems: General Architecture
Text Summarization

General Architecture of Spoken Dialogue System

Human Computer System

Input Speech Natural Language

Interface Recognition Understanding

I/O
Server Dialogue
Knowledge
Base
Manager

Output Text-to-Speech Natural Language

Interface Synthesis Generation

These components do not exist in Text-Based Dialogue Systems.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 59/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
Dialogue Systems: General Architecture
Text Summarization

 In the process of Natural Language Understanding, there are many ways to represent
the meaning of sentences.
 For dialogue systems, the most common is “frame and slot semantics”
representation.

Show me morning flights from Addis Ababa to London on Tuesday [Example]

SHOW:
FLIGHTS:
ORIGIN
CITY: Addis Ababa
DATE: Tuesday
TIME: morning
DESTINATION
CITY: London
DATE:
TIME:

Department of Computer Science, SC, DDIT, DDU Applications of NLP 60/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
Dialogue Systems: General Architecture
Text Summarization

 “Frame and slot semantics” can be generated by a semantic grammar.

Semantic grammar for the aforementioned request

LIST show me | I want | can I see

DEPARTTIME (after|before|around) HOUR | morning | evening
HOUR  one | . . . | twelve | (am|pm)
FLIGHTS  a (flight) — flights
ORIGIN  from CITY
DESTINATION  to CITY
CITY  Addis Ababa | Dire
Dawa | Bahir Dar | London |
Paris | New York

Department of Computer Science, SC, DDIT, DDU Applications of NLP 61/59

 Information Retrieval
Information NL Interfaces for Human-Machine Interaction
Extraction Machine Question Answering Sy st em s
Translation Dialogue Sy st ems
Question-Answering and Dialogue
Sy st em s
Dialogue Systems: Using Deep Learning
Text Summarization

 The General Architecture of Spoken Dialogue System contain different components

including:
 Speech Recognition
 Natural Language Understanding

 Domain Identification
 User Intent Detection
 Slot Filling
 Dialogue Manager

 Dialogue State Tracking (DST)

 Dialogue Policy Optimization
 Natural Language Generation
 Text-to-Speech Synthesis

 Deep Learning algorithms are applied to all these components of the Dialogue System,
and achieved a state of the art result.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 62/59

 Information Retrieval
Information
Genres and Types of Text Summarization
Extraction Machine
Computational Approach
Translation
Question-Answering and Dialogue Deep Learning Approach to Text Summarization
Sy st em s
Genres and Types of Text SummarizationText Summarization

 Automatic Text Summarization refers to the generation of a shortened version of a text

that still contains the most important points of the original text.
 The content of the summary depends on the purpose and/or type of summarization.
 Types of summaries can be the following:
 Indicative vs. informative
 Used for quick categorization vs. content processing.

 Extract vs. abstract

 Lists fragments of text vs. re-phrases content coherently.

 Generic vs. query-oriented

 Provides author’s view vs. reflects user’s interest.

 Background vs. just-the-news

 Assumes reader’s prior knowledge is poor vs. up-to-date.

 Single-document vs. multi-document source

 Based on one text vs. fuses together many texts.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 63/59

 Information Retrieval
Information
Genres and Types of Text Summarization
Extraction Machine
Comput ational Appro ach
Translation
Question-Answering and Dialogue Deep Learning Approach to Text Summarization
Sy st em s
Computational Approach: Procedures
Text Summarization

 The following general procedure used to build Text Summarization system:

 Given a corpus of documents and their summaries

 Label each sentence in the document as summary-worthy
or not compute document keywords
score document sentences with respect to these keywords
 Learn which sentences are likely to be included in a summary
 Given an unseen (test document) classify sentences as
summary-worthy or not
Cohesion and coherence check: Spot anaphoric references and modify text accordingly
Balance and coverage: modify summary to have an appropriate text structure

 There are two computational approaches to Text Summarization:

 top-down; and
 bottom-up

Department of Computer Science, SC, DDIT, DDU Applications of NLP 64/59

 Information Retrieval
Information
Genres and Types of Text Summarization
Extraction Machine
Comput ational Appro ach
Translation
Question-Answering and Dialogue Deep Learning Approach to Text Summarization
Sy st em s
Computational Approach: Top-Down
Text Summarization

 Top-down approach is considered as query-driven summarization.

 User needs only certain types of information.
 System needs particular criteria of interest, used to focus search.
 Criteria of interest can be modeled using:
 Templates with slots having semantic characteristics; or
 A set of important terms.
 Top-down approach can be implemented using Information Extraction (IE).
 IE task: Given a template and a text, find all the information relevant to each
slot of the template and fill it in.
 IE-for-summarization task: Given a query, select the best template, fill it in,
and generate the contents.
 Problems:
 IE works only for very particular templates; can it scale up?
 What about information that doesn’t fit into any template?
 Pros: higher quality, and also supports abstracting.
 Cons: low speed, and still needs to scale up to robust open-domain summarization.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 65/59

 Information Retrieval
Information
Genres and Types of Text Summarization
Extraction Machine
Comput ational Appro ach
Translation
Question-Answering and Dialogue Deep Learning Approach to Text Summarization
Sy st em s
Computational Approach: Bottom-Up
Text Summarization

 Bottom-up approach is considered as text-driven summarization.

 User needs any information that is important.
 System uses strategies (importance metrics) over representation of whole text.
 Generic importance metrics can be modeled using:
 Degree of connectedness in semantic graphs; or
 Frequency of occurrence of tokens.
 Bottom-up approach can be implemented using Information Retrieval (IR).
 IR task: Given a query, find the relevant document(s) from a large set of
documents.
 IR-for-summarization task: Given a query, find the relevant passage(s)
from a
set of passages (i.e., from one or more
documents).
 Problems:
 IR techniques work on large volumes of data; can they scale down?
 IR works on words; do abstracts require abstract representations?
 Cons: lower quality, and inability to manipulate information at abstract levels.
 Pros: robust, and good for query-oriented summaries.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 66/59

 Information Retrieval
Information
Genres and Types of Text Summarization
Extraction Machine
Comput ational Appro ach
Translation
Question-Answering and Dialogue Deep Learning Approach to Text Summarization
Sy st em s
Computational Approach: Evaluation
Text Summarization

 Given the original text T and summary S, two measures are commonly used to
evaluate Text Summarization systems:

 Compression Ratio (CR)

Length (S)
CR =
Length (T)

 Retention Ratio (RR)

Information (S)
RR =
Information (T)
 Measuring length:
 Number of letters
 Number of words
 Number of sentences
 Measuring information:
Question Game: quantify
 Shannon test reader’s understanding.
information content.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 67/59

 Information Retrieval
Information
Genres and Types of Text Summarization
Extraction Machine
Computational Approach
Translation
Question-Answering and Dialogue Deep Learning Approach t o Text Summarization
Sy st em s
Deep Learning Approach to Text Summarization
Text Summarization

 Text Summarization is a sequence to sequence problem.

 Sequence-to-sequence learning is a training model that can convert sequences of one
input domain into the sequences of another output domain.
 It is generally used when the input and output of a model can be of variable lengths.

 Variants of Recurrent Neural Networks (RNNs), i.e. Gated Recurrent Neural Network (GRU)
or Long Short Term Memory (LSTM), are preferred as the encoder and decoder
components. This is because they are capable of capturing long term dependencies by
overcoming the problem of vanishing gradient.

 The general architecture of the end-to-end for text summarization is as follow.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 68/59

 Information Retrieval
Information
Genres and Types of Text Summarization
Extraction Machine
Computational Approach
Translation
Question-Answering and Dialogue Deep Learning Approach t o Text Summarization
Sy st em s
Deep Learning Approach to Text Summarization: Architecture
Text Summarization

Department of Computer Science, SC, DDIT, DDU Applications of NLP 69/59

 Information Retrieval
Information
Genres and Types of Text Summarization
Extraction Machine
Computational Approach
Translation
Question-Answering and Dialogue Deep Learning Approach t o Text Summarization
Sy st em s
Deep Learning Approach to Text Summarization
Text Summarization

 End-to-end or Sequence-to-sequence model with an attention mechanism consists of

encoder, decoder and an attention layer.

 Word embeddings are a type of word representation that allows words with similar
meaning to have a similar representation.

 Attention mechanism is used to secure individual parts of the input which are more
important at that particular time.
 It can be implemented by taking inputs from each time steps and giving
weightage to time steps.
 The weightage depends on the contextual importance of that particular time
step.
 It helps pay attention to the most relevant parts of the input data sequence so
that the decoder can optimally generate the next word in the output sequence.

Department of Computer Science, SC, DDIT, DDU Applications of NLP 70/59

 Information Retrieval
Information
Genres and Types of Text Summarization
Extraction Machine
Computational Approach
Translation
Question-Answering and Dialogue Deep Learning Approach t o Text Summarization
Sy st em s
Deep Learning Approach to Text Summarization: Evaluation
Text Summarization

 Precision = Number of important sentences/Total number of sentences summarized.

 Recall = Total number of important sentences Retrieved / Total number of important

sentences present.

 BLEU - measures precision - how much the words (and/or n-grams) in the machine
generated summaries appeared in the human reference summaries.

 ROUGE - measures recall - how much the words (and/or n-grams) in the human
reference summaries appeared in the machine generated summaries.


F1-score - F1 = 2 * (Bleu * Rouge) / (Bleu + Rouge)

Department of Computer Science, SC, DDIT, DDU Applications of NLP 71/59

TOC: Course Syllabus

Previous: Approaches to NLP

Current: Applications of NLP-Part-I

Next:
Applications of NLP-Part- II