基于知识图谱的问答系统关键技术

肖仰华（Yanghua Xiao）
复旦大学知识工场实验室(Kw.fudan.edu.cn)
上海数眼科技发展有限公司(shuyantech.com)

kw.fudan.edu.cn/qa
Outline
• KBQA background
• 不倒翁问答系统
• Template based KBQA
• Conclusion
Backgrounds
• Question Answering (QA) systems answer natural language
questions.
IBM Watson Google Now Apple Siri Amazon Alexa Microsof Cortana

kw.fudan.edu.cn/qa
Why QA
• QA application:
• One of the most natural human-computer
interaction
• Key components of Chatbot, which attracts
wide research interests from industries

• QA for AI:
• One of most important tasks to evaluate the
machine intelligence: Turing test
• Important testbed of many AI techniques,
such as machine learning, natural
language processing, machine cognition Turing test
kw.fudan.edu.cn/qa
Why KBQA?
More and More Knowledge bases are created
• Google Knowledge graph, Yago，WordNet, FreeBase, Probase, NELL, CYC, DBPedia
• Large scale and high quality

A piece of knowledge base, which consist of

The boost of knowledge bases triples such as (d, population, 390k)
kw.fudan.edu.cn/qa
实用化知识问答-机遇与挑战
• 机遇：从人工智障到人工智能 准确
• 大量的问答模型研究 性
• 大量的知识库与语料
复杂 召回
• 挑战 问答 率
• 如何构建一个实用化问答系统
挑战

泛化 健壮
为什么现在人工智能助理都像人工智障？
能力 性
2017十大“人工智障”事件 科技巨头无一幸免
从人工智能变成“人工智障”，聊天机器人殇在哪？ 样本
稀缺
Outline
• KBQA background
• 不倒翁问答系统
• Template based KBQA
• Conclusion
总览

• “不倒翁“知识问答DEMO
• http://218.193.131.250:20013/ 输入问题

• http://shuyantech.com/qa

支持问题例子
• 知识问答
• 知识图谱、深度学习、规则系统

答案

额外信息
“不倒翁”知识问答系统架构
元数据
Mention2Entity Synonymous Property mentions
Entity Types Type Properties Domain-ranges for
Taxonomy Question Types Properties

Entity
Q
Linking

专家系统 问答模型
枚举 Deep QA 知识图谱
（CN-DBpedia）
常识 IRQA
是否 Matching based 互联网语料
递归 QA （文本、问答对）
Template based
比较
QA
单知识点
A Table QA
类型检查 Commonsense 常识问答库
QA
模块列表
• Entity Linking
• 实体识别、消歧、链接到知识库
• 专家系统
• 基于复杂规则解决递归、比较等复杂问题
• 决定子模块的选择性调用
• 复杂问题存在规律性且缺少训练数据，规则系统是较优选择
• 元数据
• 提供问答系统所需要的语言知识、词汇知识、本体等元数据
模块列表
• Matching based QA • Template based QA
• 基于深度学习对问题库进行模糊匹配 • 基于taxonomy、问答语料对训练生成
• 解决常识类问答 概念语义模板
• e.g. 为什么天空是蓝色的 • 利用概念语义模板对问题进行理解，
完成回答
• Deep QA • 适用于对于新实体的问答
• 基于深度学习将问句匹配到知识库中 • Eg. 华为P20多大尺寸？
的某条具体知识
• 解决单条事实类问答 • Table/list QA
• e.g. 七里香是谁唱的 • 基于table、list回答枚举类问题
• Eg，复旦有哪些院士？
• IRQA
• 到互联网上搜索，并使用深度机器阅
读理解模型获取答案 • Commonsense QA
• 获取到的答案可以用于补全知识库 • 回答常识问答类问题
• 解决知识库缺失问题 • Eg，天空为什么是蓝色的？
• e.g. 鲁迅的身高是多少
Mention2Entity库
• 形式： 字符串（mention）à 实体（entity）或 实体列表 （entity list）
• 示例
• “复旦”à“复旦大学”
• “周董”à“周杰伦”
• “雨神” à [“萧敬腾（华语流行男歌手）”, “雨神（中国神仙）”]

• 作用：提高QA系统识别实体的能力
等价属性库
• 形式：属性 à 其他等价属性列表
• 示例
• “妻子”à [“夫人”，“老婆”，“伉俪”，“爱人”，“配偶”]
• “出生日期”à [“生日”，“出生年月”，“出生时间”，“诞辰”]
• “主演”à [“主要演员”，“影片主演”，“领衔主演”]

• 提高QA系统识别属性的能力
属性元数据库
• 形式：属性 à domain à range
• 示例
• “妻子”à 人物 à 人物
• “出生日期”à 人物 à 日期
• “主演”à 电影 à 演员
• 提高QA系统的准确率

人物 人物 电影

人物
日期 演员
针对QA的实体链接 Short text

Online

• 段解析 Segment Resolution

• 检测出所有的mention Mention Detection

• 在有限的上下文进行局部链接 Local Disambiguation

• 利用实体概念作为细粒度的主题
• 用词向量相似度来解决稀疏文本的问题 Entity Recognition

• 实体识别 Text Segmentation

Entity Recognition
• 利用局部链接的分数进行语义文本划分
• 根据段的语法语义信息进行实体识别
Subject Entity
• 主体实体选择 Selection

• 选择作为询问主体的实体
• 实体关系 & 实体流行度 Linked Entities
Running example

• Input: text sequence: 刘若英语怎么样

• Segment Resolution
• Detect all possible mention: “刘若”、“刘若英”、“英语”
• Candidate entity generation: 刘若1KB、刘若2KB 、刘若英KB、英语KB…
• Local disambiguation: φ(刘若,刘若1KB)=0.4、φ(刘若,刘若2KB)=0.3…
• Entity Recognition φ 𝑚, 𝑒 = ε · 𝑠𝑖𝑚* 𝑚, 𝑒 + (1 − ε) · 𝑠𝑖𝑚0 𝑚, 𝑒 Topic coherence: 𝑠𝑖𝑚* 𝑚, 𝑒
Textual similarity: 𝑠𝑖𝑚0 𝑚, 𝑒
• Text segmentation: 刘若|英语|怎么样 ?

• Entity recognition: 刘若、英语 𝑊 = 𝑎𝑟𝑔𝑚𝑎𝑥 𝑂 𝑊 = log 𝑃 𝑊 = < 𝑃 𝑤>

>@A
•Global linking: rel(刘若,刘若1KB)=0.01、rel(刘若,刘若2KB)=0.2 Calculate from KB
• Output: mention and entity {(刘若, 刘若2KB), (英语, 英语KB)}
 结果
• Entity linking for QA 结果：
• 准确率：90%，召回率: 94.3% +
• F1: 92.0% +
ort Text Entity Linking with Fine-grained Topics CIKM2018, October 2018, Lingotto, Turin, Italy

ust local approach is dominant. Our approach outperforms others our dataset. But there are still some types of entities in our dataset
both datasets. do李娜（中国女子网球名将）：人物、体育人物、运动员、名将
not belong to named entities. Therefore, the recall score of the
NER method tends to be low, while the precision score is confident.
3 Effectiveness of Text Segmentation Our李娜（流行歌手、佛门女弟子）：人物、演员、歌手、弟子
method has the best performance in both precision and recall.
test the text segmentation performance of our approach on the
asets published in [8]. We choose the two simplified Chinese Table 4: Results of entity recognition task
mentation datasets, MSR (Microsoft Research) and PKU (Peking
Methods Prec. Reca. F1
iversity), as our evaluation dataset. We compare our approach
Stanford NER [28] 58.7 39.5 47.2
h several popular Chinese segmentation tools, including:
Baidu Entity Annotation [2] 65.5 78.2 71.3
• jieba [23]. It is a Chinese segmentation tool based on a prefix
Our Method 91.0 89.4 90.2
dictionary structure to achieve efficient word graph scanning.
It uses dynamic programming to find the most probable com-
Entity Linking已经广泛应用在新闻、出版等领域 中文通用Entity Linking的准确率
bination based on the word frequency. For unknown words, a
6 RELATED WORK
Deep QA
• 目标：
• 单知识点、且三元组知识库包含答案的问题

Deep Deep
Predicate Pattern Deep QA
Matching Matching

解决训练数据充足的问法和属性之间的 Few-Shot Learning：解决训练数据稀少的问法

匹配问题，具有强大的错误容忍能力 和属性之间的匹配问题，能处理垂直领域属性
和特殊问法，无需训练即可添加规则
专家系统
• 复杂问题存在规律性且缺少训练数据，规则系统是较优选择
• 目标：
• 基于复杂规则决定子模块的选择和组合调用策略
• 解决递归、比较等复杂问题

• 规则配置：

条 动 规
件 作 则
测试
• NLPCC-2016 KBQA，Testset，
1000 QA-pairs
• 91.6% Accuracy
• SougouQA Factoid Reading
Comprehension Task，验证集
• 73.3% Exact Match
• IRQA Hand-made Testset
• Exact Match 83.3%
• Acceptable 90.0%

* 由于搜索引擎返回结果可能改变，IRQA的返回结果也可能会有不同
Outline
• KBQA background
• 不倒翁问答系统
• Template based KBQA
• Conclusion
Our approach
• Representation: concept based templates.
• Questions are asking about entities. The semantic of the question is reflected by its corresponding
concept.
• Advantage: Interpretable, user-controllable
• Learn templates from QA corpus, instead of manfully construction.
• 27 million templates, 2782 intents
• Understand diverse questions

How many people are there in $City?

Conceptualization
By Probase
How many people are there in Shanghai? How many people are there in Beijing?

Learn from
QA Corpora and KB
Population Population
Shanghai 2420万 Beijing 2172万
kw.fudan.edu.cn/qa
ways map to the predicate population, no matter what specific an entity may belong to multiple categories. We obtain concept
$city it is. We can conclude that for certain probability the tem- distribution of e through context-aware conceptualization [32]. For

System Architecture
plate maps to population. Learning templates that map to a com-
plex knowledge base structure employs a similar process. The only
difference is that we find “expanded predicates” that correspond to
example, question q1 in Table 3 contains entity a in Figure 1. Since
a belongs to two categories: $Person, $Politician, we can derive
two templates from the question: When was $Person born?
a path consisting of multiple edges which lead from an entity to a and When was $Politician born?.
certain value (e.g., marriage ! person ! name).
• Offline
1.4
procedure
Paper Organization
The•rest
Learn theis mapping
of the paper fromIn templates
organized as follows. Sec 2, we give anto
overviewpredicates:
of KBQA. The major P (p|t),
contribution of this paper is learn-
ing templates from QA corpora. All technique parts are close-
• Input:
ly related to it. Secqa3 shows
corpora, large
the online scale
question taxonomy,
answering with KB
templates. Sec 4 elaborates the predicates inference for templates,
which•is Output:
the key step toP(P|T)
use templates. Sec 5 extends our solution to
answer a complex question. Sec 6 extends the ability of templates
to infer complex predicates. We present experimental studies in
• Online procedure
Sec 7, discuss more related works in Sec 8, and conclude in Sec 9.
2. SYSTEM
• Parsing,OVERVIEW
predicate inference and answer
In this section, we introduce some background knowledge and
retrieval
give an overview of KBQA. In Table 2, we list the notations used
in this•paper.
Input: binary factoid questions (BFQs)
Table 2: Notations
• Output:
NotationDescriptionanswersNotation
in KGDescription Figure 3: System Overview
q question s subject
a answer p predicate System Architecture. Figure 3 shows the pipeline of our QA sys-
QA QA corpus o object tem, which consists of two major procedures:
e entity K knowledge base
v value c category • Online procedure: When a question comes in, we first parse
 wo predicates in Freebase: the “placegeneration
of birth”procedure
and “place inlived
Example 1,for and
(a).shows
And the depend
we then ch
”. In DBpedia, who founded cy of these random variables in
$organization? re-Figure 4. Based
base on the
for (b). genera
If there ar
Problem Model
predicates f ounder and f ather. model, we compute P (q, e, t, p, v) in Eq them(2).uniform
Now Problem
probabil
reduced to Eq (3).
We optimize the com
LEM D•EFINITION 1. Given
Given a question aPquestion
q, our goal
(q, e, is toq,v)
t, p, our
find
=anPgoal is(e|q)P
answer
(q)P tov with maximal
byq)P
(t|e, answer. Asp)illus
q’s(p|t)p(v|e,
nswer v probability
with maximal (v isprobability
a simple value)
(v is a simple value): of entity-value pairs E
X values are from the an
arg max P (V = v|Q = q) arg max (1) P (v|q, e, t, p)
v v
e,t,p
equal probability to be
ustrate how a value is found for a given question, we e: entity; t: template; p: predicate
• Basic idea：We
d a generative proposed
model. Starting froma generative model toq,explain how a value isP (e|qi )
the user question
found for a given question,
generate/identify its entity e according to the distribution
After knowing the question and the entity, we generate the ,where [.] is the Iverson
• Rationality
t according to the of probabilistic
distribution inference
P (t|q, e). The predicate is more accurate than d
• uncertainty (e.g. some questions’ intents are vague) Template distribut
epends on• t,Incompleteness
which enables us to infer the predicate p by
(e.g. the knowledge base is almost always incomplete),
Finally, given the(e.g.
• noisy entity e and
answers in the predicate
the QA p, webegenerate
corpus could wrong) When was $perso
wer value v by P (v|e, p). v can be directly returned or em- with the mention of an
n a natural language sentence as the answer a. We illustrate the category of the ent
question2answer: a generative
process
• A qa pair
• Q: How many people live in Honolulu?
• A: It’s 390K.

kw.fudan.edu.cn/qa
 question2answer: entity linking

How many people live in Honolulu?

kw.fudan.edu.cn/qa
 question2answer:
conceptualization

How many people live in Honolulu?

How many people live in $city?

kw.fudan.edu.cn/qa
 question2answer: predicate
inference

How many people live in Honolulu?

How many people live in $city? population

kw.fudan.edu.cn/qa
 question2answer: value lookup

390K

How many people live in Honolulu?

How many people live in $city? population

kw.fudan.edu.cn/qa
osed a generative model. Starting from the user question q,
rst generate/identify its entity e according to the distribution
Probabilistic graph model
q). After knowing the question and the entity, we generate the ,where [.] is the Iverso
is more accurate than
late t according to the distribution P (t|q, e). The predicate
ly depends on t, which enables us to infer the predicate p by Template distribu
t). Finally, given the entity e and the predicate p, we generate When was $perso
nswer value v by P (v|e, p). v can be directly returned or em- with the390K
mention of a
ed in a natural language sentence as the answer a. We illustrate the category of the en
How many people live in Honolulu? Let t = t(q, e, c) in
eneration procedure in Example 1, and shows the dependen-
these random variables in Figure 4. Based on the generative entity e in q by e’s cat
el, we compute P (q, e, t, p, v) in Eq (2).HowNow
many people live in $city?
Problem 1 is population

ced to Eq (3).
, where P (c|q, e) is th
P (q, e, t, p, v) = P (q)P (e|q)P (t|e, q)P (p|t)p(v|e, p) (2) our work, we directly
X to compute P (c|q, e).
arg max P (v|q, e, t, p) (3) Value (answer) di
v
e,t,p predicate p of e, it is
ing up the knowledge
kw.fudan.edu.cn/qa
Probability Computation
• Source
• QA corpora (42M Yahoo! Answers)
• Knowledge base such as Freebase
• Probase(a large scale taxonomy)

• Directly estimated from data

• Entity distribution P (e|q)
• Template distribution P(t|q,e)
• Value (answer) distribution P(v|e,p)

Question Answer
When was Barack Obama born? The politician was born in 1961.
When was Barack Obama born? He was born in 1961.
How many people are there in Honolulu? It’s 390K.
Yahoo! Answers QA pairs
kw.fudan.edu.cn/qa
P(P|T) estimation
• We treat P(P|T) as parameters, and learn the parameter using
maximum likelihood estimator, maximizing the likelihood of
observing QA corpora
• An EM algorithm is used for parameter estimation

kw.fudan.edu.cn/qa
 until we meet a BFQ or the substring is a single word. However, as
shown in Example 3, many question decompositions are not valid de
Answering
(answerable). Hence,complex questions
we need to measure how likely a decomposi-
tion sequence is answerable. More formally, let A(q) be the set of
Co
all possible
• When decompositions
was Barack Obama’s wifeof q.born?
For a decomposition A 2 A(q),
de
let
• (Who be the Obama’s
is) Barack
P (A) probability that A is a valid (answerable) question
wife?
• When wasOutMichelle Obama born? cas
sequence. problem thus is reduced to
• How to decompose the question into a series of binary questions? (2)
arg max P (A) (25)
ma
A2A(q) aq
• A binary question sequence is meaningful, only if each of the binary qu
question
Next, is
wemeaningful.
elaborate the estimation of P (A) and how we solve the Th
optimization problem efficiently in Sec 5.2 and 5.3, respectively.

• A5.2 Metric
dynamic programming (DP) algorithm is employed to find the wh
optimal
The decomposition.
basic intuition is that A = (q̌i )ki=0 is a valid question se- pri
quence if each individual question q̌i is valid. Hence, we first es-
kw.fudan.edu.cn/qa (q
 Experiments
We list 5 expanded predicates we KBQA learned in Table 18. We found
Bootstrapping
that all these expanded predicates found
Corpus 41M QAbypairs
KBQA are meaningful.
256M sentences
We further choose one expanded predicate,
Templates 27,126,355marriage ! person471,920
! name, to see whether the templates learned for this predicate
Predicates 2782 283
are correct or meaningful. We list five learned templates in Ta-
Templates
ble 17. per predicate
These templates in general 9751
are reasonable. 4639
KBQA finds significantly more templates and predicates than its competitors despite that the corpus
Tablesize
16:of bootstrapping
Effectivenessis of Predi- Table 17:
larger. Templates for
cate Expansion marriage ! person ! name
Length #Template #Predicate Who is $person marry to?
1 467,393 246 Who is $person’s husband?
2 to k 26,658,962 2536 What is $person’s wife’s name?
Ratio 57.0 10.3 Who is the husband of $person?
Who is marry to $person?
Table 18: Examples ofConcept
Expanded
based Predicates
templates are meaningful
Expanded predicate Semantic
 Experiments

Results over QALD-5. The results verify the effectiveness of KBQA over BFQs.
Experiments
Hybrid systems
• First KBQA
• If KBQA gives no reply, then baseline systems.

Results of hybrid systems on QALD-3 over DBpedia. The results verify the
effectiveness of KBQA for a dataset that the BFQ is not a majority.
Outline
• KBQA background
• 不倒翁问答系统
• Template based KBQA
• Conclusion
Conclusion
• 样本增强可以显著提升问答模型的健壮性
• 基于阅读理解的IRQA可以显著提升召回率
• Pattern+DL可以显著提升问答模型的泛化能力
• 规则系统可以提升问答系统的复杂问题的回答能力
• 支撑性数据是确保问答系准确性的前提

• 如何实现一个实用化的知识问答系统？
• 实用化问答系统的关键技术研究十分缺乏
• 如何有效评测一个知识问答系统？
• 有效的知识问答评测数据集、评测指标的研究仍然十分缺乏
Reference
• Wanyun Cui, et al., KBQA: Learning Question Answering over QA
Corpora and Knowledge Bases, (VLDB 2017)
• Wanyun Cui, et al., KBQA: An Online Template Based Question
Answering System over Freebase, (IJCAI 2016), demo
• Lihan Chen, et al, Entity Linking for Short Text, Technique report of
KW.
• Jiaqing Chen, et al, Incorporating complicated rules in deep
generative network. Technique report of KW
 Thank you!

DEMO
http://218.193.131.250:20013/
http://shuyantech.com/qa