Knowledge Graphs Srihari

Probabilistic Knowledge Graphs

Sargur N. Srihari
[email protected]

1
Knowledge Graphs Srihari

Topics
• Knowledge Graphs (KGs)
• Statistical Relation Learning (SRL) for KGs
• Latent Feature Models
– RESCAL, ER-MLP, Latent distance
– Training SRL
• Markov Random Fields from KGs
• References

2
Knowledge Graphs Srihari

Google Knowledge Graph (KG)

Knowledge panels
information boxes that appear when you search for entities
(people, places, organizations, things)
that are in the Knowledge Graph

They are meant to help get a quick snapshot of information

on a topic based on available content on the web.

Google’s KG, stores 18 billion facts about 570 million entities,

For its search engine

KGs provide semantically structured information interpretable

by computers
an important ingredient to build more intelligent machines
3
Knowledge Graphs Srihari

Knowledge Graphs use Relations

• Binary Relation
– Set of ordered pairs of objects
• Pairs in set have the relation while those not in set do not
• Relation in the context of AI
– The relation plays the role of a verb, while two
arguments to the relation play the role of its subject
and object
– These sentences take the form of a triplet of tokens
(subject, verb, object) or (subject, predicate, object)
• with values (entityi, relationj, entityk)
4
Knowledge Graphs Srihari

Knowledge Graph Example

• Fact: Leonard Nimoy was an actor who played the character Spock
in the science-fiction movie Star Trek
SPO triples Subject and Object are entities and
Predicate is the relation between them
Subject Predicate Object
(LeonardNimoy, profession, Actor) [RDF standard: represent facts in the
(LeonardNimoy, starredIn, StarTrek)
form of binary relationships]
(LeonardNimoy, played, Spock)
(Spock, characterIn, StarTrek)
(StarTrek, genre, ScienceFiction)
Combine SPO triples to form a multi-graph:
Knowledge Graph Nodes represent entities
(all subjects and objects)
Fiction Directed edges represent relationships.

Edge direction indicates whether entities

occur as subjects or objects,
(edge points from subject to object)

Different relations are represented via

different types of edges (also called edge labels)
Knowledge Graphs Srihari

Expanded Knowledge Graph

Nodes represent entities

Edge labels represent types of relations
Edges represent existing relationships

In addition to being a collection of facts, knowledge graphs provide type hierarchies

(Leonard Nimoy is an actor,which is a person, which is a living thing)
and type constraints (e.g., a person can only marry another person, not a thing) 6
Knowledge Graphs Srihari

Open vs Closed-world assumption

• In closed world assumption (CWA)
– a non-existing triple is a false relationship
• No starredIn edge from Leonard Nimoy to Star Wars means
that Nimoy did not star in this movie
• In open world assumption (OWA)
– a non-existing triple is interpreted as unknown
– Corresponding relationship is either true or false
• Missing edge does not mean that Nimoy did not star in
Star Wars
• RDF and the Semantic Web make the OWA
• CWA often used for training relational models 7
Knowledge Graphs Srihari

Definition of Attribute
• We can also define an attribute, a concept
analogous to a relation, but taking only one
argument:
(entity i , attribute j)
• For example, we could define the has_fur
attribute, and apply it to entities like dog

8
Knowledge Graphs Srihari

Knowledge Graph with attributes

Part of a semantic KG
representing relationships
between the entities
Roberto Benigni (e1) and
the film Life is Beautiful (e2)

rk
ei ej

9
Knowledge Graphs Srihari

Training Datasets
• Inferring relations between entities from data
• From Knowledge Bases
1. Databases conveying commonsense knowledge
about everyday life
• Freebase, OpenCyc, WordNet, or Wikibase, etc
2. Databases with expert knowledge about an
application AI system
• GeneOntology
• From Unstructured natural language

10
Knowledge Graphs Srihari

KG Construction Approaches
• Curated:
– Triples created by experts
• Cyc, WordNet, UMLS

• Collaborative:
– Triples created by volunteers
• Wikidata, Freebase

• Automated semi-structured
– Triples from text via rules:
• YAGO, DBPedia, Freebase

• Automated unstructured Knowledge Graph Sizes

– Triples text via ML, NLP

• Knowledge Vault, Nell

11
Knowledge Graphs Srihari

Neural Models of Relations/Entities

• In addition to training data, we also need to
define a model family to train
• Extend neural language models to model
entities and relations
• Neural language models learn:
1. A vector that provides a distributed representation
of each word
2. About interactions between words
• such as which word is likely to come after a sequence of
words, by learning functions of these vectors
• Extend approach to entities and relations by
learning an embedding vector for each relation 12
Knowledge Graphs Srihari

Applications of neural knowledge models

• Link prediction:
– Predicting missing arcs in the knowledge graph
• Entity Resolution
– Is A. Guiness the same as Alec Guiness?
• Link-based Clustering
– Extend feature-based clustering to relation based
on similarity of entities and relations
• Word sense disambiguation
– Deciding which of the senses of a word is the
appropriate one, in some context 13
Knowledge Graphs Srihari

Link Prediction
• Predicting existence (probability of correctness)
of (typed) edges in graph (i.e.,triples)
– KGs often miss facts, and some edges are incorrect

Local Similarity measure:

Common Neighbors (CN)
G=(V,E)
e=(x,y)
Similarity score sxy

14
Knowledge Graphs Srihari

Entity Resolution
• Problem of identifying which objects in
relational data refer to the same underlying
entities

Nodes 1 and 3 refer to the identical entity, the actor Alec Guinness.
Node 2 on the other hand refers to Arthur Guinness, the founder of the Guinness
brewery.
The surface name of node 2 (“A. Guinness”) alone would not be sufficient to perform a
correct matching as it could refer to both Alec Guinness and Arthur Guinness.
However, since links in the graph reveal the occupations of the persons,
15
a relational approach can perform the correct matching
Knowledge Graphs Srihari

Link-based Clustering
• Extend feature-based clustering to relational
learning
– Group entities in relational database using similarity
• Entities are not only grouped by similarity of
their features but also by similarity of their links
• In social networks, it is community detection

16
Knowledge Graphs Srihari

Statistical Relation Learning (SRL)

• Creation of statistical models for relational data
• Triples are assumed to be incomplete and noisy
• Entities and relation types may contain
duplicates

17
Knowledge Graphs Srihari

Probabilistic Knowledge Graphs

• E = {e1, e2, …, eNe} is the set of entities
• R = {r1, r2, …, rNr} the set of all relations in a KG
– Each possible triple is defined as Xijk = (ei , rk , e j)
• over set of entities E and relations R
• as a binary r.v. yijk ∈{0,1} that indicates its existence
• All possible triples in E x R x E can be grouped
in an Adjacency tensor (a three-way array)
– Y ∈ {0,1} Ne x Ne xNr, whose entries are set such that

• Each possible realization of Y can be interpreted as a

possible world.
Knowledge Graphs Srihari

Tensor Representation
• Binary relational data is represented as a tensor
Fibers of a 3rd order tensor

Ne
Slices of a 3rd order tensor
Ne
Nr

19
Knowledge Graphs Srihari

Model for Knowledge Graph

• We are interested in estimating the joint
distribution P(Y), from a subset D⊆ExRxE x{0,1}
of observed triples
• In doing so, we are estimating a probability
distribution over possible worlds, which allows
us to predict the probability of triples based on
the state of the entire knowledge graph

20
Knowledge Graphs Srihari

Size of the Adjacency Tensor

• Y can be enormous for large knowledge graphs
– E.g., in Freebase, with over 40 million entities and
35,000 relations, the no. of possible triples ExRxE
exceeds 1019 elements
• Of course, type constraints reduce this number
considerably
• Even among syntactically valid triples, only a
tiny fraction are likely to be true.
– For example, there are over 450,000 actors and
over 250,000 movies stored in Freebase
– But each actor stars only in a small no. of movies
Knowledge Graphs Srihari

Statistical Properties of KGs

• KGs typically adhere to some deterministic
rules, such as type constraints and transitivity
– Ex: if Leonard Nimoy was born in Boston, and
Boston is located in the USA, then we can infer that
Leonard Nimoy was born in the USA
• Also various “softer” statistical patterns or
regularities, which are not universally true but
nevertheless have useful predictive power
– Homophily: tendency of entities to be related to
other entities with similar characteristics
• US-born actors are more likely to star in US-made movies
22
Knowledge Graphs Srihari

Types of SRL Models

• Triples correlated with certain other triples
– i.e., random variables yijk correlated with each other
• Three main ways to model these correlations:
1.M1: Latent feature models
• yijk are conditionally independent given latent features
associated with SPO type and additional parameters
2.M2: Graph feature models
• yijk are conditionally independent given observed graph
features and additional parameters
3.M3: Markov random fields
• Assume all yijk have local interactions
23
Knowledge Graphs Srihari

Probability Models M1 and M2

• Model classes M1 and M2 predict the existence
of a triple xijk via a score function f (xijk; 𝛳)
– which represents the model’s confidence that a
triple exists given the parameters 𝛳
• The conditional independence assumptions of
M1 and M2 allow the probability model to be
written as follows:

24
Knowledge Graphs Srihari

Latent Feature Models

• Variables yijk are conditionally independent
given global latent features and parameters:

• We discuss forms for score function f (xijk ; 𝛳)

below
– All models explain triples via latent features
25
Knowledge Graphs Srihari

Example of Latent Features

• Fact: Alec Guinness received Academy Award
• Possible Explanation: he is a good actor
– Uses latent features of entities (good actor)
• “Latent” because not directly observed in the data
• Task: infer these features from data
– Denote latent features of entity ei by vector ei ∈ RHe
where He is no. of latent features in the model
– A model for:
• Alec Guinness is a good actor and
• Academy Award is prestigious

26
 Types of Latent Feature Models
Knowledge Graphs Srihari

• Intuition behind relational latent feature models:

– Relationships between entities can be derived from
interactions of their latent features
• Methods to model interactions, and to derive
the existence of a relationship from them
1.Bilinear model
2.Other tensor factorization models
3.Matrix factorization methods
4.Multi-layer perceptrons
• Based on entities (E) and entity relationships (ER)
ha is an additive
hidden layer
27
Knowledge Graphs Srihari

RESCAL
• RESCAL is an embedding method for learning
from knowledge graphs
– For tasks like link prediction and entity resolution
– Scalable to KGs with millions of entities and billions
of facts
– Provide access to relational information for deep
learning methods
• RESCAL = Relational + Scalable

28
Knowledge Graphs Srihari

RESCAL (Bilinear Model)

• Explains triples via pairwise latent features
– The score of a triple xijk is modeled as

– where Wk ∈ RHe X He with entries wabk specify how

latent features a and b interact in the kth relation
• It is bilinear, since interactions between entity vectors are
multiplicative
– Ex: Model the pattern that good actors receive
prestigious awards via

29
Knowledge Graphs Srihari

RESCAL as Tensor Factorization

RESCAL is similar to methods used in recommendation systems,
and to traditional tensor factorization methods

Equation can be written compactly as

Fk=EWkET

where Fk ∈ RNe X Ne is the matrix holding all scores for the kth relation and the
ith row of E ∈ RNe X He holds the latent representation of ei
30
Knowledge Graphs Srihari

Multi-Layer Perceptrons
• We can interpret RESCAL as creating
composite representations of triples and
predicting their existence from this
representation
• In particular, we can rewrite RESCAL as

– Where wk=vec(Wk)

31
Knowledge Graphs Srihari

RESCAL as an MLP
MLP based on entities (E)

One disadvantage of the E-MLP is that it has to define a vector wk and a

matrix Ak for every possible relation,
which requires Ha+(Ha x 2He) parameters per relation.
An alternative is to embed the relation itself, using a Hr-dimensional vector rk

MLP based on entities and relationships (ER)

ER-MLP uses a global weight vector for all relations.

This model was used in the KV project since it has many fewer
parameters than the E-MLP; the reason is that C is independent of the relation k 32
Knowledge Graphs Srihari

RESCAL and ER-MLP as Neural Nets

He=Hr=3 and Ha=3. Note, that the inputs are latent features.
The symbol g denotes the application of the function g (.) 33
Knowledge Graphs Srihari

Embedding of relations in ER-MLP

• Nearest neighbors of latent representations of
selected relations computed with a 60
dimensional model on Freebase
– Nos. represent squared Euclidean distances.
• Semantically related relations near each other
– Closest relations to children relation are:
• parents, spouse, and birth-place

34
Knowledge Graphs Srihari

Neural Tensor Networks (NTNs)

• NTN is a combination of traditional neural
networks with bilinear models
– The NTN model:

• Here Bk is a tensor, where the lth slice Blk has size He x He,
and there are Hb slices
• hbijk is a bilinear hidden layer, since it is derived from a
weighted combination of multiplicative terms
– With more parameters than E-MLP or RESCAL
models it tends to overfit 35
Knowledge Graphs Srihari

Latent Distance Models

• Derive probability relationships from distance
between latent representations of entities
– Entities are likely related if representations are near
– model probability xij via score f (ei,ej) = - d (ei,ej)
1.Structure Embeddings Model
• Ask,Aok transform global latent features of entities to model
relationships specifically for the kth relation

2.TransE Model
– Translates latent features via a relation-specific
offset instead of matrix multiplications.
• Score of a triple xijk is defined as: f TransEijk := - d(ei+rk , ej)
Knowledge Graphs Srihari

Summary of Latent Feature Models

Best model will be dataset dependent

ER-MLP model outperformed the NTN model on a particular dataset
RESCAL worked best on two link prediction tasks.

37
Knowledge Graphs Srihari

Graph Feature Models

• Existence of an edge predicted by extracting
features from observed edges in the graph
– Consider existence of the path
John-- parentOf àAnne ßparentOf -- Mary
representing a common child
– So we could predict the triple (John, marriedTo, Mary)
• Explains triples directly from observed triples in
the KG

38
Knowledge Graphs Srihari

Training Statistical Relation Learning

• We have a set of Nd observed triples and let the
nth triple be denoted by xn
• Each observed triple is either true (denoted
yn=1) or false (denoted yn=0)
• The labeled dataset is D= {(xn,yn)} |n=1,...,Nd}
• Given this, a natural way to estimate the
parameters 𝛳 is to compute the maximum a
posteriori (MAP) estimate:

39
• where λ controls the strength of the prior
Knowledge Graphs Srihari

Loss Function
• We can equivalently state this as a regularized
loss minimization problem

• Where L(p,y) =-log Ber(y|p) is the log loss

• Another loss is squared loss, L(p,y) = (p –y)2
– Using the squared loss can be especially efficient in
combination with a closed-world assumption (CWA)
– Minimization for RESCAL becomes

40
Knowledge Graphs Srihari

Markov Random Fields

• We drop the assumption that the random
variables yijk in Y are conditionally independent
• However, in the case of relational data and
without the conditional independence
assumption, each yijk can depend on any of the
other Ne x Ne x Nr - 1 random variables in Y
• Due to this enormous number of possible
dependencies, it is intractable to estimate the
joint distribution P(Y) without further constraints
41
Knowledge Graphs Srihari

MRFs are Dependency graphs

• Each random variable (in our case, a possible
fact yijk) is represented as a node in the graph,
while each dependency between random
variables is represented as an edge
• To distinguish MRFs from knowledge graphs,
we refer to them as dependency graphs
– While knowledge graphs encode the existence of
facts, dependency graphs encode statistical
dependencies between random variables

42
Knowledge Graphs Srihari

Definition of MRF
• MRF has the form

• where ψ(yc|θ) ≥ 0 is a potential function on the cth clique in

the dependency graph, and

• is the partition function to ensure distribution sums to one

• potential functions capture local correlations between
variables in each clique c
• Defines a distribution over “possible worlds”
– i.e., over random variables Y
• Structure of MRF derived using Markov logic
a template language based on logical formulae
 Markov Logic Networks
Knowledge Graphs Srihari

• Combining probability and first-order logic in a

single representation
– PGMs enable efficient handling of uncertainty
– First-order logic enables compact representation of
a wide variety of knowledge

44
 First Order Knowledge Base
Knowledge Graphs Srihari

– A first-order knowledge base(KB) is a set of

sentences or formulas in first order logic
– Formulas are constructed using four types of
symbols: constants, variables, functions, and
predicates.
– Constant symbols represent objects in the domain
of interest (e.g., people: Anna, Bob, Chris, etc.).
– Variable symbols range over objects in the domain
– Function symbols (e.g.,MotherOf) represent
mappings from tuples of objects to objects.
– Predicate symbols represent relations among
objects in the domain (e.g.,Friends) or attributes of
objects(e.g.,Smokes) 45
Knowledge Graphs Srihari

First-Order Logic
• An atomic formula or atom is a predicate symbol applied to a
tuple of terms e.g., Friends(x;MotherOf(Anna))
• Formulas are recursively constructed from atomic formulas
using logical connectives and quantifiers.
• If F1 and F2 are formulas, the following are also formulas ¬F1
(negation), which is true iff F1 is false;
• F1 ∧ F2(conjunction), which is true iff both F1 and F2 are true
• F1 ∨ F2 (disjunction),which is true iff F1 or F2 is true; F1)
• F1 ⃗ F2 (implication),which is true iff F1 is false or F2 is true
• F1↔ F2 (equivalence), true iff F1 and F2 have same truth value
• ∀x F1(universal quantification), which is true iff F1 is true for
every object in the domain; and
• ∃x F1(existential quantification),which is true iff F1 is true for at
least one object x in the domain. 46
Knowledge Graphs Srihari

Definition of MLN
• An MLN L is a set of pairs (Fi; wi), where Fi is a
formula in first-order logic and wi is a real no.
• Together with constants C={c1; c2; : : : ; c|C|}, it
defines MRF ML;C ( ) as follows:
– 1. ML;C contains one binary node for each possible
grounding of each predicate appearing in L
• Value of node is 1 if ground atom is true, and 0 otherwise
– 2. ML;C contains one feature for each possible
grounding of each formula Fi in L
• The value of this feature is 1 if the ground formula is true,
and 0 otherwise. The weight of the feature is the wi
associated with Fi in L
47
Knowledge Graphs Srihari

Dependency Graph from Markov Logic

• Given formulae F = {Fi}Li=1, create an edge
between nodes if corresponding facts occur in
at least one grounded formula
– A grounding of formula Fi is given by the
assignment of entities to the variables in Fi
• Furthermore, we define ψ(yc|θ) such that

– where xc denotes no. of true groundings of Fc in Y,

and θc denotes the weight for formula Fc
• If θc > 0, we prefer worlds where formula Fc is satisfied
• If θc < 0, we prefer worlds where formula Fc is violated
• If θc = 0, then formula Fc is ignored
Knowledge Graphs Srihari

An example
• Two types of entities: adults and children
• Two types of relations: parentOf and marriedTo
– These relations (edges) are correlated
• since people who share a common child are often
married, while people rarely marry their own children
• In Markov logic, we represent these
dependencies using formulae such as:
• F1: (x, parentOf, z) ^ (y, parentOf, z) => (x, marriedTo, y)
• F2: (x, marriedTo, y) => ¬ (y, parentOf, x)

49
Knowledge Graphs Srihari

Simple Knowledge Graph

There are 4 entities (circles): 3 adults (a1,a2,a3)
and 1 child c
There are 2 types of edges:
adults may or may not be married to each other,
as indicated by the red dashed edges,
and the adults may or may not be parents of the child,
as indicated by
the blue dotted edges.

F1: (x, parentOf, z) ^ (y, parentOf, z) => (x, marriedTo, y)

F2: (x, marriedTo, y) => ¬ (y, parentOf, x)

Six possible facts corresponding to the edges

50
Knowledge Graphs Srihari

Knowledge Graph relations

• We add binary random variables (represented

by diamonds) to each KG edge
– m: married, p: parent 51
Knowledge Graphs Srihari

Knowledge Graph to Dependency Graph

Create edges between these nodes if the

corresponding facts occur in grounded formulae
F1: (x, parentOf, z) ^ (y, parentOf, z) => (x, marriedTo, y)
F2: (x, marriedTo, y) => ¬ (y, parentOf, x)

– For instance, grounding F1 with x=a1, y=a3, and z=c,

creates the edges m13àp1c, m13àp3c, and p1càp3c
Knowledge Graphs Srihari

MRF graph differs from KG

• The process of generating the MRF graph by
applying templated rules to a set of entities is
known as grounding or instantiation
• The topology of the resulting graph is quite
different from the original KG
– In particular, we have one node per possible KG
edge, and these nodes are densely connected
• This can cause computational difficulties

53
Knowledge Graphs Srihari

MRF Inference Tasks

• The inference problem consists of estimating:
1. The most probable configuration y*= arg maxy p(y|θ)
2. The posterior marginals p(yi|θ)
• Both problems are computationally intractable
so heuristic approximations must be used
– Solutions
• For MAP Task: Max Product Linear Programming
• For posterior marginals: Gibbs sampling
• If potential functions are restricted to disjunctions (OR and
NOT, but no AND), then we get a hinge loss MRF (HL-MRF)
– For which efficient convex algorithms can be applied, based on a
continuous relaxation of the binary random variables
54
Knowledge Graphs Srihari

MRF Learning Task

• Deals with specifying:
– Form of potential functions (called structure learning)
– Values for the numerical parameters θ
• In the case of MRFs for KGs, potential functions
are often specified in the form of logical rules
– In this case, structure learning is equivalent to rule
learning
– The parameter estimation problem (which is usually
cast as MAP estimation), although convex, is in
general quite expensive, since it needs to call
inference as a subroutine 55
Knowledge Graphs Srihari

Conclusion on MRFs
• Approaches based on MRFs are very flexible
• But harder to make scalable inference and
devise learning algorithms
– Compared to methods based on observable or
latent feature models

56
Knowledge Graphs Srihari

References
1. M. Nickel, K. Murphy, V. Tresp, E. Gabrilovich, “A Review of
Relational Machine Learningfor Knowledge Graphs”
Proceedings of the IEEE 104(1): 11-33 (2016)
2. M.Richardson , P.Domingos, “Markov logic
networks”, Machine Learning 62, 107–136(2006)

57