Bayesian Networks: Construction,

Inference, Learning and Causal

Interpretation
Volker Tresp
Summer 2014

1
 Introduction

• So far we were mostly concerned with supervised learning: we predicted one or several
target variables based on information on one or several input variables

• If applicable, supervised learning is often the best solution and powerful models such
as Neural Networks and SVMs are available

• But there are cases where supervised learning is not applicable: when there is not one
target variable of interest but many, or when in each data point different variables
might be available or missing

• Typical example: medical domain with many kinds of diseases, symptoms, and context
information: for a given patient little is known and one is interested in the prediction
of many possible diseases

• Bayesian networks can deal with these challenges, which is the reason for their popu-
larity in probabilistic reasoning and machine learning

2
 Bayes Nets

• Deterministic rule-based systems were the dominating approach during the first phase
of AI. A major problem was that deterministic rules cannot deal with uncertainty and
ambiguity

• Probability was rejected since a naive representation would require 2M parameters;

how should all these numbers be specified?

• In Bayes nets one constructs a high-dimensional probability distribution based on a

set of local probabilistic rules

• Bayes nets are closely related to a causal world model

• Bayes Nets started as a small community and then developed to one of the main
approaches in AI

• They also have a great influence on machine learning

3
 Definition of a Bayes Net

• The random variables in a domain are displayed as nodes (vertices)

• Directed links (arcs, edges) represent direct (causal) dependencies between parent
node and child node

• Quantification of the dependency:

– For nodes without parents one specifies a priori probabilities

P (A = i) ∀i

– For nodes with parents, one specifies conditional probabilities. E.g., for two parents

P (A = i|B = j, C = k) ∀i, j, k

4
 Joint Probability Distribution

• A Bayes net specifies a probability distribution in the form

M
Y
P (X1, . . . XM ) = P (Xi|par(Xi))
i=1
where par(Xi) is the set of parent nodes. This set can be zero.

5
 Factorization of Probability Distributions

• Let’s start with the factorization of a probability distribution (see review on Probability
Theory)
M
Y
P (X1, . . . XM ) = P (Xi|X1, . . . , Xi−1)
i=1

• This decomposition can be done with an arbitrary ordering of variables; each variable
is conditioned on all predecessor variables

• The dependencies can be simplified if a variable does not depend on all of its prede-
cessors
P (Xi|X1, . . . , Xi−1) = P (Xi|par(Xi))
with
par(Xi) ⊆ X1, . . . , Xi−1

6
 Causal Ordering

• When the ordering of the variables corresponds to a causal ordering, we obtain a causal
probabilistic network

• A decomposition obeying the causal ordering typically yields a representation with the
smallest number of parent variables

• Deterministic and probabilistic causality: it might be that the underlying true model is
deterministic causal. The probabilistic model leaves out many influential factors. The
assumption is that the un-modeled factors should only significantly influence individual
nodes (and thus appear as noise), but NOT pairs or larger sets of variables (which
would induce dependencies)!

7
 Design of a Bayes Net

• The expert needs to be clear about the important variables in the domain

• The expert must indicate direct causal dependencies by specifying the directed link in
the net

• The expert needs to quantify the causal dependencies: define the conditional proba-
bility tables

8
 Inference

• The most important important operation is inference: given that the state a set of
random variables is known, what is the probability distribution of one or several of the
remaining variables

• Let X be the set of random variables. Let X m ⊆ X be the set of known (measured)
variables and let X q ∈ X \X m be the variable of interest and let X r = X \(X m ∪
X q ) be the set of remaining variables

9
 Inference: Marginalization and Conditioning

• In the simplest inference approach one proceeds as follows :

– We calculate the probability distribution of the known variables and the query
variable via marginalization
X
q m
P (X , X ) = P (X1, . . . XM )
Xr

– The normalization is calculated as

X
P (X m) = P (X q , X m)
Xq

– Calculation of the conditional probability distributions

q m P (X q , X m)
P (X |X ) =
P (X m)

10
 Inference in Bayes nets without Cycles in Undirected Net

• By construction there are no cycles in the directed net; the structure of a Bayesian
net is a directed acyclic graph (DAG)

• But there might be cycles when one ignores the directions

• Let’s first consider the simpler case without cycles in the undirected graph; the struc-
ture of the the Bayes net is a poly-tree: there is at most one directed path between
two nodes

11
 Marginalization

• The marginalization can be computationally expensive: the sum contains exponentially

r
many terms; for binary variables 2|X |

• The goal is now to exploit the structure of the net to calculate the marginalization
efficient

12
 Example: Markov Chain

• Consider the Markov chain of length M with binary variables where the first variable
X1 and the last variable XM are known

• We can apply the iterative formula for propagating information on X1 down the chain
X X
π(Xi) = P (Xi|X1) = P (Xi, Xi−1|X1) = P (Xi|Xi−1)P (Xi−1|X1)
Xi−1 Xi−1

• We can apply the iterative formula for propagating information on XM up the chain
X
λ(Xi) = P (XM |Xi) = P (XM , Xi+1|Xi)
Xi+1
X
= P (Xi+1|Xi)P (XM |Xi+1)
Xi+1

13
 Posterior Probability

• Then we can calculate for any Xi

P (Xi = 1, X1, XM )
P (Xi = 1|X1, XM ) =
P (Xi = 1, X1, XM ) + P (Xi = 0, X1, XM )
P (X1 )P (Xi = 1|X1 )P (Xi = 1|XM )
=
P (X1 )P (Xi = 1|X1 )P (Xi = 1|XM ) + P (X1 )P (Xi = 0|X1 )P (Xi = 0|XM )

π(Xi = 1)λ(Xi = 1)
=
π(Xi = 1)λ(Xi = 1) + π(Xi = 0)λ(Xi = 0)

14
 Propagation in Polytrees

• Inference in polytrees can be performed in a similar (but more complex) way

• π and λ propagation is generalized to polytrees

15
 Max-Propagation

• With similar efficiency, one can calculate the maximum likely configuration (Max-
product Rule)

r
Xmax = arg max P (X r , X m)
Xr

16
 Hidden Markov Models

• Hidden Markov Models (HMMs) are the basis of modern speech recognition systems.
An HMM is a Bayesian network with latent variables

• States corresponds to phonemes; measurements correspond to the acoustic spectrum

• The HMM contains the transition probability between states P (Xi|Xi−1) and emis-
sion probabilities P (Y |X).

• To find the most likely sequence of states, the Viterbi Algorithm is employed, which
is identical to the Max-Prop Algorithm

17
Loopy Belief Propagation: Belief Propagation in Bayes Nets
with Loops

• When there a loops in the undirected Bayes net, belief propagation is not applicable:
There cannot be a local message passing rule since information arriving at a node
from different paths can be correlated

• Loopy Belief Propagation the application of belief propagation to Bayes nets with
cycles (although strictly not correct)

• The local update rules are applied until convergence is achieved (which is not always
the case)

• Loopy Belief Propagation is applied in Probabilistic Relational Models which are ty-
pically large Bayes nets describing domains where relationships are important, e.g.,
friend of

18
 Loopy Belief Propagation in Decoding

• Most interesting and surprising: Turbo codes and Low-Density Parity-Check (LDPC)
Codes use Loopy Belief Propagation for decoding. Both become very close to the
Shannon limit and require long code words and thus produce delays. Thus they are
not useful for interactive communication but for broadcasting of information (mobile
communication) and in space applications (NASA)

19
 Junction tree algorithm: Correct Inference

• Most (non-)commercial Bayes nets contain the junction tree algorithm which realizes
correct probabilistic inference

• The junction tree algorithm combined variables such that the resulting net has no
loops

• The junction tree can be inefficient when the combines variables have many states

20
 Example: Rumor Network

• Alan hears a rumor that the CEO gets fired with P (A) = 0.5

• Mary and Susan completely believe Alan P (M |A) = P (S|A) = 1 but they also
have independent sources P (M |¬A) = P (S|¬A) = 0.2

• Furthermore let’s assume that P (J|M, S) = 1, P (J|M, ¬S) = P (J|¬M, S) =

0.8, P (J|¬S, ¬M ) = 0

• What is the probability that Jack believes the rumor?

X X
P (J) = P (J|M, S)P (M |A)P (S|A)P (A) = P (J|M, S)P (M, S)
A,M,S M,S
where
X
P (M, S) = P (M |A)P (S|A)P (A)
A

• In the example, we get P (J) = 0.648

21
 Rumors: Loopy Belief Propagation

• One correctly calculates

X
P (M ) = P (M |A)P (A)
A
X
P (S) = P (S|A)P (A)
A

• But then incorrectly

X
P (J) = P (J|M, S)P (M )P (S)
M,S
since P (M, S) 6= P (M )P (S). In the example, we get P (J) = 0.744

• Incidentally, P (J|A) and of course P (J|M, S) would be calculated correctly. Con-

ditioning can break the loop!

22
 Rumors: Junction Tree

• One forms the variable Z with four states and P (Z = z1,1|A) = P (M |A)P (S|A),
P (Z = z1,0|A) = P (M |A)(1 − P (S|A)), P (Z = z0,1|A) = (1 −
P (M |A))P (S|A), P (Z = z0,0|A) = (1−P (M |A))(1−P (S|A)), P (Z =
z1,1|¬A) = P (M |¬A)P (S|¬A), ...

• Also, P (J|Z = z1,1) = P (J|M, S), P (J|Z = z1,0) = P (J|M, ¬S),

P (J|Z = z0,1) = P (J|¬M, S), P (J|Z = z0,0) = P (J|¬M, ¬S)

• Then
X
P (Z) = P (Z|A)P (A)
A
and
X
P (J) = P (J|Z)P (Z)
Z
gives the correct solution

23
 Design of a Bayes Net

• The expert needs to be clear about the important variables in the domain

• The expert must indicate direct causal dependencies by specifying the directed link in
the net

• The expert needs to quantify the causal dependencies: define the conditional proba-
bility tables

• This can be challenging if a node has many parents: if a binary node has n binary
parents, then the expert needs to specify 2n numbers!

• To simplify this task one often makes simplifying assumptions; the best-known one is
the Noisy-Or Assumption

24
 Noisy-Or

• Let X ∈ {0, 1} be a binary node with binary parents U1, . . . , Un

• Let qi be the probability, that X = 0 (symptom), when Ui = 1 (disease) and all

other (diseases) Uj = 0; this means, that ci = 1 − qi is the influence of a single
parent (disease) on X (the symptom)
Qn
• Then one assumes, P (X = 0|U1, . . . , Un) = i=1 Uiqi, or equivalently,
n
Y
P (X = 1|U1, . . . , Un) = 1 − Uiqi
i=1

• This means that if several diseases are present that can cause the symptom, then the
probability of the symptom increases (if compared to the probabilities for the single
disease

25
 Maximum Likelihood Learning with Complete Data

• We assume that all nodes in the Bayesian net have been observed for N instances
(e.g., N patients)

• ML-parameter estimation simply means counting

• Let θi,j,k be defines as

θi,j,k = P (Xi = k|par(Xi) = j)

• This means that θi,j,k is the probability that Xi is in state k ist, when its parents
are in the state j (we assume that the states of the parents can be enumerated in a
systematic way)

• Than the ML estimate is

ML = Ni,j,k
θi,j,k P
k Ni,j,k
Here, Ni,j,k is the number of samples in which Xi = k and par(Xi) = j.

26
 MAP-estimate for Integrating Prior Knowledge

• Often counts are very small and a ML-estimate has high variance

• One simply specifies efficient counts (counts from virtual data) which then can be
treated as real counts Let αi,j,k ≥ 0 be virtual counts for Ni,j,k

• One obtains the maximum a posteriori (MAP) estimate as

M AP = αi,j,k + Ni,j,k
θi,j,k P
k (αi,j,k + Ni,j,k )

27
 Missing Data

• The problem of missing data is an important issue in statistical modeling

• In the simplest case, one can assume that data are missing at random

• Data is not missing at random, if for example, I analyse the wealth distribution in a
city and rich people tend to refuse to report their income

• For some models the EM (Expectation Maximization)-algorithm can be applied and

leads to ML or MAP estimates

• Consider a particular data point l. In the E-step we calculate the probability for mar-
ginal probabilities of interest given the known information in that data point dl and
given the current estimates of the parameters θ̂, using belief proagation or the junction
tree algorithm. Then we get
N
X
E(Ni,j,k ) = P (Xi = k, par(Xi) = j|dl , θ̂)
l=1
28
 Note that if the parent and child nodes are known then P (Xi = k, par(Xi) =
j|dl , θ̂) is either zero or one, otherwise a number between zero and one

• Based on the E-step, we get in the M-step

ML = E(Ni,j,k )
θ̂i,j,k P
k E(Ni,j,k )

• E-Step and M-Step are iterated until convergence. One can show that EM does not
decrease the likelihood in each step; EM might converge to local optima, even if there
might only be a global optimum for the model with complete data
 EM-Learning with the HMM and the Kalman Filter

• HMMs are trained using the so-called Baum-Welch-Algorithm. This is exactly an EM

algorithm

• Offline Versions of the Kalman filter can also be performed via EM

29
 Beyond Table Representations

• For learning the conditional probabilities, many approaches have been employed

• Decision Trees, Neural networks, log-linear models, ...

30
 Structural Learning in Bayes Nets

• One can also consider learning the structure of a Bayes net and maybe even discover
causality

• In structural learning, several points need to be considered

• There are models that are structural equivalent. For example in a net with only two
variables A and B one might show that there is statistical correlation between the
two variables, but it is impossible to decide if A → B or A ← B. Colliders (nodes
where arrow-head meet) can make directions identifiable

• If C is highly correlated with A and B and A and B are also highly correlated, it
might be clear from the data that C depends on both A and B but difficult to decide
if it only depends on A or only depends on B

• In general, the structure of the Bayes model and its parameters model the joint dis-
tribution of all the variables under consideration

31
• Recall that the way the data was collected can also have influence: I will get a different
distribution if I consider data from the general population and data collected from
patients visiting a specialists in a rich neighborhood
 Causal Interpretation of Structure an Parameters

• One has to be very cautious with a causal interpretation od a learned model

• One assumption is that the world under consideration is causal

• Another assumption is that all relevant information in part of the model. Variables
that influence more than one variable in the domain are confounders and can lead
to structures and quantified dependencies which do not agree with the causal world
assumption (a gene for the ability of eating with sticks; smoking and lung cancer,
considering the correlation with income of some genetic mutation)

• Sometimes temporal information is available which constraints the direction of links

(Granger causality)

32
 Structure Learning via Greedy Search

• In the most common approaches one defines a cost function and looks for the structure
that is optimal under the cost function. One has to deal with many local optima

• Greedy Search: One starts with an initial network (fully connected, empty))and makes
local changes (removal of directed link, adding a link, reversing direction of a link, ...)
and accepts the change, when the cost function improves

• Greedy search can be started from different initial conditions

• Alternatives: Simulated Annealing, Genetic Algorithms

33
 Cost Functions

• As a cost function one might use a cross-validation set

• Often BIC (Bayesian Information Criterion) is used:

1 M
log L − log N
N 2N
(M is the number of parameters; N is the number of data points)

• Recall, that Bayesian model selection is based on P (D|M). For some models with
complete data, this term can be calculated explicitly and can be used for model selec-
tion

34
 Constrained-Based Methods for Structural Learning

• One performs statistical independence tests and uses those to decide on network
structure

• For the example in the V-structure in the image M and S are marginally mutually
independent but they might become dependent given J. J is dependent on both M
and S

• In the other structure S and M are dependent and S and S are dependent. Also, M
and S are dependent. But now M and S become independent given that J is known!

• The structure with the collider can be identified. The structures (2), (3), (4) are all
structurally equivalent

35
 Concluding Remarks

• Bayesian networks are used as expert systems in medical domains

• The underlying theory can be used to derive the likelihood of complex models (e.g.,
HMMs)

• Markov Nets are related to Bayesian networks and use undirected edges; they typical
do not have a causal interpretation

• Bayes nets and Markov nets are the most important representatives of the class of
graphical models

• In the lecture we focussed on nets with discrete variables. Also commonly studied are
nets with continous variables and Gaussian distributions

36