Reasoning Under Uncertainty:

Variable Elimination for Bayes Nets

Alan Mackworth

UBC CS 322 – Uncertainty 6

March 22, 2013

Textbook §6.4, 6.4.1

 Announcements (1)
• Assignment 4 due Wednesday, April 3rd

• Final exam
– Thursday, April 18th, 8:30 – 11am in PHRM 1101
– Same general format as midterm (~60% short questions)
• List of short questions will be on Connect
– Practice final to be provided
– More emphasis on material after midterm
– How to study?
• Practice exercises, assignments, short questions,
lecture notes, text, problems in text, …
• Use TA and my office hours (extra office hours TBA if needed)
• Review sessions: last class plus more TBA if needed
• Submit topics you want reviewed on Connect

2
 Announcements (2)
• Teaching Evaluations are online
– You should have received a message about them
– Secure, confidential, mobile access
• Your feedback is important!
– Allows us to assess and improve the course material
– I use it to assess and improve my teaching methods
– The department as a whole uses it to shape the curriculum
– Teaching evaluation results are important for instructors
• Appointment, reappointment, tenure, promotion and merit, salary
– UBC takes them very seriously (now)
– Evaluations close at 11:59PM on April 9, 2013.
• Before exam, but instructors can’t see results until after we submit grades
– Please do it!
• Take a few minutes and visit https://eval.olt.ubc.ca/science
3
 Lecture Overview

• Recap Observations and Inference

• Inference in General Bayesian Networks
– Factors:
• Assigning Variables
• Summing out Variables
• Multiplication of Factors
– The variable elimination algorithm
– Example trace of variable elimination

4
 Learning Goals For Previous Class
• Build a Bayesian Network for a given domain
• Understand basics of Markov Chains and Hidden Markov
Models
• Classify the types of inference:
– Diagnostic, Predictive, Mixed, Intercausal

Assignment 4: Q1, Q2, Q3 and Q4 NOW.

Q5: variable elimination (VE) this class.

5
 Inference in Bayesian Networks

Given:
– A Bayesian Network BN, and
– Observations of a subset of its variables E: E=e
– A subset of its variables Y that is queried
Compute: The conditional probability P(Y|E=e)
How: Run the Variable Elimination (VE) algorithm

N.B. We can already do all this: See lecture “Uncertainty2”

Inference by Enumeration topic.
The BN represents the JPD. Could just multiply out the BN to
get full JPD and then do Inference by Enumeration BUT that’s
extremely inefficient - does not scale.
6
 Inference in Bayesian Networks

Given:
– A Bayesian Network BN, and
– Observations of a subset of its variables E: E=e
– A subset of its variables Y that is queried
Compute: The conditional probability P(Y|E=e)
How: Run the Variable Elimination (VE) algorithm

The VE algorithm manipulates conditional probabilities in the

form of “factors”. So first we have to introduce factors and the
operations we can perform on them.

7
 Lecture Overview

• Recap Observations and Inference

• Inference in General Bayesian Networks
– Factors:
• Assigning Variables
• Summing out Variables
• Multiplication of Factors
– The variable elimination algorithm
– Example trace of variable elimination

8
 Factors
• A factor is a function from a tuple of random variables to
the real numbers R
• We write a factor on variables X1,… ,Xj as f(X1,… ,Xj)

X Y Z val
• P(Z|X,Y) is a factor f (X,Y,Z)
t t t 0.1
– Factors do not have to sum to one
t t f 0.9
– P(Z|X,Y) is a set of probability
t f t 0.2
distributions: one for each
t f f 0.8
combination of values of X and Y
f t t 0.4
f(X, Y)Z=f f t f 0.6
• P(Z=f|X,Y) is a factor f(X,Y) f f t 0.3
f f f 0.7
9
Operation 1: assigning a variable
• We can make new factors out of an existing factor

• Our first operation:

we can assign some or all of the variables of a factor.

X Y Z val What is the result of

assigning X= t ?
t t t 0.1
t t f 0.9 f(X=t,Y,Z) =f(X, Y, Z)X = t
t f t 0.2
Y Z val
f(X,Y,Z): t f f 0.8
t t 0.1
f t t 0.4
t f 0.9
f t f 0.6
f t 0.2
f f t 0.3
f f 0.8
f f f 0.7
Factor of Y,Z
10
 More examples of assignment
X Y Z val f(X=t,Y,Z) Factor of Y,Z
t t t 0.1
Y Z val
t t f 0.9
t t 0.1
t f t 0.2
t f 0.9
f(X,Y,Z): t f f 0.8
f t 0.2
f t t 0.4
f f 0.8
f t f 0.6
f f t 0.3
f f f 0.7
f(X=t,Y,Z=f):

Y val
f(X=t,Y=f,Z=f): 0.8
t 0.9
Number f 0.8

Factor of Y
11
Operation 2: Summing out a variable
• Our second operation on factors:
we can marginalize out (or sum out) a variable
– Exactly as before. Only difference: factors don’t have to sum to 1
– Marginalizing out a variable X from a factor f(X1,… ,Xn) yields a new
factor defined on {X1,… ,Xn } \ {X}
⎛ ⎞
⎜ ∑
⎜ X f ⎟⎟(X 2 ,…, X j ) = ∑ f ( X 1 = x, X 2 ,…, X j )
⎝ 1 ⎠ x∈dom( X 1 )

B A C val (∑Bf3)(A,C)
t t t 0.03
A C val
t t f 0.07
t t 0.57
f t t 0.54
f t f 0.36 t f 0.43

f3= t f t 0.06 f t 0.54

t f f 0.14 f f 0.46
f f t 0.48
12
f f f 0.32
Operation 3: multiplying factors
A B Val
t t 0.1
f1(A,B)× f2(B,C):

f1(A,B): t f 0.9 A B C val

t t t 0.03
f t 0.2
t t f 0.1x0.7
f f 0.8
t f t 0.9x0.6
t f f …
B C Val f t t
t t 0.3 f t f
f2(B,C): t f 0.7 f f t

f t 0.6 f f f

f f 0.4

13
Operation 3: multiplying factors

( f1 × f 2 )( A, B, C) = f1 ( A, B) f 2 ( B, C)

14
 Lecture Overview

• Recap Observations and Inference

• Inference in General Bayesian Networks
– Factors:
• Assigning Variables
• Summing out Variables
• Multiplication of Factors
– The variable elimination algorithm
– Example trace of variable elimination

15
General Inference in Bayesian Networks
Given
– A Bayesian Network BN, and
– Observations of a subset of its variables E: E=e
– A subset of its variables Y that is queried

Compute the conditional probability P(Y=y|E=e)

Marginalization over Y:
Definition of P(E=e) = Σy’∈dom(Y) P(E=e,Y=y’)
conditional probability

P(Y = y, E = e) P(Y = y, E = e)
P(Y = y | E = e) = =
P ( E = e) ∑ P(Y = y' , E = e)
y '∈dom (Y )

All we need to compute is the

joint probability of the query variable(s) and the evidence!
16
 Variable Elimination: Intro (1)
• We can express the joint probability as a factor
Query Observed Other variables not involved in the query
– f(Y, E1…, Ej , Z1…,Zk )

• We can compute P(Y, E1=e1, …, Ej=ej) by

– Assigning E1=e1, …, Ej=ej
– Marginalizing out variables Z1, …, Zk, one at a time
• the order in which we do this is called our elimination ordering

P(Y , E1 = e1 ,…, E j = e j ) = ∑  ∑ f (Y , E1 ,.., E j , Z1 ,.., Z k ) E1 =e1 ,…, E j =e j

Zk Z1

• Are we done?
– No. This would still represent the whole JPD (as a single factor)
– We need to exploit the compactness of Bayesian networks 17
 Variable Elimination: Intro (2)

fi({Xi}  pa(Xi)) = P(Xi|pa(Xi))

P(Y , E1 = e1 ,…, E j = e j ) = ∑  ∑ f (Y , E1 ,.., E j , Z1 ,.., Z k ) E1 =e1 ,…, E j =e j

Zk Z1
n
= ∑  ∑∏ ( f i ) E1 =e1 ,…, E j =e j
Zk Z1 i =1

18
 Computing sums of products
• Inference in Bayesian networks thus reduces to computing
the sums of products
– Example: it takes 9 multiplications to evaluate the expression
ab + ac + ad + aeh + afh + agh.
– How can this expression be evaluated more efficiently?
• Factor out the a and then the h giving a(b + c + d + h(e + f + g))
• This takes only 2 multiplications (same number of additions as above)
n
• Similarly, how can we compute ∑∏ f
Zk i =1
i efficiently?

– Factor out those terms that don't involve Zk, e.g.:

∑ f (Z
Zk
1 k ) f 2 (Y ) f 3 ( Z k , Y ) f 4 ( X , Y )
⎛ ⎞
= f 2 (Y ) f 4 ( X , Y )⎜⎜ ∑ f1 ( Z k ) f 3 (Z k , Y ) ⎟⎟
⎝ Z k ⎠ 19
Summing out a variable efficiently

⎛ ⎞
∑ f1 × ... × f k = f1 × ... × f i × ⎜⎜ ∑ f i +1 × ... × f k ⎟⎟
Zk ⎝ Z k ⎠
New factor! Let’s call it f’
∑f
Zk
1 × ... × f k = f1 × ... × f i × f '

20
 The variable elimination algorithm

See the algorithm VE_BN in the P&M text, Section 6.4.1, Figure 6.8, p. 254.

21
 Lecture Overview

• Recap Observations and Inference

• Inference in General Bayesian Networks

– Factors:
• Assigning Variables
• Summing out Variables
• Multiplication of Factors
– The variable elimination algorithm
– Example trace of variable elimination

22
Variable elimination example: compute P(G|H=h1)
Step 1: construct a factor for each cond. probability
P(G,H) = ∑A,B,C,D,E,F,I P(A,B,C,D,E,F,G,H,I) =
= ∑A,B,C,D,E,F,I P(A)P(B|A)P(C)P(D|B,C)P(E|C)P(F|D)P(G|F,E)P(H|G)P(I|G)
= ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f7(H,G) f8(I,G)

23
Variable elimination example: compute P(G|H=h1)
Step 2: assign observed variables their observed value
P(G,H) = ∑A,B,C,D,E,F,I P(A,B,C,D,E,F,G,H,I) =
= ∑A,B,C,D,E,F,I P(A)P(B|A)P(C)P(D|B,C)P(E|C)P(F|D)P(G|F,E)P(H|G)P(I|G)
= ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f7(H,G) f8(I,G)

Observe H=h1:
P(G,H=h1)=∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C)
f5(F, D) f6(G,F,E) f9(G) f8(I,G)

Assigning the variable H=h1:

f7(H,G) H=h1 = f9(G)

24
 Variable elimination example: compute P(G|H=h1)
Step 3: decompose sum
P(G,H=h1) = ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f9(G) f8(I,G)

= f9(G) ∑F ∑D f5(F, D) ∑B ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C) ∑A f0(A) f1(B,A)

Elimination ordering: A, C, E, I, B, D, F 25
 Variable elimination example: compute P(G|H=h1)
Step 3: decompose sum
P(G,H=h1) = ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f9(G) f8(I,G)

= f9(G) ∑F ∑D f5(F, D) ∑B ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C) ∑A f0(A) f1(B,A)

Elimination ordering: A, C, E, I, B, D, F 26
 Variable elimination example: compute P(G|H=h1)
Step 4: sum out non- query variables (one at a time)
P(G,H=h1) = ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f9(G) f8(I,G)

= f9(G) ∑F ∑D f5(F, D) ∑B ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C) ∑A f0(A) f1(B,A)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C)

Summing out A: ∑A f0(A) f1(B,A) = f10(B)

This new factor does not depend on C, E, or I,
so we can push it outside of those sums.

Elimination ordering: A, C, E, I, B, D, F 27
 Variable elimination example: compute P(G|H=h1)
Step 4: sum out non- query variables (one at a time)
P(G,H=h1) = ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f9(G) f8(I,G)

= f9(G) ∑F ∑D f5(F, D) ∑B ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C) ∑A f0(A) f1(B,A)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) f11(D,B,E)

Elimination ordering: A, C, E, I, B, D, F 28
 Variable elimination example: compute P(G|H=h1)
Step 4: sum out non- query variables (one at a time)
P(G,H=h1) = ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f9(G) f8(I,G)

= f9(G) ∑F ∑D f5(F, D) ∑B ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C) ∑A f0(A) f1(B,A)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) f11(D,B,E)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B) ∑I f8(I,G)

Note the increase in dimensionality:

f12(G,F,D,B) is defined over 4 variables

Elimination ordering: A, C, E, I, B, D, F 29
 Variable elimination example: compute P(G|H=h1)
Step 4: sum out non- query variables (one at a time)
P(G,H=h1) = ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f9(G) f8(I,G)

= f9(G) ∑F ∑D f5(F, D) ∑B ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C) ∑A f0(A) f1(B,A)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) f11(D,B,E)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B) ∑I f8(I,G)

= f9(G) f13(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B)

Elimination ordering: A, C, E, I, B, D, F 30
 Variable elimination example: compute P(G|H=h1)
Step 4: sum out non- query variables (one at a time)
P(G,H=h1) = ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f9(G) f8(I,G)

= f9(G) ∑F ∑D f5(F, D) ∑B ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C) ∑A f0(A) f1(B,A)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) f11(D,B,E)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B) ∑I f8(I,G)

= f9(G) f13(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B)

= f9(G) f13(G) ∑F ∑D f5(F, D) f14(G,F,D)

Elimination ordering: A, C, E, I, B, D, F 31
 Variable elimination example: compute P(G|H=h1)
Step 4: sum out non- query variables (one at a time)
P(G,H=h1) = ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f9(G) f8(I,G)

= f9(G) ∑F ∑D f5(F, D) ∑B ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C) ∑A f0(A) f1(B,A)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) f11(D,B,E)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B) ∑I f8(I,G)

= f9(G) f13(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B)

= f9(G) f13(G) ∑F ∑D f5(F, D) f14(G,F,D)

= f9(G) f13(G) ∑F f15(G,F)

Elimination ordering: A, C, E, I, B, D, F 32
 Variable elimination example: compute P(G|H=h1)
Step 4: sum out non- query variables (one at a time)
P(G,H=h1) = ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f9(G) f8(I,G)

= f9(G) ∑F ∑D f5(F, D) ∑B ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C) ∑A f0(A) f1(B,A)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) f11(D,B,E)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B) ∑I f8(I,G)

= f9(G) f13(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B)

= f9(G) f13(G) ∑F ∑D f5(F, D) f14(G,F,D)

= f9(G) f13(G) ∑F f15(G,F)

= f9(G) f13(G) f16(G)

Elimination ordering: A, C, E, I, B, D, F 33
 Variable elimination example: compute P(G|H=h1)
Step 5: multiply the remaining factors
P(G,H=h1) = ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f9(G) f8(I,G)

= f9(G) ∑F ∑D f5(F, D) ∑B ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C) ∑A f0(A) f1(B,A)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) f11(D,B,E)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B) ∑I f8(I,G)

= f9(G) f13(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B)

= f9(G) f13(G) ∑F ∑D f5(F, D) f14(G,F,D)

= f9(G) f13(G) ∑F f15(G,F)

= f9(G) f13(G) f16(G)

= f17(G)
Elimination ordering: A, C, E, I, B, D, F 34
 Variable elimination example: compute P(G|H=h1)
Step 6: normalize
P(G,H=h1) = ∑A,B,C,D,E,F,I f0(A) f1(B,A) f2(C) f3(D,B,C) f4(E,C) f5(F, D) f6(G,F,E) f9(G) f8(I,G)

= f9(G) ∑F ∑D f5(F, D) ∑B ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C) ∑A f0(A) f1(B,A)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) ∑C f2(C) f3(D,B,C) f4(E,C)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) ∑I f8(I,G) ∑E f6(G,F,E) f11(D,B,E)

= f9(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B) ∑I f8(I,G)

= f9(G) f13(G) ∑F ∑D f5(F, D) ∑B f10(B) f12(G,F,D,B)

= f9(G) f13(G) ∑F ∑D f5(F, D) f14(G,F,D)

= f9(G) f13(G) ∑F f15(G,F) P (G = g , H = h1)

P (G = g | H = h1) =
P ( H = h1)
= f9(G) f13(G) f16(G)
P (G = g , H = h1) f 17( g )
= f17(G) = =
∑ P(G = g ' , H = h1) ∑ f 17( g ' )
g '∈dom ( G ) g '∈dom ( G )
 Learning Goals For Today’s Class
• Variable elimination
– Carry out variable elimination by using factor representation and
using the factor operations
– Use techniques to simplify variable elimination

• Practice Exercises
– Reminder: they are helpful for staying on top of the material,
and for studying for the exam
– Exercise 10 is on conditional independence.
– Exercise 11 is on variable elimination
• Assignment 4 is due on Wednesday, April 3rd
– You should now be able to solve all questions: 1, 2, 3, 4 and 5

36