Illustration of the K2 Algorithm for Learning Bayes Net Structures

Prof. Carolina Ruiz

Department of Computer Science, WPI
[email protected]
http://www.cs.wpi.edu/∼ruiz

The purpose of this handout is to illustrate the use of the K2 algorithm to learn the topology
of a Bayes Net. The algorithm is taken from [CH93].
Consider the dataset given in [CH93]:

case x1 x2 x3
1 1 0 0
2 1 1 1
3 0 0 1
4 1 1 1
5 0 0 0
6 0 1 1
7 1 1 1
8 0 0 0
9 1 1 1
10 0 0 0

Assume that x1 is the classification target

1
 The K2 algorithm taken from [CH93] is included below. This algorithm heuristically searches
for the most probable belief–network structure given a database of cases.
1. procedure K2;
2. {Input: A set of n nodes, an ordering on the nodes, an upper bound u on the
3. number of parents a node may have, and a database D containing m cases.}
4. {Output: For each node, a printout of the parents of the node.}
5. for i:= 1 to n do
6. πi := ∅;
7. Pold := f (i, πi ); {This function is computed using Equation 20.}
8. OKToProceed := true;
9. While OKToProceed and |πi | < u do
10. let z be the node in Pred(xi ) - πi that maximizes f (i, πi ∪ {z});
11. Pnew := f (i, πi ∪ {z});
12. if Pnew > Pold then
13. Pold := Pnew ;
14. πi := πi ∪ {z};
15. else OKToProceed := false;
16. end {while};
17. write(’Node: ’, xi , ’ Parent of xi : ’,πi );
18. end {for};
19. end {K2};

Equation 20 is included below:

Y
qi
(ri − 1)! Y r
i
f (i, πi ) = αijk !
j=1
(Nij + ri − 1)! k=1

where:
πi : set of parents of node xi

qi = |φi |

φi : list of all possible instantiations of the parents of xi in database D. That is, if p1 , . . . , ps

are the parents of xi then φi is the Cartesian product {v1p1 , . . . , vrpp11 }x . . . x{v1ps , . . . , vrppss }
of all the possible values of attributes p1 through ps .

ri = |Vi |

Vi : list of all possible values of the attribute xi

αijk : number of cases (i.e. instances) in D in which the attribute xi is instantiated with its
kth value, and the parents of xi in πi are instantiated with the j th instantiation in φi .
P ri
Nij = k=1 αijk . That is, the number of instances in the database in which the parents of
xi in πi are instantiated with the j th instantiation in φi .

2
 The informal intuition here is that f (i, πi ) is the probability of the database D given that the
parents of xi are πi .
Below, we follow the K2 algorithm over the database above.

Inputs:

• The set of n = 3 nodes {x1 , x2 , x3 },

• the ordering on the nodes x1 , x2 , x3 . We assume that x1 is the classification target. As such
the Weka system would place it first on the node ordering so that it can be the parent of each
of the predicting attributes.

• the upper bound u = 2 on the number of parents a node may have, and

• the database D above containing m = 10 cases.

K2 Algorithm.

i = 1: Note that for i = 1, the attribute under consideration is x1 . Here, r1 = 2 since x1 has two
possible values {0,1}.

1. π1 := ∅
Qq1 (r1 −1)! Q r1
2. Pold := f (1, ∅) = j=1 (N1j +r1 −1)! k=1 α1jk !
Let’s compute the necessary values for this formula.

• Since π1 = ∅ then q1 = 0. Note that the product ranges from j = 1 to j = 0. The

standard convention for a product with an upper limit smaller than the lower limit is
that the value of the product is equal to 1. However, this convention wouldn’t work here
since regardless of the value of i, f (i, ∅) = 1. Since this value denotes a probability, this
would imply that the best option for each node is to have no parents.
Hence, j will be ignored in the formula above, but not i or k. The example on page 13
of [CH93] shows that this is the authors’ intended interpretation of this formula.
• α1 1 = 5: # of cases with x1 = 0 (cases 3,5,6,8,10)
• α1 2 = 5: # of cases with x1 = 1 (cases 1,2,4,7,9)
• N1 = α1 1 + α1 2 = 10

Hence,
(r1 −1)! Q r1 (2−1)! Q2 1
Pold := f (1, ∅) = (N1 +r1 −1)! k=1 α1 k ! = (N1 +2−1)! k=1 α1 k ! = 11! ∗ 5! ∗ 5! = 1/2772

3. Since P red(x1 ) = ∅, then the iteration for i = 1 ends here with π1 = ∅.

3
i = 2: Note that for i = 2, the attribute under consideration is x2 . Here, r2 = 2 since x2 has two
possible values {0,1}.

1. π2 := ∅
Qq2 (r2 −1)! Q r2
2. Pold := f (2, ∅) = j=1 (N2j +r2 −1)! k=1 α2jk !

Let’s compute the necessary values for this formula.

• Since π2 = ∅ then q2 = 0. Note that the product ranges from j = 1 to j = 0. The

standard convention for a product with an upper limit smaller than the lower limit is
that the value of the product is equal to 1. However, this convention wouldn’t work here
since regardless of the value of i, f (i, ∅) = 1. Since this value denotes a probability, this
would imply that the best option for each node is to have no parents.
Hence, j will be ignored in the formula above, but not i or k. The example on page 13
of [CH93] shows that this is the authors’ intended interpretation of this formula.
• α2 1 = 5: # of cases with x2 = 0 (cases 1,3,5,8,10)
• α2 2 = 5: # of cases with x2 = 1 (cases 2,4,6,7,9)
• N2 = α2 1 + α2 2 = 10

Hence,
(r2 −1)! Q r2 (2−1)! Q2 1
Pold := f (2, ∅) = (N2 +r2 −1)! k=1 α2 k ! = (N2 +2−1)! k=1 α2 k ! = 11! ∗ 5! ∗ 5! = 1/2772

3. Since P red(x2 ) = {x1 }, then the only iteration for i = 2 goes with z = x1 .
Qq2 (r2 −1)! Qr2
Pnew := f (2, π2 ∪ {x1 }) = f (2, {x1 }) = j=1 (N2j +r2 −1)! k=1 α2jk !

• φ2 : list of unique π-instantiations of {x1 } in D = ((x1 = 0), (x1 = 1))

• q2 = |φ2 | = 2
• α211 = 4: # of cases with x1 = 0 and x2 = 0 (cases 3,5,8,10)
• α212 = 1: # of cases with x1 = 0 and x2 = 1 (case 6)
• α221 = 1: # of cases with x1 = 1 and x2 = 0 (case 1)
• α222 = 4: # of cases with x1 = 1 and x2 = 1 (case 2,4,7,9)
• N21 = α211 + α212 = 5
• N22 = α221 + α222 = 5
Qq2 (r2 −1)! Q2 (2−1)! Q2 (2−1)! Q2
Pnew = j=1 (N2j +r2 −1)! k=1 α2jk ! = (5+2−1)! ∗ k=1 α21k ! ∗ (5+2−1)! ∗ k=1 α22k !
1 1 1 1 1 1
= 6! ∗ α211 ! ∗ α212 ! ∗ 6! ∗ α221 ! ∗ α222 ! = 6! ∗ 4! ∗ 1! ∗ 6! ∗ 1! ∗ 4! = 6∗5 ∗ 6∗5 = 1/900

4. Since Pnew = 1/900 > Pold = 1/2772 then the iteration for i = 2 ends with π2 = {x1 }.

4
i = 3: Note that for i = 3, the attribute under consideration is x3 . Here, r3 = 2 since x3 has two
possible values {0,1}.

1. π3 := ∅
Qq3 (r3 −1)! Q r3
2. Pold := f (3, ∅) = j=1 (N3j +r3 −1)! k=1 α3jk !

Let’s compute the necessary values for this formula.

• Since π3 = ∅ then q3 = 0. Note that the product ranges from j = 1 to j = 0. The

standard convention for a product with an upper limit smaller than the lower limit is
that the value of the product is equal to 1. However, this convention wouldn’t work here
since regardless of the value of i, f (i, ∅) = 1. Since this value denotes a probability, this
would imply that the best option for each node is to have no parents.
Hence, j will be ignored in the formula above, but not i or k. The example on page 13
of [CH93] shows that this is the authors’ intended interpretation of this formula.
• α3 1 = 4: # of cases with x3 = 0 (cases 1,5,8,10)
• α3 2 = 6: # of cases with x3 = 1 (cases 2,3,4,6,7,9)
• N3 = α3 1 + α3 2 = 10

Hence,
(r3 −1)! Q r3 (2−1)! Q2 1
Pold := f (3, ∅) = (N3 +r3 −1)! k=1 α3 k ! = (N3 +2−1)! k=1 α3 k ! = 11! ∗ 4! ∗ 6! = 1/2310

3. Note that P red(x3 ) = {x1 , x2 }. Initially, π3 = ∅. We need to compute argmax(f (3, π3 ∪

{x1 }), f (3, π3 ∪ {x2 })).
Qq3 (r3 −1)! Qr3
• f (3, π3 ∪ {x1 }) = f (3, {x1 }) = j=1 (N3j +r3 −1)! k=1 α3jk !

– φ3 : list of unique π-instantiations of {x1 } in D = ((x1 = 0), (x1 = 1))

– q3 = |φ3 | = 2
– α311 = 3: # of cases with x1 = 0 and x3 = 0 (cases 5,8,10)
– α312 = 2: # of cases with x1 = 0 and x3 = 1 (case 3,6)
– α321 = 1: # of cases with x1 = 1 and x3 = 0 (case 1)
– α322 = 4: # of cases with x1 = 1 and x3 = 1 (case 2,4,7,9)
– N31 = α311 + α312 = 5
– N32 = α321 + α322 = 5
Qq3 (r3 −1)! Q2 (2−1)! Q2 (2−1)! Q2
f (3, {x1 }) = j=1 (N3j +r3 −1)! k=1 α3jk ! = (5+2−1)! ∗ k=1 α31k ! ∗ (5+2−1)! ∗ k=1 α32k !

= 6!1 ∗ α311 ! ∗ α312 ! ∗ 6!1 ∗ α321 ! ∗ α322 ! = 6!1 ∗ 3! ∗ 2! ∗ 6!1 ∗ 1! ∗ 4! = 6∗5∗2

1 1
∗ 6∗5 = 1/1800
Qq3 (r3 −1)! Qr3
• f (3, π3 ∪ {x2 }) =) = j=1 (N3j +r3 −1)! k=1 α3jk !

– φ3 : list of unique π-instantiations of {x2 } in D = ((x2 = 0), (x2 = 1))

– q3 = |φ3 | = 2
– α311 = 4: # of cases with x2 = 0 and x3 = 0 (cases 1,5,8,10)
– α312 = 1: # of cases with x2 = 0 and x3 = 1 (case 3)

5
 – α321 = 0: # of cases with x2 = 1 and x3 = 0 (no case)
– α322 = 5: # of cases with x2 = 1 and x3 = 1 (case 2,4,6,7,9)
– N31 = α311 + α312 = 5
– N32 = α321 + α322 = 5
Qq3 (r3 −1)! Q2 (2−1)! Q2 (2−1)! Q2
f (3, {x2 }) = j=1 (N3j +r3 −1)! k=1 α3jk ! = (5+2−1)! ∗ k=1 α31k ! ∗ (5+2−1)! ∗ k=1 α32k !
1
= ∗6! α311 ! ∗ α312 ! ∗ 6!1 ∗ α321 ! ∗ α322 ! = 6!1 ∗ 4! ∗ 1! ∗ 6!1 ∗ 0! ∗ 5! = 6∗5
1
∗ 16 = 1/180
Here we assume that 0! = 1

4. Since f (3, {x2 }) = 1/180 > f (3, {x1 }) = 1/1800 then z = x2 . Also, since f (3, {x2 }) =
1/180 > Pold = f (3, ∅) = 1/2310, then π3 = {x2 }, Pold := Pnew = 1/180.

5. Now, the next iteration of the algorithm for i = 3, considers adding the remaining predecessor
of x3 , namely x1 , to the parents of x3 .
Qq3 (r3 −1)! Q r3
f (3, π3 ∪ {x1 }) = f (3, {x1 , x2 }) = j=1 (N3j +r3 −1)! k=1 α3jk !

• φ3 : list of unique π-instantiations of {x1 , x2 } in D = ((x1 = 0, x2 = 0), (x1 = 0, x2 =

1), (x1 = 1, x2 = 0), (x1 = 1, x2 = 1))
• q3 = |φ3 | = 4
• α311 = 3: # of cases with x1 = 0, x2 = 0 and x3 = 0 (cases 5,8,10)
• α312 = 1: # of cases with x1 = 0, x2 = 0 and x3 = 1 (case 3)
• α321 = 0: # of cases with x1 = 0, x2 = 1 and x3 = 0 (no case)
• α322 = 1: # of cases with x1 = 0, x2 = 1 and x3 = 1 (case 6)
• α331 = 1: # of cases with x1 = 1, x2 = 0 and x3 = 0 (case 1)
• α332 = 0: # of cases with x1 = 1, x2 = 0 and x3 = 1 (no case)
• α341 = 0: # of cases with x1 = 1, x2 = 1 and x3 = 0 (no case)
• α342 = 4: # of cases with x1 = 1, x2 = 1 and x3 = 1 (case 2,4,7,9)
• N31 = α311 + α312 = 4
• N32 = α321 + α322 = 1
• N33 = α331 + α332 = 1
• N34 = α341 + α342 = 4
Qq3 (r3 −1)! Q2
f (3, {x1 , x2 }) = j=1 (N3j +r3 −1)! k=1 α3jk !
(2−1)! Q2 (2−1)! Q2 (2−1)! Q2 (2−1)! Q2
= (4+2−1)! ∗ k=1 α31k ! ∗ (1+2−1)! ∗ k=1 α32k ! ∗ (1+2−1)! ∗ k=1 α33k ! ∗ (4+2−1)! ∗ k=1 α34k !
1 1 1 1
= 5! ∗ α311 ! ∗ α312 ! ∗ 2! ∗ α321 ! ∗ α322 ! ∗ 2! ∗ α331 ! ∗ α332 ! ∗ 5! ∗ α341 ! ∗ α342 !
1 1 1 1 1 1 1 1
= 5! ∗ 3! ∗ 1! ∗ 2! ∗ 0! ∗ 1! ∗ 2! ∗ 1! ∗ 0! ∗ 5! ∗ 0! ∗ 4! = 5∗4 ∗ 2 ∗ 2 ∗ 5 = 1/400

6. Since Pnew = 1/400 < Pold = 1/180 then the iteration for i = 3 ends with π3 = {x2 }.

6
Outputs: For each node, a printout of the parents of the node.

Node: x1 , Parent of x1 : π1 = ∅
Node: x2 , Parent of x2 : π2 = {x1 }
Node: x3 , Parent of x3 : π3 = {x2 }

This concludes the run of K2 over the database D. The learned topology is

x1 → x2 → x3

References
[CH93] Gregory F. Cooper and Edward Herskovits. A bayesian method for the induc-
tion of probabilistic networks from data. Technical Report KSL-91-02, Knowl-
edge Systems Laboratory. Medical Computer Science. Stanford University School
of Medicine, Stanford, CA 94305-5479, Updated Nov. 1993. Available at:
http://www.springerlink.com/content/85c6f40ef659d8b2/fulltext.pdf.