See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/312618342

Transfer Learning by Inductive Logic Programming

Chapter · July 2015

DOI: 10.1007/978-3-319-27992-3_20

CITATION READS

1 622

3 authors, including:

Hiroyuki Iida H. Jaap Van Den Herik

Japan Advanced Institute of Science and Technology Leiden University
394 PUBLICATIONS   1,364 CITATIONS    604 PUBLICATIONS   5,234 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Game Refinement Theory View project

The HPO framework as a Universarity model. View project

All content following this page was uploaded by Hiroyuki Iida on 24 January 2017.

The user has requested enhancement of the downloaded file.

 Transfer Learning by Inductive Logic
Programming

Yuichiro Sato1(B) , Hiroyuki Iida1 , and H.J. van den Herik2

1
School of Information Science, Japan Advanced Institute of Science
and Technology, 1-1 Asahidai, Nomi, Ishikawa, Japan
{sato.yuichiro,iida}@jaist.ac.jp
2
Leiden Institute of Advanced Computer Science, P.O. Box 9512,
2300 RA Leiden, The Netherlands
[email protected]

Abstract. In this paper, we propose a Transfer Learning method by

Inductive Logic Programing for games. We generate general knowledge
from a game, and specify the knowledge so that it is applicable in another
game. This is called Transfer Learning. We show the working of Transfer
Learning by taking knowledge from Tic-tac-toe and transfer it to Con-
nect4 and Connect5. For Connect4 the number of Heuristic functions we
developed is 30; for Connect5 it is 20.

1 Introduction
An important property of a learning process is generalization. Consequently, a
learning process is seen as an intelligent system able to generalize knowledge. For
example, if a person has learned to play a game well, then that person is able to
transfer his1 knowledge to similar games. This means that the person is able to
learn general knowledge about one game and then apply it as specific knowledge
in another game. Based on this observation, we formulate the following research
question: How do we construct a game-playing AI that has the same ability to
adapt to new games as a human being? In other words, how can we transfer
knowledge which is learned from one game to another game?
For this purpose, General Game Playing (GGP) is an appropriate research
topic. A general Game Player is able to play, in principle, all discrete, finite and
perfect information games (defined by General Games) without any human inter-
vention [1]. There exist many successful implementations for a General Game
Player [2–4]. GGP is a good test bed of algorithms generating game knowl-
edge automatically. An example of such a generation for alpha-beta search and
UCT-MC is reported by Walȩdzik and Mańdziuk [5]. Moreover, game knowl-
edge generation for Heuristic functions that are produced by neural networks
is described by Michulke and Thielscher [6]. However, these studies do generate
game knowledge for a specific game with the aim to play that game well. What
we are trying to achieve is (1) learning general knowledge from a game, and then
1
For brevity, we use ‘he’ and ‘his’, whenever ‘he or she’ and ‘his or her’ are meant.

c Springer International Publishing Switzerland 2015
A. Plaat et al. (Eds.): ACG 2015, LNCS 9525, pp. 223–234, 2015.
DOI: 10.1007/978-3-319-27992-3 20
224 Y. Sato et al.

(2) applying the acquired knowledge as specific knowledge to another game. This
is called Transfer Learning. Transfer Learning is a learning strategy that trans-
fers previously learned general knowledge to improve the learning speed of a
new game [7]. In GGP, Hinrichs and Forbus have reported Transfer Learning by
analogy [8].
A telling example is learning the power of an additional square. This knowl-
edge can be transferred to another domain. For instance, consider the domain
of Tic-tac-toe. The game theoretical value of this game is draw. An interest-
ing question is: What is the game theoretical value when we add an additional
square as shown in Fig. 1? When using this board, the game is a win for the first
player, (start at square 9, with the threat to play on square 8; the idea is to use
the diagonal 2-6-10 as additional threat).
After this learning example, we consider the game of Chess. It is well-known
that a king and two knights are unable to force mate. The highest goal to reach
is stalemate. Assume that we augment the chess-board by an additional square
e0. Then the question again reads: What is the game theoretical outcome of the
KNNK endgame on this board? The answer reads: It is a win for the KNN side.
The end position is shown in Fig. 2. The important point for transfer learning is
that the power of an additional square in one game (Tic-tac-toe) may also unex-
pectedly change the original game theoretical outcome in another game (Chess).
See also [8]. We invite readers to find analogous transfer ideas of this kind.
In this paper we apply the Inductive Logic Programming (ILP) approach
to learn general knowledge for General Games. ILP is a successful approach,
e.g., learning Chess variants and rules is reported to be possible [9]. Some ILP
algorithms are able to make a reasonable specialization from general knowledge
by winning examples only. If the examples represent normal winning situations, a
winning strategy is expected to be learned. In our method, the general knowledge
are boolean functions which represent patterns in a game position. The patterns
may be winning position and losing position. The generated general patterns are
then made specific for incorporation in Heuristic functions that apply to another
game. This is an example of Transfer Learning between games.

Fig. 1. A Tic-tac-toe game board with Fig. 2. A Chess game board with an
an additional square additional square
 Transfer Learning by Inductive Logic Programming 225

In this paper, Tic-tac-toe is chosen as the source game, Connect4 and Con-
nect5 are chosen as target games. We attempt to transfer general knowledge
that is learned from Tic-tac-toe to Connect4 and Connect5. In Sect. 2, we define
the general source concepts in such a way that they are suitable for transfer.
In Sect. 3, we generate the concepts that will be transferred from Tic-tac-toe. In
Sect. 4, we explain how ILP and Transfer Learning work. In Sect. 5, we transfer
concepts that are learned from Tic-tac-toe to Connect4 and Connect5 in order
to generate Heuristic functions for the game involved. In Sect. 6, we test the per-
formance of the generated Heuristic functions. Section 7 provides a discussion.
Section 8 concludes the paper.

2 Concepts in General Games

In GGP, games are described by a specific language, the Game Description
Language (GDL). GDL is a Lisp-like language which has sufficient keywords
to define General Games. General Games are discrete and finite; therefore, a
game position is described as a finite set of pieces which have finite arguments.
They form a string. If the game needs natural numbers, for example the x and y
coordinates of a piece, they are defined in a succ(essor) relationship by the
language [2].
There exist many types of concepts in General Games. For example, a pattern
in a game position must be a sort of concept in that game. All patterns have a
meaning. Some patterns indicate a close-to-win situation, while other patterns
have a meaning as close to loss. This must be included in the set of all concepts
of a particular game.
In GGP, patterns in General Games are also described by GDL. We are able
to convert GDL to Prolog. Therefore, pattern matching of logic programming
is able to describe patterns in General Games. Let us denote a piece in a game
position as a proposition (with the name piece). For example, in Tic-tac-toe, all
pieces are characterized by four coordinates (see Fig. 3). The first coordinate is
the type, in our case it is a cell. The second and third coordinates are the x and
y coordinates. The fourth coordinate is the occupation (x, o, or blank).

Fig. 3. A Tic-tac-toe game position as propositions

In Fig. 3, the first two pieces are described as

226 Y. Sato et al.

piece(cell,1,1,blank).
piece(cell,1,2,blank).

These two pieces are adjacent since the x coordinate is the same and the y
coordinate differs by one. Let us introduce four variables (viz. C, X, Y , and S)
to generalize a pattern in this position. Next to the above propositions, we also
have arithmetic propositions such as Y 2 is Y + 1 (notation is in Prolog). Now
consider the following pattern.

patternX :- piece(C,X,Y,S), Y2 is Y + 1, piece(C,X,Y2,S).

This pattern is a conjunction of three propositions (a conjunction is a com-

bination of propositions connected by and; in the example represented by a
comma). It is applicable to any game which has a two-dimensional game board
and symbols on it. If there exist two adjacent pieces which are represented by
the same symbol, this pattern returns true. In Fig. 4, we find in 4a and 4b the
pattern of two adjacent oo in the top row. In Fig. 4c, we see this pattern in
the second row (seen from the bottom) on position four and five. This way of
characterizing patterns is useful to distinguish game positions and is part of the
semantics of all games.

Fig. 4. A concept on Tic-tac-toe and Connect4

In games, there exist also other types of patterns. Examples are (1) a dis-
junction of propositions, and (2) patterns of time evolution of positions, e.g., a
sequence of changes of positions. For simplicity, we focus only on non-complex
patterns in a position. From now on, let us concentrate on straightforward pat-
terns in a game position (i.e., is the square occupied by x, o, or blank) and
consider them as concepts in the games.

3 Concept Generation from Tic-tac-toe

It is possible to generate concepts from game simulations. In this section, we

generate concepts from simulations of Tic-tac-toe. These concepts are useful
to play other games, e.g., Connect4 and Connect5. Below, we investigate the
generation of conjunctions and disjunctions.
 Transfer Learning by Inductive Logic Programming 227

Our procedure is as follows. We generate conjunctions by replacing the same

symbols in the arguments by a variable. If the n-th arguments of pieces are the
same, then they are replaced by a variable. If the n-th arguments of pieces are
a number, a and b, then a is replaced by a variable and b is replaced by the
sum of the variable and b − a as is done in previous work [5]. The total result
of concept generation from random game simulations of Tic-tac-toe is seen in
Fig. 5. We generated two types of concepts. One is a binary concept and the other
one is a ternary concept. Binary concepts are patterns with two pieces. Ternary
concepts are patterns with three pieces. Concepts are learned from positions
after a playout. We see that if the number of simulations increases the number
of generated concepts also increases. After 2,000 simulations, the learning process
is saturated, i.e., 81 concepts are generated.

Fig. 5. Conjunction concept generation from Tic-tac-toe simulation

A disjunction of propositions is generated by Algorithm 1 (a disjunction is

a combination of propositions connected by or). The algorithm has two types
of parameters, viz. concepts and positions. The input concepts are conjunctions
that are generated from Tic-tac-toe simulations as above. The input positions
are random simulations of Tic-tac-toe games with a winner. The algorithm gen-
erates a disjunction which matches the input positions maximally, and has a
quadratic running time. The result is given in Fig. 6. As the number of input
positions grows, the number of conjunction concepts in a disjunction also grows.
After 2,000 simulations, the learning process is saturated. Finally, a disjunction
made of 17 conjunctions is generated. Three examples in which conjunctions are
involved are as follows.

concept1 :- piece(X1, X2, X3, X4), X5 is X3 - 1, piece(X1, X2, X5, X4),

X6 is X5 - 1, piece(X1, X2, X6, X4).
concept2 :- piece(X1, X2, X3, X4), X5 is X3 + 1, piece(X1, X5, X3, X4),
concept3 :- piece(X1, X2, X3, X4), X5 is X3 - 2, X6 is X5 + 1,
piece(X1, X6, X3, X4).

One of the most complex learned pattern, learned from Tic-tac-toe, is as

follows.
228 Y. Sato et al.

Fig. 6. Disjunction concept generation from Tic-tac-toe simulation

concept11:- piece(X1, X2, X3, X4), X5 is X2 + 1, X6 is X3 + 1,

piece(X1, X5, X6, X4), X7 is X2 + 2, X8 is X3 + 2,
piece(X1, X7, X8, X4).

The example disjunction reads

disjunction :- concept1.
disjunction :- concept2.
disjunction :- concept3.

The concepts (i.e., conjunctions and disjunctions) that are generated in this
section are used to make a Heuristic function by ILP.

4 Concept Specialization by ILP

ILP is a research topic that generates theorems automatically [10]. For example,
it may create a theorem from positive examples, negative examples, and back-
ground knowledge. In our case, it is a specialization in the target game that is
built up by a number of general concepts from the source game. Concepts that
are generated in the previous section are too general to play a specific role in
that target game. Therefore they should be specified. For example, assume the
following concept (concept Y) is learned by generalization of a final position in
Tic-tac-toe (the position is assumed to be a win or a loss). For instance, take
concept Y as follows.

conceptY :- piece(X1, X2, X3, X4), X5 is X3 + 1, piece(X1, X2, X5, X4),

X6 is X5 + 1, piece(X1, X2, X6, X4).

Let us denote the following set of pieces as position 1.

piece(cell, 1, 1, o).
piece(cell, 1, 2, o).
piece(cell, 1, 3, o).
 Transfer Learning by Inductive Logic Programming 229

Algorithm 1. disjunctionGeneration(concepts, positions)

restP osition ⇐ positions
restConcepts ⇐ concepts
result ⇐ empty list
while 0 < size of restP ositions or 0 < size of restConcepts do
counts ⇐ count matchings for each restConcepts to restP ositions
if 0 < max element of counts then
maxConcept ⇐ choose a concept with maximum matching from restConcepts

append maxConcept to result

remove maxConcept from restConcepts
remove positions which matches to maxConcept from restP ositions
else
return make disjunction of result
end if
end while
return make disjunction of result

Subsequently, let us denote the following set of pieces as position 2.

piece(cell, 1, 1, x).
piece(cell, 1, 2, x).
piece(cell, 1, 3, x).

The concept Y is true for both position 1 and position 2.

Assume there exist two players: the o player and the x player. From concept
Y, it is impossible to evaluate whether o’s line is good and x’s line is bad; both
lines are evaluated as the same. To distinguish the o line from the x line, the
concept should be specified. In this case, X4 should be replaced by o or x. ILP
is useful to make (1) this kind of specialization and (2) to formulate Heuristic
functions for the specialization made under (1).
ILP algorithms find the most fitting proposition that explains the examples
by background knowledge. In this case, assume position 1 is a positive example
and position 2 is a negative example. Moreover, concept Y is taken as background
knowledge. Then, the ILP algorithm finds the only difference between position
1 and position 2, being the fourth argument. Consequently, it will make the
following specialization (i.e., the winning specialization is: replacing X4 by o).

conceptY :- piece(cell, 1, 1, o), X5 is 1 + 1, piece(cell, 1, X5, o),

X6 is X5 + 1, piece(cell, 1, X6, o).

This specialization satisfies our demand.

In the literatures we found many ILP implementations. For our ILP tool, we
used Aleph [11]. Aleph is a tool described by Muggleton and De Raedt [12,13].
What Aleph does is specifying general concepts that are learned from Tic-tac-toe
simulations in relation to a target game.
230 Y. Sato et al.

We tried two specializations, viz. a specialization to Connect4 and to Con-

nect5. Positive examples are game positions that end in a win; negative examples
are game positions that end in a loss. In these process, general knowledge is trans-
ferred from a simple game (Tic-tac-toe) to more complicated games (Connect4
and Connect5).

5 Transfer Learning by Concept Specialization

We tried Transfer Learning by specializing concepts from Tic-tac-toe to Con-

nect4 and Connect5. Positive and negative examples are given by random game
simulations. Specialized concepts are (1) a set of conjunctions and (2) a disjunc-
tion made of the conjunctions (see Sect. 3).
For Connect4, we experimented with a different number of positive examples
(and similarly negative examples). The range of the number of positive and
negative examples ran from 1 to 20 for conjunctions; and from 1 to 10 for the
best disjunction (we used only one disjunction in our experiments).
For Connect5, specializations were performed only for conjunctions. The
range of the number of positive and negative examples ran again from 1 to
20. For each set of positive and negative examples, a different specialization was
obtained.
In both cases, we see that if the number of examples increases, the number
of generated specified concepts also increases (see Fig. 7).
Let us now provide an example of a specified concept that is obtained by
the above ILP process. A Heuristic function generated by specialization of con-
junctions by 20 positive and 20 negative examples for Connect4 is made of the
following 5 specified conjunctions (see Fig. 8), consisting of concept 4, 6, 8, 10,
and 11.

concept4 :- piece(cell, 3, 2, r), Y1 is 3 + -1, Y2 is 2 + -1,

piece(cell, Y1, Y2, r).
concept6 :- piece(cell, 5, 3, w), Y1 is 5 + 2, Y2 is 3 + -1,
piece(cell, Y1, Y2, w).
concept8 :- piece(cell, 3, 2, b), Y1 is 3 + -1, piece(cell, Y1, 2, b).
concept10 :- piece(cell, 1, 2, r), Y1 is 1 + 2, Y2 is 2 + 2,
piece(cell, Y1, Y2, r).
concept11 :- piece(cell, 1, 3, w), Y1 is 1 + 2, Y2 is 3 + 1,
piece(cell, Y1, Y2, w).

The Heuristic function is made for the first player when playing Connect4.
Let us have a closer look at the specifications. We take concept 11. This concept
obtained the specification that the type of piece is characterized by cell; the x
coordinate is specified by 1, the y coordinate is specified by 3 and the occupation
by w that is the symbol for the white player (first player) in Connect4 (second
player is r).
For specialization toward Connect4, all conjunctions and the disjunction that
are obtained by Tic-tac-toe were used as background knowledge. However, not
 Transfer Learning by Inductive Logic Programming 231

Fig. 7. Specialization to Connect4 and Connect5

Fig. 8. A Heuristic function for Connect4 seen as a set of specified concepts

all of them were used for the specialization. This means that some concepts are
useful, but others are not useful for this type of game. Here, we may anticipate
on the difference in complexity of Connect4 and Connect5. For instance, we
may state that, in a Tic-tac-toe specialization process toward Connect5, only
concepts which appear in the specialization for Connect4 are used as background
knowledge. This is a meta-concept, i.e., a relationship between concepts. The
meta-concept is suitable for reducing the computation time.
Once general concepts are specified to a target game, the specialized concepts
are useful to make a Heuristic function for that game. Our Heuristic functions
are a set of specialized concepts. If a specialized concept is true in a position,
we may evaluate the position; it will have a positive constant value. If some
specialized concepts are true in a position, the evaluated value of the position is
the total sum of the constant values.
232 Y. Sato et al.

In summary, a set of specialized concepts creates a Heuristic function. From

the specialization processes in this section, we generated 30 sets of specialized
concepts for Connect4 and 20 sets for Connect5. As a direct consequence, we
generated 30 types of Heuristic functions for Connect4 and 20 types of Heuristic
functions for Connect5.

6 Performance of Transfer Learning

The performance of the Heuristics functions generated by the specializations
for Connect4 and Connect5 were tested by game simulations. Game simulations
were performed by an alpha-beta player with a Heuristic function against a ran-
dom player (or an alpha-beta player). For Connect4 we had four experiments. In
experiment 1 and 2, the opponent player was set as a random player. In experi-
ment 3 and 4, the opponent player was set as an alpha-beta player. In experiment
1, the max search depth was set at 1 and the simulation size was set at 5,000. In
experiment 2, the max search depth was set at 3 and the simulation size was set
at 500. In experiment 3, the max search depth was set at 1 and the simulation
size was set at 500. In experiment 4, the max search depth was set at 3 and
the simulation size was set at 500. For Connect5 we performed only one exper-
iment (experiment 5). In experiment 5, the max search depth was set at 1 and
the simulation size was set at 500. The Heuristic functions were indexed by the
number of examples that were generated. Heuristic function 0 means the use of
alpha-beta search without Heuristic function. The results are seen in Figs. 9, 10
and 11.

Fig. 9. Game simulation with a player using a Heuristic function vs a random player
for Connect4

There exists a tendency that the winning ratio increases as the index of the
Heuristic functions also increases (see Figs. 9, 10 and 11). This means that the
 Transfer Learning by Inductive Logic Programming 233

Fig. 10. Game simulation with heuris- Fig. 11. Game simulation with heuris-
tics player vs random player for Con- tics player vs 1-depth alpha-beta player
nect5, search depth 1 for Connect4

Heuristic functions that were generated by many positive and negative exam-
ples, have a better performance when compared to Heuristic functions that were
generated by a smaller number of positive and negative examples. The tendency
clearly appears for the depth-3 search case. We note that depth-1 searches totally
rely on Heuristic functions. However, our Heuristic functions are not perfect, they
have inaccuracies. Therefore we surmise that they guide the middle games suc-
cessfully, but miss sometimes a win in the endgame. This is why our Heuristic
functions perform better in 3-depth search than in 1-depth search, even though
the same Heuristic function is used.

7 Discussion

We observed Transfer Learning between Tic-tac-toe, Connect4, and Connect5.

The success relies on the fact that these games have a similar structure. For
example, the games have a two-dimensional game board, the goal of the game
is to make a line on the board and once a player puts a mark on the board, the
mark never moves.
To do more general Transfer Learning, more analysis for the semantics of
games is needed, e.g., the role of pieces needs to be analyzed. In other games,
there exist many pieces with a specific role. For example, in Chess and Shogi
(Japanese Chess) some pieces have the same legal moves but others do not. If
the similarity between pieces with the same movements has been learned, more
general Transfer Learning will be available.

8 Conclusions

In this paper, we successfully observed Transfer Learning by ILP between games

that have a similar structure. It was possible to produce a general background
234 Y. Sato et al.

knowledge from Tic-tac-toe simulations to make Heuristic functions for Connect4

and Connect5. Improvements of the generated Heuristic functions were observed
when we prepared an increasing number of positive and negative examples.

Acknowledgments. We would like to express great thanks to Aske Plaat for his
advice to this research, and Siegfried Nijssen for his advice on Inductive Logic Pro-
gramming.

References
1. Love, N., Hinrichs, T., Haley, D., Schkufza, E., Genesereth, M.: General Game
Playing: Game Description Language Specfication. Technical report LG-2006-01
Stanford Logic Group (2006)
2. Schiffel, S., Thielscher, M.: Fluxplayer: a successful general game player. In: The
Twenty-Second AAAI Conference on Artificial Intelligence, pp. 1191–1196 (2007)
3. Björnsson, Y., Finnsson, H.: CADIAPLAYER: a simulation-based general game
player. IEEE Trans. Comput. Intell. AI Games 1(1), 4–15 (2009)
4. Méhat, J.M., Cazenave, T.: Ary, a general game playing program. Board Games
Studies Colloquium (2010)
5. Walȩdzik, K., Mańdziuk, J.: An automatically-generated evaluation function in
general game playing. IEEE Trans. Comput. Intell. AI Games 6(3), 258–270 (2014)
6. Michulke, D., Thielscher, M.: Neural networks for state evaluation in general game
playing. In: Buntine, W., Grobelnik, M., Mladenić, D., Shawe-Taylor, J. (eds.)
ECML PKDD 2009, Part II. LNCS, vol. 5782, pp. 95–110. Springer, Heidelberg
(2009)
7. Taylor, E.M., Stone, P.: Transfer learning for reinforcement learning domains: a
survey. J. Mach. Learn. Res. 10, 1633–1685 (2009)
8. Hinrichs, R.T., Forbus, D.K.: Transfer learning through analogy in games. AI Mag.
32(1), 70–83 (2011)
9. Muggleton, S., Paes, A., Santos Costa, V., Zaverucha, G.: Chess revision: acquiring
the rules of chess variants through fol theory revision from examples. In: De Raedt,
L. (ed.) ILP 2009. LNCS, vol. 5989, pp. 123–130. Springer, Heidelberg (2010)
10. Mitchell, M.T., Keller, M.R., Kedar-cabelli, T.S.: Explanation-based generaliza-
tion: a unifying view. Mach. Learn. 1(1), 47–80 (1986)
11. Aleph. http://www.cs.ox.ac.uk/activities/machlearn/Aleph/aleph.html
12. Muggleton, S.H., De Raedt, L.: Inductive logic programming: theory and methods.
J. Logic Program. 19–20, 629–679 (1994)
13. Muggleton, S.: Inverse entailment and progol. New Gener. Comput. 13(3–4), 245–
286 (1995)

View publication stats