Learning functional programs with

function invention and reuse

Candidate number: 1025995

University of Oxford

Honour School of Computer Science - Part B

Trinity 2020
 Abstract

Inductive programming (IP) is a field whose main goal is synthesising

programs that respect a set of examples, given some form of background
knowledge. This paper is concerned with a subfield of IP, inductive func-
tional programming (IFP). We explore the idea of generating modular
functional programs, and how those allow for function reuse, with the aim
to reduce the size of the programs. We introduce two algorithms that
attempt to solve the problem and explore type based pruning techniques
in the context of modular programs. By experimenting with the imple-
mentation of one of those algorithms, we show reuse is important (if not
crucial) for a variety of problems and distinguished two broad classes of
programs that will generally benefit from function reuse.
Contents

1 Introduction 5
1.1 Inductive programming . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 Structure of the report . . . . . . . . . . . . . . . . . . . . . . . . . . 8

2 Background and related work 10

2.1 Background on IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.1 Metagol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2.2 Magic Haskeller . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.3 λ2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3 Invention and reuse . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

3 Problem description 15
3.1 Abstract description of the problem . . . . . . . . . . . . . . . . . . . 15
3.2 Invention and reuse . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

4 Algorithms for the program synthesis problem 21

4.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.1.1 Target language and the type system . . . . . . . . . . . . . . 22
4.1.2 Combinatorial search . . . . . . . . . . . . . . . . . . . . . . . 23
4.1.3 Relations concerning the program space . . . . . . . . . . . . 25
4.1.4 Partitioning the templates . . . . . . . . . . . . . . . . . . . . 27
4.2 The algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2.1 Only linear function templates algorithm . . . . . . . . . . . . 28
4.2.2 Consequences of adding branching templates . . . . . . . . . . 33
4.2.3 An algorithm for branching templates . . . . . . . . . . . . . . 35

3
5 Experiments and results 37
5.1 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
5.1.1 addN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.1.2 filterUpNum . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
5.1.3 addRevFilter . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
5.1.4 maze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.1.5 droplasts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
5.2 On function reuse and function templates . . . . . . . . . . . . . . . . 41

6 Conclusions 45
6.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6.2 Reflections, limitations and future work . . . . . . . . . . . . . . . . . 46

A Implementation details 47
A.1 Language implementation . . . . . . . . . . . . . . . . . . . . . . . . 47
A.2 Algorithms implementation . . . . . . . . . . . . . . . . . . . . . . . 48

Bibliography 49

4
Chapter 1

Introduction

1.1 Inductive programming

Inductive programming (IP) [8] - also known as program synthesis or example based
learning - is a field that lies at the intersection of several computer science topics
(machine learning, artificial intelligence, algorithm design) and is a form of automatic
programming. IP, as opposed to deductive programming [15] (another automatic pro-
gramming approach, where one starts with a full specification of the target program)
tackles the problem starting with an incomplete specification and tries to general-
ize that into a program [1]. Usually, that incomplete specification is represented by
examples, so we can informally define inductive programming to be the process of
creating programs from examples using a limited amount of background information
- we shall call this process the program synthesis problem [20]. We give an example
of what an IP system might produce, given a task:

Example 1.1
Input: The definitions of map and increment and the examples f ([1, 2, 3]) = [2, 3, 4]
and f ([5, 6]) = [6, 7]).
Output: The definition f = map increment.

One of the key challenges of IP (and what makes it attractive) is the need to
learn from small numbers of training examples, which mostly rules out statistical
machine learning approaches, such as SVMS and neural networks. This can clearly
create problems: if the examples are not representative enough, we might not get the
program we expect.
As noted in the survey by Gulwani et al [8], one of the main areas of research in
IP has been end-user programming. More often than not, an application will be used
by a non-programmer, and hence that user will probably not be able to write scripts

5
 Figure 1.1: Flash fill in action

that make interacting with that application easier. IP tries to offer a solution to that
problem: the user could supply a (small) amount of information, such as a list of
examples that describe the task, and an IP system could generate a small script that
automates the task. Perhaps one of the most noteworthy applications of this idea is
in the MS Excel plug-in Flash Fill [7]. Its task is to induce a program that generalizes
some spreadsheet related operation, while only being given a few examples - usage of
Flash Fill can be seen in figure 1.1.

1.2 Motivation
Two main areas of research in IP are inductive functional programming (IFP, which
we will focus on in this paper) and inductive logic programming (ILP). The idea of
function invention in the IFP context is not new, and indeed some systems use it, such
as IGOR 2 and λ2 . In informal terms, function invention mimics the way humans
write programs: instead of writing a long one-line program, we break the bigger
program into auxiliary functions that can be used to build a modular (equivalent)
program.
In this context, we have asked the question of whether another program writing
technique could be useful for inductive programming: reusing the functions that we
have already invented. By reuse we mean that once a function has been invented, it
can then be used in the definition of another function. While some ILP systems have
explored the idea of reusing functions (such as Metagol and Hexmil [2] and to a lesser
extent DILP [5] and ILASP [14]), function reuse and its benefits (if any) have not
really been explored in the IFP context, as noted by Cropper [1]. When investigating
the existing systems with invention capabilities, we have observed that the way the
invention process is conducted makes reuse practically impossible. Moreover, even

6
though predicate invention and reuse have been claimed as useful (at least in the ILP
context [18]), to our knowledge there has been no work that empirically demonstrates
that it is, nor any work discussing when it may be useful. To address those limitations,
in this work we are interesting in the following research questions:

Q1 Can function reuse improve learning performance (find programs faster)?

Q2 What impact does modularity have on pruning techniques, especially type based
ones?

Q3 What impact does the grammar of the synthesized programs have on function
reuse?

Q4 What classes of problems benefit from it; that is, can we describe the kinds of
programs where function reuse is useful?

1.3 Contributions
In this paper, we make the following contributions:

• We provide a formal framework to describe IFP approaches that solve the pro-
gram synthesis problem by creating modular programs and that can exploit
function reuse.

• Given this formal framework, we create two algorithms that solve the synthe-
sis problem. One of them uses type based pruning to speed up the searching
process, but uses a restrictive grammar; we have proven that for general gram-
mars, this algorithm (which uses a “natural” type inference based pruning ap-
proach) loses completeness (which in particular greatly hinders function reuse).
The second algorithm does not use type based pruning and works with general
grammars, but we propose a way in which this might be achieved.

• Our experimental work has provided positive results, which shed light on the
usefulness of reuse in the IFP context; for example, we have shown that reuse
can decrease the size of the synthesized programs and hence reduce the overall
computation time (in some cases dramatically). Through experimentation, we
have also distinguished two classes of problems for which reuse is important:
AI planning problems and problems concerned with nested data structures (we
have focused on lists).

7
1.4 Structure of the report
The rest of the paper is structured as follows:

• chapter 2: Presents background on inductive programming, function invention

and reuse, and a variety of other systems.

• chapter 3: Presents a formal framework for describing the program synthesis

problem and formalizes function reuse.

• chapter 4: Presents two algorithms that attempt to solve the program synthesis
problem, in light of the description presented in chapter 3.

• chapter 5: Explores the role of function reuse through experimentation and

contains a variety of experiments that validate our hypothesis; we also explore
the various use cases of function reuse.

• chapter 6: Presents the conclusions, limitations, possible extensions of the

project.

8
Chapter 2

Background and related work

In the previous chapter, we have informally introduced the concept of inductive pro-
gramming (IP), presented its relevance and showcased our ideas. In this chapter, we
first provide the reader with more background on IP (areas of research, approaches)
and then switch to literature review, showing different IP systems and their relevance
to ours. We finish the chapter by talking about the idea of function invention and
reuse.

2.1 Background on IP
IP has been around for almost half a century, with a lot of systems trying to tackle
the problem of finding programs from examples. It is a subject that is placed at
the crossroad between cognitive sciences, artificial intelligence, algorithm design and
software development [13]. An interesting fact to note is that IP is a machine learning
problem (learning from data) and moreover, in recent years it has gained attention
because of the inherent transparency of its approach to learning, as opposed to the
black box nature of statistical/neuronal approaches, as noted by Schmid [19].
IP has two main research areas, as noted by Gulwani et al. [8]:

• Inductive functional programming (IFP): IFP focuses on the synthesis of func-

tional programs, typically used to create programs that manipulate data struc-
tures.

• Inductive logical programming (ILP): ILP started as research on induction in a

logical context [8], generally used for learning AI tasks. It’s aim is to construct
a hypothesis (logic programs) h which explain examples E in terms of some
background knowledge B [17].

9
 As highlighted in the review by Kitzelmann [13], there have been two main ap-
proaches to inductive programming (for both IFP and ILP):

• analytical approach: Its aim is to exploit features in the input-output exam-

ples; the first systematic attempt was done by Summers’ THESIS [23] system
in 1977. He observed that using a few basic primitives and a fixed program
grammar, a restricted class of recursive LISP programs that satisfy a set of
input-output examples can be induced. Because of the inherent restrictiveness
of the primitives, the analytical approach saw little innovation in the following
decade, but systems like IGOR1, IGOR2 [11] have built on Summers’ work.
The analytical approach is also found in ILP, a well known example being Mug-
gleton’s Progol [16].

• generate-and-test approach (GAT): In GAT, examples are not used to

actually construct the programs, but rather to test streams of possible programs,
selected on some criteria from the program space. Compared to the analytical
approach, GAT tends to be the more expressive approach, at the cost of higher
computational time. Indeed, the ADATE system, a GAT system that uses
genetic programming techniques to create programs, is one of the most powerful
IP system with regards to expressivity [13]. Another well known GAT system is
Katayama’s Magic Haskeller [12], which uses type directed search and higher-
order functions as background knowledge. Usually, to compensate for the fact
that the program space is very big, most GAT systems will include some sort
of pruning that discards undesirable programs.

2.2 Related work

We now present three systems that helped us develop our ideas and contrast them
with our work.

2.2.1 Metagol
Metagol [2] is an ILP system that induces Prolog programs. It uses an idea called
MIL, or meta-interpretative learning, to learn logic programs from examples. It uses
three forms of background information:

• compiled background knowledge (CBK): those are small, first order Prolog pro-
grams that are deductively proven by the Prolog interpreter.

10
 • interpreted background knowledge (IBK): this is represented by higher-order
formulas that are proven with the aid of a meta-interpreter (since Prolog does
not allow clauses with higher-order predicates as variables); for example, we
could describe map/3 using the following two clauses:
map([], [], F ) : − and map([A|As], [B|Bs], F ) : −F (A, B), map(As, Bs, F ).

• metarules: those are rules that enforce the form (grammar) of the induced
program’s clauses; an example would be P (a, b) : −Q(a, c), R(c, b), where upper
case letters are existentially quantified variables (they will be replaced with
CBK or IBK).

The way the hypothesis search works is as follows: try to prove the required atom
using CBK; if that fails, fetch a metarule, and try to fill in the existentially quantified
variables; continue until a valid hypothesis (one that satisfies the examples) is found.
Something to note here is that Metagol generates new examples: if we select the map
metarule, based on the existing examples we can infer a set of derived examples that
the functional argument of map must satisfy. This technique is used to prune incorrect
programs from an early stage. All this process is wrapped in a depth-bounded search,
so as to ensure the shortest program is found.
Our paper has started as an experiment to see whether ideas from Metagol could
be transferred to a functional setting; hence, in the next chapters we use similar
terminology, especially around metarules and background knowledge. We will also
use depth-bounded search in our algorithm, for similar reasons to Metagol.

2.2.2 Magic Haskeller

Katayama’s Magic Haskeller [12] is a GAT approach that uses type pruning and
exhaustive search over the program space. Katayama argues that type pruning makes
the search space manageable. One of the main innovation of the system was the usage
of higher-order functions, which speeds up the searching process and helps simplify
the output programs (which are chains of function applications). Our system differs
in the fact that our programs are modular, which allow for function reuse. One of
Magic Haskeller ’s limitations is the inability to provide user supplied background
knowledge. The implementation of our algorithms enable a user to experiment with
the background functions in a programmatic manner, and we also make it fairly easy
to change the grammar of the programs.

11
2.2.3 λ2
λ2 [6] is an IFP system which combines GAT and analytical methods: the search is
similar to Magic Haskeller, in the way that it uses higher order functions and explores
the program space using type pruning, but differs in the fact that programs have a
nested form (think of where clauses in Haskell) and uses an example propagation
pruning method, similar to Metagol. However, such an approach does not allow
function reuse, since an inner function can’t use an “ancestor” function in its definition
(possible infinite loop). Our paper tries to address this, exploring the possibility of
creating non-nested programs and hence allowing function reuse.

2.3 Invention and reuse

Generally, most IP approaches tend to disregard the extra knowledge found during
the synthesis process as another form of background knowledge. In fact, systems like
λ2 and Magic Haskeller make this impossible because of how the search is conducted.
Some systems, like Igor 2 do have a limited form of it, but it is very restrictive and
does not allow function reuse in a general sense. This usually stems from what gram-
mars the induced programs use. One of our main interests has been the usefulness of
function reuse by allowing a modular (through function invention) way of generating
programs (that is, we create “standalone” functions that can then be pieced together
like a puzzle). For example, consider the drop lasts problem: given a non-empty list
of lists, remove the last element of the outer list as well as the last elements of all
the inner ones. Example 2.1 shows a possible program that was synthesized using
only invention. However, if function reuse is enabled, example 2.2 shows how we can
synthesize a simpler program, which we would expect to reduce the searching time.

Example 2.1 (droplasts - only invention)

target = f1 . f2
f1 = f2 . f4
f 2 = reverse . f 3
f 3 = map reverse
f4 = tail . f5
f 5 = map t a i l

Example 2.2 (droplasts - invention + reuse)

Note how f 1 is reused to create a shorter program.

12
target = f1 . f2
f 2 = map f 1
f 1 = reverse . f 3
f 3 = t a i l . reverse

An interesting questions when considering function reuse is what kind of programs

benefit from it, which we explore in chapter 5, but we will now move to formalizing
the program synthesis problem.

13
Chapter 3

Problem description

Before presenting algorithms that solve the synthesis problem, we need to formalize
it. We will assume, for the rest of the chapter, that all definitions will be relative to
a target language L, whose syntax will be specified in the next chapter.

3.1 Abstract description of the problem

A program synthesis algorithm’s aim is to induce programs with respect to some
sort of user provided specification. The synthesis process will create programs which
we call induced programs, that are composed of a set of functions which we call
induced functions. For each induced program we will distinguish a function called the
target function, which is to be applied to the examples to check whether a candidate
program is a solution. Intuitively, the output shall be an induced program whose
target function satisfies the provided specification.
The provided specification in this paper shall be divided in two parts: background
knowledge and input-output examples.

Definition 3.1 (Background knowledge (BK)). We define background knowledge to

be the information used during the synthesis process. The BK completely determines
the possible forms an induced program can have. There are three types of BK that
we consider:

• Background functions: represents the set of functions provided via an external

source. We require those functions to be total so as to not introduce non-
termination in an induced program. We use the notation BKF to refer to this
kind of knowledge.

14
 • Invented functions: represents the set of functions that are invented during the
synthesis process; this set grows dynamically during the synthesis process (with
each new invented function). We use the notation BKI to refer to this kind of
knowledge.

• Function templates: a set of lambda calculus-style contexts that describe the

possible forms of the induced functions. We use the notation BKT to refer to
it.

Let us unpack this definition. We have referred to both BKI and BKF to be sets
of functions: more precisely, they are sets containing pairs of the form (fname , fbody ):
fname represents the identifier (function name) of function f , whereas fbody corre-
sponds to the body of its definition. When we write f , we refer to the pair. Example
3.1 shows an example of two functions that might be part of BKF .

Example 3.1 (Background functions)

r e v xs = i f xs == [ ]
then [ ]
e l s e r e v ( t a i l xs ) ++ [ head xs ]
addOne x = x + 1

Function templates represent the blueprints that we use when defining invented
functions. They are contexts in the meta-theory of lambda calculus sense, that repre-
sent the application of a higher-order combinator, where the “holes” are place-holders
for the required number of functional inputs for such a combinator. Those place-
holders signify that there is missing information in the body of the induced function
that needs to be filled with either background functions or synthesized functions. We
have chosen those to specify the grammar of the invented functions because higher-
order functions let us combine the background and invented functions in complex
ways, and provide great flexibility. We note the similarity of our function templates
to metarules [3] and sketches [21], which serve similar purposes in the respective sys-
tems where they are used. Example 3.2 shows the form of a few such templates. For
convenience, we number the “holes”, e.g. i , with indices starting from 1.

Example 3.2 (Function templates)

Conditional templates: if 1 then 2 else 3 .
Identity: 1 .
Higher-order templates: 1 . 2 , map 1 , filter 1 .

15
 We say an induced function is complete if its body has no holes and all functions
mentioned in it have a definition, or incomplete otherwise. Similarly, we say an
induced program is complete if it is composed of complete functions. We give a
short example to see how templates and functions interact with each other, which
will provide some intuition for the algorithmic approach to the inductive process
presented in the next chapter.

Example 3.3 (Derivation)

Suppose we wish to find the complete function F = map reverse. The following
process involving the BK will take place: we invent a new function F , and assign it
the map template to obtain the definition F = map 1 ; we then fill the remaining
hole using reverse.

The second part of the specification is represented by examples.

Definition 3.2 (Examples). Examples are user provided input-output pairs that
describe the aim of the induced program. We shall consider two types of examples:

• Positive examples: those specify what an induced program should produce;

• Negative examples: those specify what an induced program should not produce;

We use the relation in 7→+ out to refer to positive examples, and the relation
in 7→− out to refer to negative ones. While the positive examples have a clear role
in the synthesis process, the negative ones serve a slightly different one: they try to
remove ambiguity, which is highlighted in example 3.4. Something to note is that
both the positive and the negative examples need to have compatible types, meaning
that if a type checker is used, all the inputs would share the same type, and so should
the outputs.

Example 3.4 (Examples)

Given the positive examples [3, 2, 1] 7→+ [1, 2, 3] and [5, 4] 7→+ [4, 5], and the negative
examples [1, 3, 2] 7→− [2, 3, 1] and [5, 4] 7→− [5, 4] then the program we want to induce
is likely to be a list sorting algorithm. Note that if we only look at the positive
examples, another possible induced program is reverse, but the negative example
[1, 3, 2] 7→− [2, 3, 1] removes this possibility.

Definition 3.3 (Satisfiability). We say an complete induced program P whose target

function is f satisfies the relations 7→+ and 7→− if:

• ∀(in, out) ∈7→+ .f (in) = out

16
 • ∀(in, out) ∈7→− .f (in) 6= out

Definition 3.4 (Program space). Assume we are given background knowledge BK

and let Tcheck be a type checking algorithm for L. We define the program space PBK
to be composed of programs written in L such that:

1. the bodies of the induced functions are either function templates (which still
have holes in them) or function applications (for completed functions).

2. the inputs for the higher-order functions (of the templates) are either functions
from BKI or BKF .

3. they are typeable w.r.t. Tcheck .

4. they contain no cyclical definitions (guard against non-termination).

Note how the PBK contains induced programs whose functions could still have
unfilled holes. We now describe the solution space, which contains the programs we
consider solutions.

Definition 3.5 (Solution space). Given BK, 7→+ and 7→− , we define the solution
space SBK,→+ ,→− ⊂ PBK to be composed of complete programs whose target functions
satisfy both →+ and →− .

We now formulate the program synthesis problem.

Definition 3.6 (Program Synthesis problem). Given:

• a set of positive input/output examples,

• a set of negative input/output examples,

• background knowledge BK

the Program Synthesis problem is to find a solution S ∈ SBK,7→+ ,7→− that has the
minimum number of functions (textually optimal solution).

17
3.2 Invention and reuse
For this section, suppose A is an algorithm that solves the Program Synthesis problem.
First we formalize the concepts of invention and reuse, which we mentioned in chapters
1 and 2.

Definition 3.7 (Invention). We say that A can invent functions if at any point during
the synthesis process it is able to fill a hole with a fresh function name (i.e. does not
appear in any of the previous definitions).

Definition 3.8 (Reuse). We say that A can reuse functions if at any point during the
synthesis process it is able to fill a hole with a function name that has been invented
at some point during the synthesis process (be it defined or yet undefined).

As we can see, the two definitions are intertwined: we can not have reuse without
invention. The motivation for inventing functions is that this creates modular pro-
grams, which naturally support function reuse. As we shall see in the next chapter,
one of the main consequences with modularity is its effect on type based pruning
techniques.
When function reuse is used, certain problems will benefit from this (such as
droplasts from chapter 2): we could find solutions closer to the root, which can have
noticeable effects on the computation time. However, enabling function reuse means
that the BK increases with each newly invented function, and hence the branching
factor of the search tree increases dynamically: in the end, function reuse can be seen
as a trade-off between the depth and the branching factor of the search tree; this will
benefit some sorts of problems, but for others it will just increase the computation
time. The concerns we talked in this paragraph are related to the research questions
posed in section 1.2, which we will address in the next two chapters.

18
Chapter 4

Algorithms for the program

synthesis problem

As previously presented, we want to create an algorithm that is able to create modular

programs and hence able to reuse already induced functions. Two of the main aims of
such algorithms should be soundness and completeness, which we define next (assume
BK, 7→− and 7→+ are given).

Definition 4.1 (Completeness). We say an algorithm that solves the program syn-
thesis problem is complete if it is able to synthesize any program in SBK,7→+ ,7→− .

Definition 4.2 (Soundness). We say an algorithm that solves the program synthesis
problem is sound if the complete programs it synthesizes have their target function
satisfy 7→+ and 7→+ .

Motivated by Q2 from section 1.2, we are interested in another property of such

algorithms, namely type pruning: we wish to discard undesirable programs based on
their types (e.g. when they become untypable). As we shall see, this third property
will lead to the creation of two algorithms, but next we present some preliminaries
required for understanding them.

4.1 Preliminaries
4.1.1 Target language and the type system
We have chosen the target language to be a λ-like language that supports contexts,
since we don’t want to introduce too much added complexity, while still having enough
expressivity. Its syntax can be seen in figure 4.1. For simplicity, when we provide code
snippets in this language we will adopt a slightly simpler (but equivalent) notation:

19
for example, val f = map ( Variable g ) will be written as f = map g. The language
supports both recursive and non-recursive definitions for the background knowledge.
It also has a number primitives such as +, ==, nil or (:) (the last two let us work
with lists).
To support type based pruning, our language will be fully typed, the typing rules
being shown in figure 4.2 (for brevity we have omitted the typing rules for primitives,
apart for nil and (:)). We define some standard type theoretic terms we will use later:

• typing environment: Usually denoted using Γ, it represents a map that asso-

ciates a type (quantified or not) to a name or a type variable.

• substitution: This represent a way to replace existing type variables in an un-

quantified type with other types; those can also be applied to typing environ-
ments (i.e. mapping the application of the substitution over all the types in the
environment).

• type inference: The process of inferring the type of an expression, given some
typing environment.

• free variable: The free variables of a type or environment are those typing
variables which are not bound; we use the notation ftv (...) to denote the set of
free variables.

• generalizing: We can generalize over a type by closing over the free variables of
that type.

• instantiating: We can instantiate a quantified type by replacing all the bound

variables in it with fresh type variables (and hence make it unquantified).

• unification: Given two types, unification is the process that yields a substitution
that when applied to the types makes them equal; we will use the function unify
to denote this process (which can fail; when we write unify(τ1 , τ2 ) in a condition,
we implicitly mean ”if they can unify”).

4.1.2 Combinatorial search

Most systems that have a generate-and-test or hybrid approach use some form of com-
binatorial search as a means to find new programs. Metagol uses iterative deepening
depth first search (IDDFS), MagicHaskeller uses BFS etc. It is natural then that we

20
 hdecl i ::= ‘val’ hidenti ‘=’ hexpr i – non recursive definition
| ‘rec’ hidenti ‘=’ hexpr i – recursive definition
| ‘Pex’ hexpr i ⇒ hexpr i – a way to specify positive examples
| ‘Nex’ hexpr i ⇒ hexpr i– a way to specify negative examples

hexpr i ::= ‘Num’ n

| ‘Char’ c
| ‘True’
| ‘False’
| ‘Variable’ hidenti
| ‘Lambda’ [hidenti] hexpr i
| hexpr i hexpr i
| ‘If’ hexpr i ‘then’ hexpr i ‘else’ hexpr i’
| i – we need to represent holes in the syntax

Figure 4.1: BNF Syntax

shall also use a form of combinatorial search, more specifically we use IDDFS. We
have chosen this approach because we want to synthesize programs that gradually
increase in size, so as to ensure that the induced program will be the shortest one in
terms of the number of synthesized functions.

4.1.3 Relations concerning the program space

We first formalize what was meant by “cyclical definitions” in the definition of the
program space.

Definition 4.3 (Name usage). We say that a function f directly uses another function
f 0 if fname
0
appears in fbody .
We say that a function f uses another function f 0 if f directly uses f 0 or f uses g
and g directly uses f 0 (transitive closure of directly uses).
We say that a program is acyclic if no function in P uses itself, or cyclic otherwise.

Intuitively, if a complete program is cyclic, then it might not terminate (although

this is not always the case, e.g. programs that include co-recursive functions), so we
want to avoid such programs to not introduce non termination.

Definition 4.4 (Specialisation step). Given a function f , a typing environment Γ,

the specialisation step is a relation on Expr × (Expr , TypingEnvironment) indexed by
a typing environment defined by the following rule:

21
 n∈Z (TNum)
Γ ` N um n : Integer

c ∈ {0 a0 , ...0 z 0 ,0 A0 ...0 Z 0 ,0 00 , ...,0 90 }

(TChar)
Γ ` Char c : Character

- (TTrue) - (TFalse)
Γ ` T rue : Boolean Γ ` F alse : Boolean

x:τ ∈Γ Γ ` e0 : τ → τ 0 Γ ` e1 : τ
(TVar) (TApp)
Γ `Variable x : τ Γ ` e0 e1 : τ 0

Γ ` e : Boolean Γ ` e0 : τ Γ ` e00 : τ (TIf)

Γ ` If e then e0 else e00 : τ

- (TNil) Γ ` e1 : τ Γ ` e2 : [τ ]
Γ ` nil : ∀α . [α] (TCons)
Γ ` (e1 : e2 ) : [τ ]

Γ, x : τ ` e : τ 0 Γ`e:τ ᾱ ∈
/ f tv(Γ)
(TAbs) (TGen)
Γ ` λx . e : τ → τ 0 Γ ` e : ∀ᾱ . τ

Γ`e:τ α is an instantiation of τ (TInst)

Γ`e:α

α 6∈ f tv(Γ)
(THole)
Γ ` i : ∀α . α

Figure 4.2: Typing rules for expr

22
 • if

1. a hole i appears in fbody ;

2. g ∈ BKI ∪ BKF and g does not use f OR g is a fresh function we invent
and add to BKI , whose type in Γ is a fresh type variable (use an existing
function or invent);
3. ρ = unify(τ, Γ(gname )), where τ is the type inferred for i in the type
derivation tree for fbody ;
Γ
then we write fbody →
− (fbody [ gname / i ], ρ Γ) (note, ρ Γ means we apply the
substitution ρ to Γ).

We briefly give some intuition on why the environment is updated. What we do

in step 3 is we try to mimic the TApp typing rule. Since the type of the hole we
are to fill represents a “minimum requirement” for what types can fill it (see the
THole rule), it suffices to make sure the type of the filler function is unifiable with
the type of the hole. Now, because of our top-down approach to typing, we keep the
types of the invented functions unquantified, since the types of those functions are
intertwined. Hence, changes we make in one can lead to changes in other functions’
types, so we need to apply the substitution to the whole environment (note that both
the background functions and the higher-order combinators are quantified, so this
won’t affect them).
Based on this relation, we give an ordering on pairs of induced programs and their
typing environment (we will call those pairs program states).

Definition 4.5 (Ordering). We say that a program state (P, Γ) is more concrete than
another program state (P 0 , Γ0 ) if either

• (specialize) names(P ) = names(P 0 ) and there exist exactly two functions f ∈ P

and f 0 ∈ P 0 such that
0
1. fname = fname
Γ 0
2. fbody →
− (fbody , Γ0 )

• or (define) P 0 = P ∪ {fname = T }, with

1. f used by another function but not yet defined in P

2. T ∈ BKT

23
 3. ρ = unify(τ, Γ(fname )) (where τ is the type that can be inferred for the
template T ) and Γ0 = ρ Γ.

We write this as (P, Γ) 4 (P 0 , Γ0 ).

Here, step 3 in the define rule makes sure that the type of the template “agrees”
with all the constraints collected from other previous uses of the to be defined function
(and again we need to update the whole environment).
Let 4∗ be the reflexive, transitive closure of 4, defined by:

• P = P 0 ⇒ P 4∗ P 0

• ∃P 00 .P 4∗ P ” ∧ P ” 4 P 0 ⇒ P 4∗ P 0 .

4.1.4 Partitioning the templates

Before continuing with the algorithm, we will divide the templates in two categories,
based on their type signatures.

Definition 4.6 (Linear function templates). We define a linear function template to

be a template where the types of the functional inputs of the associated higher-order
function (the function that is applied in the template) share no type variables.

Example 4.1 (Linear function templates)

The map template is a linear function template. The type of the associated combi-
nator is ∀ab.(a → b) → [a] → [b], so the only input (the function we are mapping)
has type a → b, hence it is trivially a linear function template.

Definition 4.7 (Branching function templates). We define a branching function tem-

plate to be a template where the types of the functional inputs of the associated
higher-order function share type variables.

Example 4.2 (Branching function templates)

The composition template is a branching function template. Its type is ∀abc.(b →
c) → (a → b) → a → c, so we have two functional inputs to the template, whose
types share a type variable, namely b.

24
4.2 The algorithms
We will now present two algorithms that solve the program synthesis problem and
focus on synthesizing modular programs that allow function reuse to be employed.
The reason we show two such algorithms is because we wish to explore Q2 from
section 1.2 here:

• we will show that when only linear templates are considered, there exists a
sound and complete algorithm that is capable of effective type pruning.

• we show how the addition of branching function templates breaks the “naive”
algorithm’s completeness, which highlights that “natural” type pruning tech-
niques do not transpose well when general background knowledge is used (and
hence it is not a trivial task).

4.2.1 Only linear function templates algorithm

We begin with the algorithm that uses only linear function templates. We assume
that the BK for the remainder of the subsection will only contain linear templates.
Also assume we are given the relations 7→+ and 7→− .
The core parts of the algorithm are 3 procedures that work together: program-
Search, expand and check. programSearch does the actual search and represents the
entry point for the algorithm; expand and check are both used in programSearch to
generate new programs and to check whether a certain program is a solution, respec-
tively.
We now describe each one in more detail:

• programSearch: this procedure does a depth bounded search; it takes 3 inputs: a

procedure that expands nodes in the search tree (expand ), a function that checks
whether the current node is a solution (check ) and an initial program state.
The initial state is defined to be (Pempty , Γempty ), where Pempty is the empty
set and Γempty is the typing environment that contains the (quantified ) types
of the higher-order functions used in templates and the background functions.
The procedure uses an auxiliary function, boundedSearch, which actually does
the search by expanding and checking nodes in a similar fashion to DFS, but
the search is conducted to a certain depth (cut-off point), which is gradually
increased. Pseudocode for this procedure can be seen in Pseudocode 1.

25
 • expand : given a program state (P, Γ), this procedure implements one use of the
4 relation to create a stream of more concrete programs. Note that because of
the typing conditions in the rules of 4 it is here that we are pruning untypable
programs. To make this procedure deterministic, we must specify how the
two rules from 4 are used to create the stream: if P contains a function that
has at least a hole, pick that hole and fill it using the specialize rule. If no
hole remains, use the define rule to define one of the previously invented but
undefined functions (the target function is the first invented function). The
reason behind this strategy is that it makes sure we are filling the holes as
soon as possible, and hence detect untypable programs early. Of course, when
applying the specialize rule, we will try to fill the hole in all the possible ways
(by either using previously invented or background functions OR by using a
freshly invented function), to ensure determinism; similarly, when the define
rule is used, we will try to assign all the possible templates to the function we
want to define.

• check : checks whether a given node (a program state) is a solution. First, we

make sure that the type of the candidate program’s target is compatible (unifi-
able) with the type of the examples. If it is, we then check whether the program
satisfies the examples using an interpreter for the language (using a program
environment that contains the definitions of the background functions and the
higher-order functions, to which we add the invented functions); pseudocode for
this can be found in Pseudocode 2.

Pseudocode 1 progSearch
procedure progSearch(expand, check, init)
Output: An induced program
for depth = 1 to ∞ do
result ← boundedSrc(expand, check, init, depth)
if result 6= f ailure then
return result
end if
end for
end procedure

We shall call this algorithm Alinear . We will prove Alinear is sound and complete.
One of the main difficulties is to show that our approach to typing works correctly
even thought we go about it in a top-down manner (as opposed to the usual bottom-
up one): there are functions whose types we do not fully know yet; this means that

26
Pseudocode 2 check
procedure check(progEnv, exam, progState)
Output: True if the program satisfies the examples, false otherwise
target ← getT argetF unction(progState) . the fn. that must satisfy exam
if not compatible(exam.type, target.type) then
return f alse
end if
for def ∈ progState.complete do . add complete defn. to env
addDef (progEnv, def )
end for
for pos ∈ exam.positive do
if eval(progEnv, (apply(target.body, pos.in))) 6= pos.out then
return f alse
end if
end for
for neg ∈ exam.negative do
if eval(progEnv, (apply(target.body, neg.in))) = neg.out then
return f alse
end if
end for
return true
end procedure

we work with partial information a lot of the time, and by acquiring more and more
constraints on the types we reach the final ones. In contrast, a normal type inference
algorithm knows the full types of a program’s function.

Lemma 4.1. Let P4 = { P |(Pempty , Γempty ) 4∗ (P, Γ) ∧ Γ is a consistent typing

environment for P }. Then we have P4 = PBK (where BK only contains linear
templates).

Proof. We do so by double inclusion.

⊆ : We need to show that if P ∈ P4 then P ∈ PBK . This follows from the rules
of 4, since the programs in P4 :

– only use functions from BKI and BKF to fill holes, and templates from
BKT when assigning templates;
– are not cyclical (see the second condition in the specialization step relation);
– they are always well typed: this can be proven using a short inductive
argument. The empty state is well typed because it contains the empty
program. Now, suppose we have reached a program state (P, Γ), where P

27
 is typable wrt. Γ and we have (P, Γ) 4 (P 0 , Γ0 ). If this resulted via the
define rule, P 0 must be typable wrt. Γ0 by how Γ0 was defined: the to be
defined function takes all the constraints that were created from using it in
other definitions, and makes sure that the template we apply is compatible
with them. Now, if the specialize rule was used, we only replace a hole
when the function we fill it with has a unifiable type with the type that
can be inferred for the hole (we essentially use the TApp rule). So P 0 must
be typable.

The highlighted points correspond to the conditions a program must satisfy to

be in PBK .

⊇ : we must prove that if P ∈ PBK then P ∈ P4 . Since P ∈ PBK , we know it is

typable wrt. a typing environment Γ. Now, observe that if we pick a function
name that occurs in P , and replace that with a hole, the resulting program
must be typable (by the THole rule) wrt. a typing environment (1). Also,
when the body of a function is a template which has no filled holes, we can
delete that definition and still have a typable program (the function’s name will
still be in the typing environment, but its type becomes more general because
we removed a constraint) (2). Those two actions can be seen as the reverse
of the define and specialize rules. Now, consider the following process: pick
a complete function; replace each function name with a hole, until we are left
with a template that contains only holes; then delete that function; repeat the
process again. It is clear that this process will eventually lead to the empty
program, since at each step we either delete a function, or get closer to deleting
a function. Using (1) and (2), we know that all the intermediary programs
will be typable, wrt. some typing environments. Furthermore, each of those
programs will be acyclical and will only contain information from BKI and
BKF . This means that our process creates a 4-path in reverse, say (P, Γ) <
(P1 , Γ1 ) < · · · < (PN , ΓN ) < (Pempty , Γempty ), for some states (Pi , Γi ), since
what we have essentially done is apply the two rules of 4 in reverse. We do
need to make the following note though: this reverse constructions works when
we consider only linear templates, because those guarantee that no undesirable
side effects occur when a hole is filled, so we can construct a 4-path starting at
either end. Hence, we have that P ∈ P4 .

From the above cases, we have the conclusion.

28
Theorem 4.1. Alinear is sound and complete when considering only linear templates.

Proof. By lemma 4.1 and since expand implements the 4 relation (and hence synthe-
sizes all programs in P4 ), we have that expand produces all programs in PBK (note
that it does not matter that expand uses the rules in a specific order, since the proof
of lemma 4.1 did not consider a specific application order). Since SBK,7→+ ,7→− ⊂ PBK
and check is used on all programs synthesized by expand, we are certain that we will
be able to synthesize all functions in SBK,7→+ ,7→− (1). Furthermore, since a program
is the output of the algorithm only if it satisfies the examples (enforced by check ),
this means that the programs we synthesize are indeed solutions (2). From (1) and
(2) we have that the algorithm is sound and complete.

Hence, From theorem 4.1 we have that Alinear solves the program synthesis prob-
lem.
Motivated by question Q2 from section 1.2, in this subsection we have focused on
creating an algorithm that employs effective and early type based pruning. We have
seen that if linear templates are used, the pruning technique is a “natural” extension
of what a normal type inference system would do. While only using linear templates
might seem restrictive, we note that templates such as map, filter, fold (with the type
of the base element a base type, so that we don’t have shared type variables) are all
linear templates, and they can express a wide range of problems.
We make one last observation here. One can notice that expand does not care
about the types of the examples being compatible with the type of the incomplete
programs’ target type (this is done in check ). To increase the amount of pruning, we
can make expand discard those programs whose target type does not agree with the
type of the examples. We observe that this does not break the completeness of the
algorithm, since the programs we wish to synthesize are in the solution space, and
hence must have their target’s type compatible with the type of the examples.

4.2.2 Consequences of adding branching templates

We now investigate what effect branching templates have on Alinear . We first prove
the next important result.

Theorem 4.2. Alinear is no longer complete when its input contains branching func-
tion templates.

29
Proof. We will show that Alinear is unable to synthesize the following problem: given
a list of lists, reverse all inner lists as well as the outer list. Suppose that the available
function templates are map, composition and identity and that the only background
function is reverse. Given those, we should be able to infer the following program:
target = gen1 . gen2
gen2 = map gen1
gen1 = reverse
Since we have fixed the order in which we define functions and fill the holes (see the
description of expand ), the following derivation sequence will happen on the path to
finding the solution.

1. target = 1 . 2 (we have no holes to fill, define the target function)

2-3. target = gen1 . gen2 (before inventing any other function fill the 2 existing
holes)

4. target = gen1 . gen2, gen2 = map 3 (all holes filled, define gen2)

5. target = gen1 . gen2, gen2 = map gen1 (before inventing any other function fill
the existing hole)

...

After step 1, because (.) is applied to the two holes, we infer that the types of 1
and 2 must be of the form b → c and a → b, respectively. In steps 2-3, since we
invent the functions gen1 and gen2, their types will be b → c and a → b (since they
don’t have a definition, and are also not used elsewhere, they just take the types of
the holes they filled). In step 4, the type of gen2 will change: we must then unify
gen2 ’s type with that of the map template ([d] → [e]). After this unification takes
place, gen2 ’s type will be [d] → [e], the type that can be inferred for 3 is d → e,
but gen1’s type will also change (since it shares a type variable with gen2’s type) to
[e] → c. Now, in step 5, we need to unify gen1 ’s type with the type of 3 , and we
will get [e] → e for gen1. Now, since we know that the type of reverse is of the form
[f ] → [f ], it is clear that the definition gen1 = reverse is impossible, since gen1 ’s
type is incompatible with reverse’s type. We conclude the program we considered
can not be synthesized and hence the algorithm is no longer complete.

The problem here is caused by combining general polymorphism with branching

templates. The fact that we are trying to type programs progressively, in top-down

30
manner, creates the possibility of constraining types too early. An example can be
seen in the previous proof: gen1 and gen2 initially shared a type variable, because we
eagerly adjusted their types to fit with the type of the composition template; this, in
turn, lead to the type of gen1 being ultimately incompatible with the type of reverse.
Normal type inference can deal with this problem because it fully knows the types of
all functions (whereas we are progressively approaching those final types).
Our attempt to solving this involved transforming the branching templates into
linear templates and generating unification constraints on relevant type variables
to correctly deal with polymorphism. For example, take the composition template,
whose “branching” type is (b → c) → (a → b) → (a → c); the new “linear” type
would be (b1 → c) → (a → b0 ) → (a → c), with the constraint that b0 and b1 are
unifiable. The idea here was that this gives us some “wiggle room” when deciding
the types (and hence potentially fixes the problem introduced by polymorphism).
However, we were unable to use this approach to develop an algorithm we were certain
was sound and complete. After this attempt, we managed to find two papers that talk
about typing a lambda like language with first class contexts, by Hashimoto and Ohori
[10] and by Hashimoto [9]. The former paper provides a theoretical basis by creating
a typed calculus for contexts, while the latter develops an ML-like programming
language that supports contexts and furthermore provides a sound and complete
inference system for it. This suggests that there might be a way to have meaningful
type pruning when branching templates are allowed, but we will reserve exploring
this avenue for future work.

4.2.3 An algorithm for branching templates

We now propose an algorithm that is complete and sound when considering branching
templates, which we will call Abranching . Motivated by the observations shown in the
last subsection, we will completely disregard early type pruning for the purposes of
this algorithm. Abranching is similar in structure to Alinear , but it defers type checking
until after we synthesize a complete program (no early type pruning):

• progSearch will remain the same.

• expand will follow the same filling and defining strategy as before and make
sure the synthesized programs are acyclical, but will now completely disregard
anything type related.

31
 • check must now have an additional step, checking whether the program is indeed
typable (using a “normal” inference algorithm based on the typing rules of the
language) before checking for example satisfiability.

It is easy to see that this algorithm is sound (because of the extra check in check )
and complete (expand produces every possible program that is not cyclical, which
is clearly a superset of the solution space), and hence solves the program synthesis
problem. For the experiments involving reuse in the next chapter we shall use an im-
plementation of this algorithm, because, as we will see in the next chapter, branching
templates are important for effective function reuse.

32
Chapter 5

Experiments and results

In this chapter, through experimentation, we will attempt to answer questions

Q1, Q3 and Q4 from section 1.2, which we reiterate:

Q1 Can function reuse improve learning performance (find programs faster)?

Q3 What impact does the grammar of the synthesized programs have on function
reuse?

Q4 What classes of problems benefit from it?

The implementation we shall use was written in Haskell and closely follows Abranching ,
whose outline can be found in Appendix A. We focus on this algorithm’s implementa-
tion because, as we will see in section 5.2, using only linear templates makes it almost
impossible to reuse functions.

5.1 Experiments
We begin by showcasing the experiments we conducted in order to answer Q1. For
simplicity (and since they have enough expressive power for our purpose), we will
use three templates: map, filter and composition. The results of the experiments
are summarised in tables 5.1 (output programs) and 5.2 (learning times); the times
shown are of the form [mean over 5 repetitions] ± standard error.

5.1.1 addN
We begin with a simple yet interesting example.

Problem: Given a number, add N to it.

33
 40

learning time (sec)

30 Reuse
No reuse
20

10

0
4 5 6 7 8 9 10
N

Figure 5.1: Learning times for addN

Method: We will be considering this problem for N ∈ {4, 5, . . . , 10}. The

only background function we use is add1. Example wise, we will use 2 positive
examples of the form x →+ x + N and 2 negative examples of the form x →−
x + M , with M 6= N .

Results: Figure 5.1 plots the mean of the learning times for different values
of N (5 repetitions). As we can see, function reuse is vital here: by creating a
function that adds two, we can reuse it to create a function that adds 4, and
so on; this means that there is a logarithmic improvement in program size from
the no reuse variant, as can be seen in table 5.1 (which shows the solution for
add8 ), which in turn leads to an increase in performance, as can be seen in table
5.2. Something to note is that for N = 16, if reuse is used, the solution is found
in under a second, whereas if reuse is not used no solution is found even after
10 minutes. The result here suggest that the answer to question Q1 is yes.

5.1.2 filterUpNum
Problem: Given a list of characters, remove all upper case and numeric ele-
ments from it.

Method: We use the following background functions: isUpper, isAlpha, isNum,

not and will use 2 positive and 2 negative examples.

34
 Results: As can be seen in the table 5.1, this is a problem where only function
invention suffices, and reuse shows no improvement wrt. program size. However,
this is a good example that shows that, for programs which have a reasonably
small number of functions, reuse does not introduce too much computational
overhead: in our case it doubles the execution time, but this is not noticeable
since both execution times are under half a second.

5.1.3 addRevFilter
Problem: Given a list of integers, add 4 to all elements, filter out the resulting
even elements and reverse the new list.

Method: The background functions used are: add1, add2, isOdd, reverse; we
use 2 positive examples and 2 negative examples.

Results: Again, in this case function reuse does not lead to a shorter solution.
However, in this case there is a noticeable increase in the execution time when
reuse is used, from around 2 seconds to around 9 seconds. Another interesting
observation here is that, while both programs in table 5.1 are the smallest such
programs (given our BK), the one that employs reuse is actually less efficient
than the one that only uses invention: one maps add2 twice over the list,
whereas the other one creates a function that adds 4 and then maps that over
the list once. The result here, together with the result in filterUpNum suggest
the following about Q1: while function reuse could be very helpful in some
situations, sometimes it will not help in finding a shorter solution; furthermore,
while in some cases the computational overhead is not too noticeable, in others
the overhead will be quite sizeable.

5.1.4 maze
Problem: Given a maze that can have blocked cells, a robot must find its way
from the start coordinate to the end coordinate.

Method: The background functions used represent the basic movements of the
robot: mRight, mLeft, mDown, mUp (if the robot tries to move out of the maze,
ignore that move). The mazes we will consider will be 4x4, 6x6 and 8x8; the
start will always be cell (0, 0) and the goals (3, 3), (5, 5) and (7, 7), respectively.
We will use one positive example and no negative examples (no need for them
in such a problem).

35
 Results: Reuse has a dramatic effect on the learning times, as can be seen in
table 5.2 (for the 4x4 problem). Interesting here are the 6x6 and 8x8 variants,
since when enabling reuse we managed to find solutions for both in under 10
seconds, but when reuse was not employed, the system was not able to produce
results even after 10 minutes. The result here enforces our previous assertion
about question Q1: when reuse is applicable, it can make a big difference.

5.1.5 droplasts
Problem: Given a list of lists, remove the last element of the outer list as well
as the last elements of the inner lists.

Method: The background functions we use are reverse, tail, addOne, addTwo,
isOdd, id (the latter 4 functions are noise, to put stress on the system).

Results: From the formulation, we can get a sense that tail combined with
reverse will represent the building block for the solution, since intuitively this
operation would need to be performed on both the outer list as well as on the
inner lists. Indeed, the solution that uses function reuse is both shorter and it
is found much faster than the no reuse variant. As we can see, reuse has had a
major impact here, drastically reducing the computation time (as can be seen
in table 5.2).
Curious about how the system (using reuse) would behave when varying the
number of background functions, we have conducted a few more experiments to
test this. To make it challenging, we have only retained reverse and tail, and all
the functions we added were the identity functions (with different names), so
even if type based pruning would be used, it would not really make a difference.
The results can be seen in figure 5.2 (we plotted the means of 3 executions
± the standard error). The results here enforce our previous assertion about
question Q1, solidifying our belief that reuse is indeed useful, while also showing
that the system behaves respectably when increasing the number of background
functions.

5.2 On function reuse and function templates

After attempting to answer Q1 in the previous subsections, we now consider Q3
and Q4. As previously discussed, function reuse does not come without a cost: in

36
 25

20

learning time (sec)

15

10

0
2 4 6 8 10
# of BK functions used

Figure 5.2: Learning times for droplasts (with reuse)

some cases it negatively affects the execution time. Furthermore, it is clear that not
all programs take advantage of function reuse, so an answer to question Q4 is quite
important.
We have been able to distinguish two classes of problems that benefit from function
reuse: problems that involve repetitive tasks (especially planning in the AI context)
and problems that involve operations on nested structures. For the former class, we
can give the maze problem as an example. In that case, function reuse lead to the
creation of a function that was equivalent to moveRight then moveUp, which helped
reach shorter solutions because the robot used this combination of moves frequently.
For the latter class, droplasts is a perfect illustration. The solution acts on both the
inner and outer lists, which is a good indication that the operation might be repetitive
and hence benefit from the reuse of functions. We note that both classes of programs
we presented contain quite a lot of programs (lots of AI planning tasks can be encoded
in our language and tasks that act on nested structures are common), which is a good
indication that reuse is applicable and can make a difference in practical applications.
Another interesting point (raised by Q3) is how the presence of various function
templates (which induce the grammar of our programs) affects function reuse, and
whether the partition we have presented in the previous chapter plays a part in this.
If we think about the graph the uses relation induces on the invented functions (call
it a functional dependency graph, or FDG), the programs our algorithms synthesize
have acyclic FDGs, because we never introduce cyclical definitions (see definition
4.2). Now, the fact that the types of the holes do not share type variables for linear

37
 Problem Reuse + Invention Only invention
g3 = add1.add1
g5 = add1.add1
g3 = add1.add1 g7 = add1.add1
add8 gen2 = g3.g3 g6 = add1.add1
target = g2.g2 g4 = g6.g7
g2 = g4.g5
target = g2.g3
g3 = filter isAlpha
g4 = not.isUpper
filterUpNum same as R + I
g2 = filter g4
target = g2.g3
g5 = g4.reverse g5 = add2.add2
g4 = map add2 g4 = map g5
addRevFilter g3 = g4.g5 g3 = g4.reverse
g2 = filter isOdd g2 = filter isOdd
target = g2.g3 target = g2.g3
g3 = mRight.mUp
g3 = mRight.mUp g5 = mRight.mRight
maze(4x4) g2 = g3.g3 g4 = mUp.mUp
target = g2.g3 g2 = g4.g5
target = g2.g3
g6 = reverse.tail
g4 = reverse.tail g3 = g6.reverse
g3 = g4.reverse g5 = reverse.tail
dropLasts
g2 = map g3 g4 = g5 reverse
target = g2.g3 g2 = map g4
target = g2.g3

Table 5.1: Programs output for the experimental problems

Problem Reuse + Invention Only invention

add8 13.34 ms ± 0.57 1.18 sec ± 0.01
filterUpNum 338.29 ms ± 11.16 153.90 ms ± 4.57
addRevFilter 9.11 sec ± 0.06 1.97 sec ± 0.01
maze(4x4) 67.50 ms ± 4.85 5.37 sec ± 0.05
dropLasts 1.84 sec ± 0.01 252.24 sec ± 7.16

Table 5.2: Learning times for the experimental problems

templates suggests that in practice the majority of those templates are actually likely
to have one single hole. If this is the case, this means they create linear FDGs, which
make function reuse impossible (otherwise, the FGD would be cyclic). Indeed, we can

38
make the following remark: to enhance function reuse, branching templates should
always be used. In particular, composition and other similar branching templates
that encapsulate the idea of chaining computations are very effective: they create
branches in the FDG, and a function invented on such a branch could be reused on
another one.

39
Chapter 6

Conclusions

6.1 Summary
This project was motivated by the fact that, to the best of our knowledge, inven-
tion and in particular function reuse has not been properly researched in the context
of inductive functional programming.
In chapter 3, we formalized the program synthesis problem, which provided us
with the language necessary to create algorithms. Chapter 4 represents an impor-
tant part of the project, since that is where we have presented two approaches to
solving the program synthesis problem, namely Alinear (which works with a specific
type of background knowledge) and Abranching (which works on general background
knowledge). An interesting result we found was that the form of type pruning Alinear
uses, which relies on a normal type inference process (extended in a natural way to
work with contexts), breaks its completeness if general background knowledge is used:
hence, we observed that type pruning is not a trivial task when synthesizing modular
programs. In chapter 5 we have relied on the implementations of Abranching to show
a variety of situations where function reuse is important: examples such as droplasts,
add8 and maze showed how crucial it can be. We have also distinguished two broad
classes of programs that will generally benefit from function reuse and discussed the
impact the used background knowledge has on reuse.
Overall, we have seen that there is value in exploring the ideas of modular pro-
grams and function reuse, and believe that this project can serve as the base for future
work in this direction.

40
6.2 Reflections, limitations and future work
The project is limited in a few ways and can be further enhanced. One major
limitation is the lack of any form of pruning for Abranching (the algorithm that works
with linear templates benefits from type based pruning). As we have stated in chap-
ter 4, a possible way to overcome the problems of typing with contexts could be
solved by attempting to create a type system and type inference system similar to
the ones described in [10] and [9]. Furthermore, a possible extension of the project
could examine the benefits of pruning programs through example propagation, in a
similar way to how λ2 does it [6]. An interesting point to explore is whether branch-
ing templates would hinder this approach to pruning in any way (more specifically,
whether templates such as composition would prevent any such pruning to be done
before the program is complete). Another avenue to explore would be to see whether
there are other major classes of programs that benefit from function reuse, specifically
problems related to AI and game playing.

41
Appendix A

Implementation details

We briefly give some implementation details for the algorithm Abranching . The im-
plementation can be found at https://github.com/reusefunctional/reusefunctional,
which also contains details on how to run the system.

A.1 Language implementation

The language that we have presented in chapter 4 is very similar to the language
Fun, presented in the Oxford PoPL course [22]. Hence, we have used the parser and
lexer, together with parts of the interpreter for our implementation, but those have
been extended in multiple ways. We have added types to the language and added
a type inference system (which can be found in the files Types.hs and Infer.hs).
The inference system follows classical algorithms, and a similar implementation by
Stephen Diehl can be found in [4], which we have used as a guide. To support the
synthesis process inside the language, we have added three constructs to the language
presented in chapter 4.
Listing A.1 shows the test file used for the add problem mentioned in chapter 5,
which highlights most features of the language. Note that the first three functions
represent the implementation of the higher order functions used in the templates
(this is part of our idea that templates should be easy to modify) and the fourth is a
background function (hence the BK prefix).
Listing A.1: add8 test file
v a l comp ( f , g ) = lambda ( x ) f ( g ( x ) ) ; ;

r e c map( f ) = lambda ( xs )
( i f xs = n i l
then n i l

42
 e l s e f ( head ( xs ) ) :map( f ) ( t a i l ( xs ) ) ) ; ;

r e c f i l t e r ( p ) = lambda ( xs )
( i f xs = n i l
then n i l
else
i f ( p ( head ( xs ) ) )
then head ( xs ) : f i l t e r ( p ) ( t a i l ( xs ) )
e l s e f i l t e r ( p ) ( t a i l ( xs ) ) ) ; ;

v a l BK addOne( x ) = x + 1 ; ;

NEx ( 1 ) => 2 ;;
NEx ( 3 ) => 5 ;;
PEx ( 1 ) => 9 ;;
PEx ( 7 ) => 15 ; ;
Synthesize ( Int ) => Int ; ;

A.2 Algorithms implementation

The implementation of the searching algorithm closely follows the algorithm described
in chapter 4 and can be found in the file Search.hs. The uses relation and the cycle
check is done by creating a dependency graph and checking for cycles when adding
edges during the creation of declarations (DepGraph). The check function has been
implemented with the help of an interpreter for our language (Interpreter.hs). To
have the induced functions in the right order when defining them for the purpose
of checking the examples, we use the topological ordering of the dependency graph’s
nodes (which denote the functions) to make sure a function can only be defined once
all the functions that appear in its body are also defined.

43
References

[1] Andrew Cropper. Efficiently learning efficient programs. PhD thesis, Imperial
College London, UK, 2017.

[2] Andrew Cropper, Rolf Morel, and Stephen H. Muggleton. Learning higher-order
logic programs. CoRR, abs/1907.10953, 2019.

[3] Andrew Cropper and Sophie Tourret. Logical reduction of metarules. CoRR,
abs/1907.10952, 2019.

[4] Stephen Diehl. Hindley-Milner Inference. http://dev.stephendiehl.com/fun/006 -

hindley milner.html.

[5] Richard Evans and Edward Grefenstette. Learning explanatory rules from noisy
data. CoRR, abs/1711.04574, 2017.

[6] John K. Feser, Swarat Chaudhuri, and Isil Dillig. Synthesizing data structure
transformations from input-output examples. In David Grove and Steve Black-
burn, editors, Proceedings of the 36th ACM SIGPLAN Conference on Program-
ming Language Design and Implementation, Portland, OR, USA, June 15-17,
2015, pages 229–239. ACM, 2015.

[7] Sumit Gulwani, William R. Harris, and Rishabh Singh. Spreadsheet data ma-
nipulation using examples. Commun. ACM, 55(8):97–105, August 2012.

[8] Sumit Gulwani, José Hernández-Orallo, Emanuel Kitzelmann, Stephen H. Mug-

gleton, Ute Schmid, and Benjamin G. Zorn. Inductive programming meets the
real world. Commun. ACM, 58(11):90–99, 2015.

[9] Masatomo Hashimoto. First-class contexts in ML. Int. J. Found. Comput. Sci.,
11(1):65–87, 2000.

[10] Masatomo Hashimoto and Atsushi Ohori. A typed context calculus. Theor.
Comput. Sci., 266(1-2):249–272, 2001.

44
[11] Martin Hofmann, Emanuel Kitzelmann, and Ute Schmid. A unifying framework
for analysis and evaluation of inductive programming systems. In Proceedings
of the 2nd Conference on Artificiel General Intelligence (2009), pages 74–79.
Atlantis Press, 2009/06.

[12] Susumu Katayama. Systematic search for lambda expressions. In Marko C. J. D.

van Eekelen, editor, Revised Selected Papers from the Sixth Symposium on Trends
in Functional Programming, TFP 2005, Tallinn, Estonia, 23-24 September 2005,
volume 6 of Trends in Functional Programming, pages 111–126. Intellect, 2005.

[13] Emanuel Kitzelmann. Inductive programming: A survey of program synthe-

sis techniques. In Ute Schmid, Emanuel Kitzelmann, and Rinus Plasmeijer,
editors, Approaches and Applications of Inductive Programming, Third Interna-
tional Workshop, AAIP 2009, Edinburgh, UK, September 4, 2009. Revised Pa-
pers, volume 5812 of Lecture Notes in Computer Science, pages 50–73. Springer,
2009.

[14] Mark Law, Alessandra Russo, and Krysia Broda. The ILASP system for induc-
tive learning of answer set programs. CoRR, abs/2005.00904, 2020.

[15] Zohar Manna and Richard J. Waldinger. A deductive approach to program

synthesis. ACM Trans. Program. Lang. Syst., 2(1):90–121, 1980.

[16] Stephen Muggleton. Inverse entailment and progol. New Generation Comput.,
13(3&4):245–286, 1995.

[17] Stephen Muggleton. Inductive logic programming: Issues, results and the chal-
lenge of learning language in logic. Artif. Intell., 114(1-2):283–296, 1999.

[18] Stephen Muggleton, Luc De Raedt, David Poole, Ivan Bratko, Peter A. Flach,
Katsumi Inoue, and Ashwin Srinivasan. ILP turns 20 - biography and future
challenges. Mach. Learn., 86(1):3–23, 2012.

[19] Ute Schmid. Inductive programming as approach to comprehensible machine

learning. In Christoph Beierle, Gabriele Kern-Isberner, Marco Ragni, Frieder
Stolzenburg, and Matthias Thimm, editors, Proceedings of the 7th Workshop
on Dynamics of Knowledge and Belief (DKB-2018) and the 6th Workshop KI
& Kognition (KIK-2018) co-located with 41st German Conference on Artificial
Intelligence (KI 2018), Berlin, Germany, September 25, 2018, volume 2194 of
CEUR Workshop Proceedings, pages 4–12. CEUR-WS.org, 2018.

45
[20] Ehud Y. Shapiro. Algorithmic Program DeBugging. MIT Press, Cambridge, MA,
USA, 1983.

[21] Armando Solar-Lezama. The sketching approach to program synthesis. In Zhen-

jiang Hu, editor, Programming Languages and Systems, 7th Asian Symposium,
APLAS 2009, Seoul, Korea, December 14-16, 2009. Proceedings, volume 5904 of
Lecture Notes in Computer Science, pages 4–13. Springer, 2009.

[22] Mike Spivey and Sam Statton. POPL course Oxford, 2020.
https://www.cs.ox.ac.uk/teaching/materials19-20/principles/.

[23] Phillip D. Summers. A methodology for LISP program construction from exam-
ples. J. ACM, 24(1):161–175, 1977.

46