Nested Datatypes

Richard Bird1 and Lambert Meertens2

1
Programming Research Group, Oxford University
Wolfson Building, Parks Road, Oxford, OX1 3QD, UK
[email protected]
2
CWI and Department of Computer Science, Utrecht University
P.O. Box 94079, 1090 GB Amsterdam, The Netherlands
[email protected]

Abstract. A nested datatype, also known as a non-regular datatype, is

a parametrised datatype whose declaration involves different instances of
the accompanying type parameters. Nested datatypes have been mostly
ignored in functional programming until recently, but they are turning
out to be both theoretically important and useful in practice. The aim of
this paper is to suggest a functorial semantics for such datatypes, with
an associated calculational theory that mirrors and extends the standard
theory for regular datatypes. Though elegant and generic, the proposed
approach appears more limited than one would like, and some of the
limitations are discussed.

Hark, by the bird’s song ye may learn the

nest.
Tennyson The Marriage of Geraint

1 Introduction
Consider the following three datatype definitions, all of which are legal Haskell
declarations:
data List a = NilL | ConsL (a, List a)
data Nest a = NilN | ConsN (a, Nest (a, a))
data Bush a = NilB | ConsB (a, Bush (Bush a))
The first type, List a, describes the familiar type of cons-lists. Elements of the
second type Nest a are like cons-lists, but the lists are not homogeneous: each
step down the list, entries are “squared”. For example, using brackets and com-
mas instead of the constructors NilN and ConsN , one value of type Nest Int
is
[7, (1, 2), ((6, 7), (7, 4)), (((2, 5), (7, 1)), ((3, 8), (9, 3)))]
This nest has four entries which, taken together, contain fifteen integers.
In the third type Bush a, at each step down the list, entries are “bushed”.
For example, one value of type Bush Int is

Johan Jeuring (Ed.): MPC’98, LNCS 1422, pp. 52–67, 1998.

c Springer-Verlag Berlin Heidelberg 1998
 Nested Datatypes 53

[ 4,
[ 8, [ 5 ], [ [ 3 ] ] ],
[ [ 7 ], [ ], [ [ [ 7 ] ] ] ],
[ [ [ ], [ [ 0 ] ] ] ]
]

This bush contains four entries, the first of which is an element of Int, the
second an element of Bush Int , the third an element of Bush (Bush Int ), and so
on. In general, the n-th entry (counting from 0) of a list of type Bush a has type
Bush n a.
The datatype List a is an example of a so-called regular datatype, while
Nest a and Bush a are examples of non-regular datatypes. Mycroft [17] calls
such schemes polymorphic recursions. We prefer the term nested datatypes. In a
regular datatype declaration, occurrences of the declared type on the right-hand
side of the defining equation are restricted to copies of the left-hand side, so
the recursion is “tail recursive”. In a nested datatype declaration, occurrences
of the datatype on the right-hand side appear with different instances of the
accompanying type parameter(s), so the recursion is “nested”.
In a language like Haskell or ML, with a Hindley-Milner type discipline, it is
simply not possible to define all the useful functions one would like over a nested
datatype, even though such datatype declarations are themselves perfectly legal.
This remark applies even to recent extensions of such languages (in particular,
Haskell 1.4), in which one is allowed to declare the types of problematic functions,
and to use the type system for checking rather than inferring types. To be sure, a
larger class of functions can now be defined, but one still cannot define important
generic functions, such as fold, over nested types.
On the other hand, the most recent versions of Hugs and GHC (the Glas-
gow Haskell Compiler) both support so-called rank-2 type signatures, in which
one can universally quantify over type constructors as well as types (see [20]).
By using such signatures one can construct most of the functions over nested
datatypes that one wants. We will return to this point below. However, rank-2
type signatures are not yet part of standard Haskell.
The upshot of the current situation is that nested datatypes have been rather
neglected in functional programming. However, they are conceptually important
and evidence is emerging (e.g. [3,18,19]) of their usefulness in functional data
structure design. A brief illustration of what they can offer is given in Section 2.
Regular datatypes, on the other hand, are the bread and butter of func-
tional programming. Recent work on polytypic programming (e.g. [2,9,15]) has
systematised the mathematics of program construction with regular datatypes
by focusing on a small number of generic operators, such as fold, that can be
defined for all such types. The basic idea, reviewed below, is to define a regular
datatype as an initial object in a category of F -algebras for an appropriate func-
tor F . Indeed, this idea appeared much earlier in the categorical literature, for
instance in [10]. As a consequence, polytypic programs are parametrised by one
or more regular functors. Different instances of these functors yield the concrete
programs we know and love.
54 Richard Bird and Lambert Meertens

The main aim of this paper is to investigate what form an appropriate func-
torial semantics for nested datatypes might take, thereby putting more ‘poly’
into ‘polytypic’. The most appealing idea is to replace first-order functors with
higher-order functors over functor categories. In part, the calculational theory
remains much the same. However, there are limitations with this approach, in
that some expressive power seems to be lost, and some care is needed in order
that the standard functorial semantics of regular datatypes may be recovered as
a special case. It is important to note that we will not consider datatype dec-
larations containing function spaces in this paper; see [6,16] for ways of dealing
with function spaces in datatype declarations.

2 An example
Let us begin with a small example to show the potential of nested datatypes.
The example was suggested to us by Oege de Moor. In the De Bruijn notation
for lambda expressions, bound variables introduced by lambda abstractions are
represented by natural numbers. An occurrence of a number n in an expression
represents the bound variable introduced by the n-th nested lambda abstraction.
For example, 0 (1 1) represents the lambda term
λ x .λy.x (y y)
On the other hand, 0 (w 1) represents the lambda term
λ x .λy.x (w y)
in which w is a free variable.
One way to capture this scheme is to use a nested datatype:
data Term a = Var a | App (Term a, Term a) | Abs (Term (Bind a))
data Bind a = Zero | Succ a
Elements of Term a are either free variables (of type Var a), applications, or
abstractions. In an abstraction, the outermost bound variable is represented by
Var Zero, the next by Var (Succ Zero), and so on. Free variables in an abstraction
containing n nested bindings have type Var (Succ n a). The type Term a is nested
because Bind a appears as a parameter of Term on the right-hand side of the
declaration.
For example, λ x .λy.x (w y) may be represented by the following term of type
Term Char :
Abs (Abs (App (Var Zero, App (Var (Succ (Succ ‘w’)), Var (Succ Zero)))))
The closed lambda terms – those containing no free variables – are elements of
Term Empty, where Empty is the empty type containing no members.
The function abstract, which takes a term and a variable and abstracts over
that variable, can be defined in the following way:
abstract :: (Term a, a) → Term a
abstract (t , x ) = Abs (lift (t, x ))
 Nested Datatypes 55

The function lift is defined by

lift :: (Term a, a) → Term (Bind a)
lift (Var y, x ) = if x = y then Var Zero else Var (Succ y)
lift (App (u, v), x ) = App (lift (u, x ), lift (v, x ))
lift (Abs t, x ) = Abs (lift (t, Succ x ))
The β-reduction of a term is implemented by
reduce :: (Term a, Term a) → Term a
reduce (Abs s, t) = subst (s, t )
where
subst :: (Term (Bind a), Term a) → Term a
subst (Var Zero, t ) =t
subst (Var (Succ x ), t ) = Var x
subst (App (u, v), t) = App (subst (u, t), subst (v, t))
subst (Abs s, t) = Abs (subst (s, term Succ t))
The function term f maps f over a term:
term :: (a → b) → (Term a → Term b)
term f (Var x ) = Var (f x )
term f (App (u, v)) = App (term f u, term f v)
term f (Abs t) = Abs (term (bind f ) t)
The subsidiary function bind f maps f over elements of Bind a:
bind :: (a → b) → (Bind a → Bind b)
bind f Zero = Zero
bind f (Succ x ) = Succ (f x )
It is a routine induction to show that
reduce (abstract (t , x ), Var x ) = t
for all terms t of type Term a and all x of type a.
Modulo the requirement that a and Bind a be declared as equality types
(because elements are compared for equality in the definition of lift ) the programs
above are acceptable to Haskell 1.4, provided the type signatures are included
as part of the definitions.

3 Datatypes as initial algebras

The standard semantics (see e.g. [8,10]) of inductive datatypes parametrised by
n type parameters employs functors of type C×· · · ×C → C, where the product
has n + 1 occurrences of C. For simplicity, we will consider only the case n = 1.
The category C cannot be arbitrary: essentially, it has to contain finite sums and
products, and colimits of all ascending chains. The category Fun (also known
56 Richard Bird and Lambert Meertens

as Set), whose objects are sets and whose arrows are typed total functions, has
everything needed to make the theory work.
To illustrate, the declaration of List as a datatype is associated with a binary
functor F whose action on objects of C × C is defined by
F (a, b) = 1 + a × b
Introducing the unary functor Fa , where Fa (b) = F (a, b), the declaration of
List a can now be rewritten in the form
α
data List a ←−
a
Fa (List a)
in which αa :: Fa (List a) → List a. For the particular functor F associated
with List , the arrow αa takes the form (NilLa , ConsLa ), where NilLa :: 1 →
List a and ConsLa :: a × List a → List a. This declaration can can be
interpreted as the assertion that the arrow αa and the object List a are the
“least” values with this typing. More precisely, given any arrow
f :: Fa (b) → b
the assertion is that there is a unique arrow h :: List a → b satisfying the
equation
h · αa = f · F (ida , h)
The unique arrow h is denoted by fold f . The arrow h is also called a catamor-
phism, and the notation ([f ]) is also used for fold f . In algebraic terms, List a is
the carrier of the initial algebra αa of the functor Fa and fold f is the unique
Fa -homomorphism from the initial algebra to f .
A surprising number of consequences flow from this characterisation. In par-
ticular, fold αa is the identity arrow on List a. Also, one can show that αa is an
isomorphism, with inverse fold (F (ida , αa )). As a result, one can interpret the
declaration of List as the assertion that, up to isomorphism, List a is the least
fixed point of the equation x = F (a, x ).
The type constructor List can itself can be made into a functor by defining
its action on an arrow f : a → b by
list f = fold (αb · F (f , id))
In functional programming list f is written map f . Expanding the definition of
fold, we have
list f · αa = αb · F (f , list f )
This equation states that α is a natural transformation of type α :: G → List,
where G a = F (a, List a).
The most important consequence of the characterisation is that it allows one
to introduce new functions by structural recursion over a datatype. As a simple
example, fold (zero, plus) sums the elements of a list of numbers.
Functors built from constant functors, type functors (like List), the identity
and projection functors, using coproduct, product, and composition operations,
 Nested Datatypes 57

are called regular functors. For further details of the approach, consult, e.g., [12]
or [1].
For Nest and Bush the theory above breaks down. For example, introducing
Q a = a × a for the squaring functor, the corresponding functorial declaration
for Nest would be
α
data Nest a ←−
a
F (a, Nest (Q a))
where F is as before, and αa applies NilN to left components and ConsN to
right components. However, it is not clear over what class of algebras αa can be
asserted to be initial.

4 A higher-order semantics
There is an appealing semantics for dealing with datatypes such as Nest and
Bush, which, however, has certain limitations. We will give the scheme, then
point out the limitations, and then give an alternative scheme that overcomes
some of them.
The idea is to use higher-order functors of type
Nat(C) → Nat(C),
where Nat (C) is the category whose objects are functors of type C → C
and whose arrows are natural transformations. We will use calligraphic letters
for higher-order functors, and small Greek letters for natural transformations.
Again, the category C cannot be arbitrary, but taking C = Fun gives everything
one needs. Here are three examples.

Example 1. The declaration of List can be associated with a higher-order functor

F defined on objects (functors) by
F(F )(a) = 1 + a × F (a)
F(F )(f ) = id1 + f × F (f )
These equations define F(F ) to be a functor for each functor F . The functor F
can be expressed more briefly in the form
F(F ) = K 1 + Id × F
The constant functor K a delivers the object a for all objects and the arrow
ida for all arrows, and Id denotes the identity functor. The coproduct (+) and
product (×) operations are applied pointwise.
The action of F on arrows (natural transformations) is defined in a similar
style by
F(η) = idK 1 + id × η
Here, idK 1 delivers the identity arrow id1 for each object of C. If η :: F → G,
then F(η) :: F(F ) → F(G). We have F(id) = id, and F(η · ψ) = F(η) · F(ψ),
so F is itself a functor.
58 Richard Bird and Lambert Meertens

The previous declaration of List can now be written in the form

α
data List ←− F(List)

and interpreted as the assertion that α is the initial F-algebra.

Example 2. The declaration of Nest is associated with a functor F, defined on

objects (functors) by

F(F )(a) = 1 + a × F (Qa)

F(F )(f ) = id1 + f × F (Qf )

where Q is the squaring functor. More briefly,

F(F ) = K 1 + Id × (F · Q)

where F · Q denotes the (functor) composition of F and Q. Where convenient,

we will also write this composition as FQ for brevity.
The action of F on arrows (natural transformations) is defined by

F(η) = idK 1 + id × ηQ

where ηQ :: FQ → GQ if η :: F → G.

Example 3. The declaration of Bush is associated with a functor F, defined on

functors by

F(F ) = K 1 + Id × (F · F )

and on natural transformations by

F(η) = idK 1 + id × (η ? η)

The operator ? denotes the horizontal composition of two natural transforma-

tions. If θ :: F → G and ψ :: H → N , then θ ? ψ :: FH → GN is defined by
θ ? ψ = θN · F ψ. In particular, if η :: F → G, then η ? η :: FF → GG.
Consider again the declaration of Nest given in the Introduction, and rewrite
it in the form
α
data Nest ←− F(Nest)

The assertion that α is the initial F-algebra means that for any arrow ϕ ::
F(F ) → F , there is a unique arrow θ :: Nest → F satisfying the equation

θ · α = ϕ · F(θ).

The unique arrow θ is again denoted by fold ϕ.

We can express the equation above in Haskell. Note that for the particular
functor F associated with Nest, the arrow ϕ takes the form ϕ = (ε, ψ), where
ε :: K 1 → F and ψ :: Id × FQ → F . For any type a, the component εa is
 Nested Datatypes 59

an arrow delivering a constant e of type F a, while ψa is an arrow f of type

(a, F (a, a)) → F (a). Hence we can write
fold (e, f ) NilN =e
fold (e, f ) (ConsN (x , xps)) = f (x , fold (e, f ) xps)
However, no principal type can be inferred for fold under the Hindley-Milner type
discipline, so the use of fold in programs is denied us. Moreover, it is not possible
to express the type of fold in any form that is acceptable to a standard Haskell
type checker. On the other hand, in GHC (The Glasgow Haskell Compiler) one
can declare the type of fold by using a rank-2 type signature:
fold :: (∀f .∀b. ((∀a.f a), (∀a.(a, f (a, a)) → f a)) → Nest b → f b)
This declaration uses both local universal quantification and abstraction over a
type constructor. Such a signature is called a rank-2 type signature. With this
asserted type, the function fold passes the GHC type-checker.
Observe that in the proposed functorial scheme, unlike the previous one for
regular datatypes, the operator fold takes natural transformations to natural
transformations. In particular, the fact that Nest is a functor is part of the
assertion that Nest is the least fixed point of F. The arrow nest f cannot be
defined as an instance of fold since it is not a natural transformation of the right
type.
The typing α :: F(Nest) → Nest means that, given f :: a → b, the following
equation holds:
nest f · αa = αb · F(nest) f
We can express this equation at the point level by
nest f NilN = NilN
nest f (ConsN (x , xps)) = ConsN (f x , nest (square f ) xps)
where square f (x , y) = (f x , f y) is the action on arrows of the functor Q. The
fact that nest is uniquely defined by these equations is therefore a consequence
of the assertion that α is a natural transformation.
Exactly the same characterisation works for Bush. In particular, the arrow
bush f satisfies
bush f NilB = NilB
bush f (ConsB (x , xbs)) = ConsB (f x , bush (bush f ) xbs)

5 Examples
To illustrate the use of folds over Nest and Bush, define τ :: Q → List by
τ (x , y) = [x , y]
Using τ and the natural transformation concat :: List · List → List, we have
concat · list τ :: List · Q → List, and so
αList · F(concat · list τ ) :: F(List) → List
60 Richard Bird and Lambert Meertens

where F(F ) = K 1 + Id × FQ is the higher-order functor associated with Nest.

The function listify, defined by
listify = fold (αList · F(concat · list τ ))

therefore has type listify :: Nest → List. For example, listify takes
[0, (1, 1), ((2, 2), (3, 3))] to [0, 1, 1, 2, 2, 3, 3]

The converse function nestify :: List → Nest can be defined by

nestify = fold (αNest · F(nest δ))

where F(F ) = K 1 + Id × F is the higher-order functor associated with List, and

δ a = (a, a) has type δ :: Id → Q. For example, nestify takes
[0, 1, 2] to [0, (1, 1), ((2, 2), (2, 2))]

For another example, define σ :: Q → Bush by

σ(x , y) = [x , [y]]

Then bush σ :: Bush · Q → Bush · Bush, and so

αBush · F(bush σ) :: F(Bush) → Bush

where F(F ) = K 1 + Id × FQ is the functor associated with Nest. Hence

bushify = fold (αBush · F(bush σ))

has type bushify :: Nest → Bush. For example, bushify sends

[1, (2, 3), ((4, 5), (6, 7))] to [1, [2, [3]], [[4, [5]], [[6, [7]]]]]

6 The problem

The basic problem with the higher-order approach described above concerns
expressive power. Part of the problem is that it does not generalise the standard
semantics for regular datatypes; in particular, it does not enable us to make use
of the standard instances of fold over such datatypes. To see why not, let us
compare the two semantics for the datatype List .
Under the standard semantics, fold f :: List a → b when f :: 1 + a × b → b.
For example,

fold (zero, plus) :: List Int → Int

sums a list of integers, where zero :: 1 → Int is a constant delivering 0, and
plus :: Int × Int → Int is binary addition.
As another example,

fold (nil , cat) :: List (List a) → List a

 Nested Datatypes 61

concatenates a list of lists; this function was called concat above. The binary
operator cat has type cat :: List a × List a → List a and concatenates two lists.
Under the new semantics, fold ϕ :: List → F when ϕ :: K 1 + Id × F → F .
We can no longer sum a list of integers with such a fold because plus is not a
natural transformation of the right type. For fold (zero, plus) to be well-typed
we require that plus has type plus :: Id × KInt → KInt . Thus,
plusa :: a × Int → Int
for all a, and so plus would have to ignore its first argument.
Even worse, we cannot define concat :: List · List → List as an instance
of fold, even though it is a natural transformation. The binary concatenation
operator cat does not have type
cat :: Id × List → List
because again it would have to ignore its first argument. Hence fold (nil , cat ) is
not well-typed.
On the other hand, αNest · F(nest δ) does have type K 1 + Id × Nest → Nest,
so the definition of nestify given in the previous section is legitimate.
Putting the problem another way, in the standard semantics, fold f is defined
by providing an arrow f :: F (a, b) → b for a fixed a and b; we cannot in general
elevate f to a natural transformation that is parametric in a.

7 An alternative
Fortunately, for lists and other regular datatypes, there is a way out of this
particular difficulty. Using the isomorphism defining List, the functor List · F
satisfies the isomorphism
List · F ∼
= (K 1 + Id × List) · F ∼
= K 1 + F × (List · F )
Hence List · F is isomorphic to the “higher-order” datatype Listr F , declared by
α
data Listr F ←− K 1 + F × Listr F
We can write the functor on the right as F(F , Listr F ), where F now is a higher-
order binary functor of type
Nat (C) × Nat (C) → Nat(C)
Over the higher-order datatype Listr F , the natural transformation fold ϕ takes
an arrow ϕ :: K 1+F ×G → G, and has type fold ϕ :: Listr F → G. If we change
Listr F to List · F in this signature, we have a useful fold operator for lists. In
particular,
fold (zero, plus) :: List · KInt → KInt
since (zero, plus) :: K 1 + KInt × KInt → KInt . The arrow fold (zero, plus) of
Nat(C) is a natural transformation; since List · KInt = K (List Int), its compo-
nent for any a is the standard fold fold (zero, plus) :: List Int → Int.
62 Richard Bird and Lambert Meertens

By a similar device, all folds in the standard semantics are definable as folds
in the new semantics, simply by lifting the associated algebra to be a natural
transformation between constant functors.
More precisely, define Type a to be the least fixed point of a regular func-
tor Fa , where Fa (b) = F (a, b). Furthermore, define Typer G to be the least
fixed point of FG , where FG (H ) = F(G, H ) and F(G, H )x = F (Gx , Hx ) for all
objects x . Take an algebra f :: F (a, b) → b, and construct the natural transfor-
mation ϕ :: F(Ka, Kb) → Kb by setting ϕ = Kf . This is type correct since
F(Ka, Kb)x = F (Ka(x ), Kb(x )) = F (a, b) and Kb(x ) = b
Then fold f :: Type a → b, and fold ϕ :: Typer Ka → Kb satisfy
fold ϕ = K (fold f )
under the isomorphism Typer Ka = K (Type a).
Thus, not only do we generalise from the defining expression for List by
replacing occurrences of List by G, we also generalise by replacing occurrences
of Id by a functor F .
However, the same idea does not work for nested datatypes such as Nest.
This time we have
Nest · F ∼
= (K 1 + Id × (Nest · Q)) · F ∼
= K 1 + F × (Nest · Q · F )
The type Nest · F is quite different from the datatype defined by
α
data Nestr F ←− K 1 + F × ((Nestr F ) · Q)
For example, Nest (List a) is the type of nests of lists over a, so the n-th entry
of such a nest has type Q n (List a). On the other hand the n-th entry of a nest
of type Nestr List a has type List (Q n a).
Even more dramatically, the type Nest Int gives a nest of integers, but
Nestr KInt b is isomorphic to ordinary lists of integers for all b. More gener-
ally, Nestr Ka is the constant functor K (List a).
On the other hand, we have Nest = Nestr Id, so the higher-order view is
indeed a generalisation of the previous one.

8 Reductions
Replacing higher-order unary functors by higher-order binary functors enables us
to integrate the standard theory of regular datatypes into the proposed scheme.
Unfortunately, while the higher-order approach is elegant and generic, it seems
limited in the scope of its applicability to nested datatypes, which is restricted
to folding with natural transformations. For example, one cannot sum a nest of
integers with a fold over nests. Such a computation is an instance of a useful
general pattern called a reduction. It is possible to define reductions completely
generically for all regular types (see [15]), but we do not know at present whether
the same can be done for nested datatypes.
 Nested Datatypes 63

One way to sum a nest of integers is by first listifying the nest and then
summing the result with a fold over lists. More generally, this strategy can be
used to reduce a nest with an arbitrary binary operator ⊕ and a seed e. For
example,

[x0 , (x1 , x2 ), ((x3 , x4 ), (x5 , x6 ))]

reduces to

x0 ⊕ (x1 ⊕ (x2 ⊕ · · · ⊕ (x6 ⊕ e)))

It can be argued that this strategy for reducing over nests is unsatisfactory
because the structure of the nest entries is not reflected in the way in which ⊕
is applied. Better is to introduce a second operator ⊗ and reduce the nest above
to

x0 ⊗ ((x1 ⊕ x2 ) ⊗ (((x3 ⊕ x4 ) ⊕ (x5 ⊕ x6 )) ⊗ e))

By taking ⊗ to be ⊕, we obtain another way of reducing a nest.

The above pattern of computation can be factored as a fold over lists after a
reduction to a list:

fold (e, ⊗) · reduce (⊕)

With (⊕) :: Q a → a, the function reduce (⊕) has type Nest a → List a. For
example, applied to the nest above, reduce (⊕) produces

[x0 , x1 ⊕ x2 , (x3 ⊕ x4 ) ⊕ (x5 ⊕ x6 )]

There is no problem with defining reduce. In a functional style we can define

reduce op NilN = NilL

reduce op (ConsN (x , xps)) = ConsL (x , reduce op (nest op xps))

In effect, reduce op applies the following sequence of functions to the correspond-

ing entries of a nest:

[id, op, op · square op, op · square op · square (square op), . . .]

The n-th element of this sequence has type Q n a → a when op :: Q a → a.

The reduction of a bush proceeds differently:

reduce (e, op) NilB = e

reduce (e, op) (ConsB (x , xbs)) =
op (x , reduce (e, op) (bush (reduce (e, op)) xbs))

At present we see no systematic way of unifying reductions over nested datatypes,

nor of relating them to the folds of previous sections.
64 Richard Bird and Lambert Meertens

9 Another approach
There is a way that higher-order folds and the reductions of the previous section
can be unified, but whether or not the method is desirable from a calculational
point of view remains to be seen. It requires a different and more complicated
notion of folding over a nested datatpe, one that involves an infinite sequence of
appropriate algebras to replace the infinite sequence of differently typed instances
of the constructors of the datatype. We will briefly sketch the construction for
the type Nest a.
The basic idea is to provide an infinite sequence of algebras to replace the
constructor α = (NilN , ConsN ) of Nest, one for each instance
α :: F (Q n a, Nest (Q n+1 a)) → Nest (Q n a)
where n is a natural number and F (a, b) = 1 + a × b. For regular datatypes the
application of fold f to a term can be viewed as the systematic replacement of
the constructors by corresponding components of f , followed by an evaluation of
the result. The same idea is adopted here for nested datatypes. However, whereas
for regular datatypes each occurrence of a constructor in a term has the same
typing, the same is not true for nested datatypes, hence the need to provide a
collection of replacements.
In more detail, consider the datatype NestAlgs defined by
data NestAlgs G (a, b) = Cons (F (a, G(Qb)) → Gb, NestAlgs G (Qa, Qb))
The datatype NestAlgs is a coinductive, infinite, nested datatype. The n-th entry
of a value of type NestAlgs G (a, b) is an algebra of type
F (Q n a, G(Q n+1 b)) → G(Q n b)
Now for fs :: NestAlgs G (a, b), define fold fs :: Nest a → Gb by the equation
fold fs · α = head fs · F (id, fold (tail fs))
where
head (Cons (f , fs)) = f
tail (Cons (f , fs)) = fs
Equivalently,
fold (Cons (f , fs)) · α = f · F (id, fold fs)
To illustrate this style of fold, suppose f :: a → b and define generate f ::
NestAlgs Nest (a, b) by
generate f = Cons (α · F (f , id), generate (square f ))
Then fold (generate f ) :: Nest a → Nest b, and in fact
nest f = fold (generate f )
 Nested Datatypes 65

The functorial action of Nest on arrows can therefore be recovered as a fold. The
proof of nest (f · g) = nest f · nest g makes use of coinduction.
As another example, suppose ϕ :: F(Id, GQ) → G is a natural transforma-
tion, where F(M , N )a = F (Ma, Na). Define repeat ϕ :: NestAlgs G by
repeat ϕ = Cons (ϕ, repeat ϕQ)
For each type a we have (repeat ϕ)a :: NestAlgs G (a, a). The relationship be-
tween the higher-order folds of the previous sections and the current style of
folds is that
fold ϕ = fold (repeat ϕ)
In particular, fold (repeat α) = id :: Nest → Nest.
We can also define reductions as an instance of the new folds. Suppose
f :: F (a, a) → a, so f = (f0 , f1), where f1 :: Qa → a. Define redalgs f ::
NestAlgs Ka (a, b) by
redalgs f = red id
where red k = Cons (f · F (k, id), red (f1 · square k))
We have fold (redalgs f ) :: Nest a → a, and we claim that
reduce f = fold (redalgs f )

10 Conclusions
The results of this investigation into nested datatypes are still incomplete and
in several aspects unsatisfactory. The higher-order folds are attractive, and the
corresponding calculational theory is familiar, but they seem to lack sufficient
expressive power. The approach sketched in the previous section for Nest is more
general, but brings in more machinery. Furthermore, it is not clear what the right
extension is to other nested datatypes such as Bush.
We have also ignored one crucial question in the foregoing discussion, namely,
what is the guarantee that functors such as Nest and Nestr do in fact exist
as least fixed points of their defining equations? The categorical incantation
ensuring the existence of an initial F -algebra in a co-complete category C is
that, provided F is co-continuous, it is the colimit of the chain
0 ,→ F 0 ,→ FF 0 ,→ · · ·
The category Fun has everything needed to make this incantation work: Fun
is co-complete (in fact, bi-complete) and all regular functors F on Fun are co-
continuous. The proof for polynomial functors can be found in [14], and the
extension to type functors is in [13].
Moreover, the category Nat (Fun) inherits co-completeness from the base
category Fun (see [11,7]). We believe that all regular higher-order functors are
co-continuous, though we have not yet found a proof of this in the literature, so
the existence of datatypes like Nest and Bush is not likely to be problematic.
66 Richard Bird and Lambert Meertens

If we adopt the higher-order approach, then there is a need to give a system-

atic account of reductions over a nested datatype. If the alternative method of
the previous section proves more useful, then there is a need to give a systematic
account of the method, not only for an arbitrary inductive nested datatype, but
also for coinductive nested datatypes.
Finally, in [4] (see also [5]) the idea was proposed that a datatype was a
certain kind of functor called a relator, together with a membership relation.
It needs to be seen how the notion of membership can be extended to nested
datatypes

Acknowledgements
The authors would like to thank Ian Bayley, Jeremy Gibbons, Oege de Moor,
Mark Jones, and Simon Peyton Jones for comments and discussions on the work.
A particular debt is owed to Ross Paterson, who commented on an earlier draft
of the paper. Thanks are also due to the anonymous referees who suggested
numerous improvements.

References
1. R. Bird and O. de Moor. Algebra of Programming. International Series in Com-
puting Science. Prentice Hall, 1996.
2. R. S. Bird, P. F. Hoogendijk, and O. De Moor. Generic programming with relations
and functors. Journal of Functional Programming, 6(1):1–28, 1996.
3. R.H. Connelly and F. Lockwood Morris. A generalisation of the trie data structure.
Mathematical Structures in Computer Science, 5(3):381–418, 1995.
4. Oege de Moor and Paul Hoogendijk. What is a datatype? Technical Report
96/16, Department of Maths and Computing Science, Eindhoven University of
Technology, 1996.
5. Paul Hoogendijk. A Generic theory of Data Types. Ph.D Thesis, Eindhoven
University of Technology, 1997.
6. L. Fegaras and T. Sheard. Revisiting catamorphisms over datatypes with em-
bedded functions. In 23rd ACM SIGPLAN-SIGACT Symposium on Principles of
Programming Languages. Association for Computing Machinery, 1996.
7. Peter Freyd. Algebraically complete categories. Springer-Verlag Lecture Notes in
Mathematics, vol 1488, 95–104, 1990.
8. T. Hagino. Category theoretic approach to data types. PhD thesis, Laboratory for
Foundations of Computer Science, University of Edinburgh, UK, 1987. Technical
Report ECS-LFCS-87-38.
9. J. Jeuring. Polytypic pattern matching. In S. Peyton Jones, editor, Functional Pro-
gramming and Computer Architecture, pages 238–248. Association for Computing
Machinery, 1995.
10. J. Lambek. A fixpoint theorem for complete categories. Mathematische Zeitschrift,
103:151–161, 1968.
11. Saunders Mac Lane. Categories for the Working Mathematician. Graduate Texts
in Mathematics. Springer-Verlag, 1971.
12. G. Malcolm. Data structures and program transformation. Science of Computer
Programming, 14(2–3):255–279, 1990.
 Nested Datatypes 67

13. G. Malcolm. Algebraic Data Types and Program Transformation. Ph.D thesis,
University of Groningen, The Netherlands, 1990.
14. E.G. Manes and M.A. Arbib. Algebraic Approaches to Program Semantics. Texts
and Monographs in Computing Science. Springer-Verlag, 1986.
15. Lambert Meertens. Calculate polytypically! In Herbert Kuchen and S. Doaitse
Swierstra, editors, Programming Languages: Implementations Logics, and Pro-
grams Proceedings Eighth International Symposium PLILP ’96, volume 1140 of
LNCS, pages 1–16. Springer-Verlag, 1996.
16. E. Meijer and G. Hutton. Bananas in space: extending fold and unfold to ex-
ponential types. In S. Peyton Jones, editor, Functional Progamming Languages
and Computer Architecture, pages 324–333. Association for Computing Machinery,
1995.
17. A. Mycroft. Polymorphic type schemes and recursive definitions. In International
Symposium on Programming, volume LNCS 167, pages 217–228. Springer-Verlag,
1984.
18. C. Okasaki. Purely Functional Data Structures. Ph.D thesis, School of Computer
Science, Carnegie Mellon University, 1996.
19. C. Okasaki. Catenable double-ended queues. In Proceedings of the 1997 ACM
SIGPLAN International Conference on Functional Programming (ICFP ’97), pages
66–74. ACM, 1997.
20. S. Peyton Jones and J. Launchbury. Explicit quantification in Haskell. See:
http://www.dcs.gla.ac.uk/people/personal/simonpj/.