Meta-operators for Enabling Parallel Planning

Using Deep Reinforcement Learning

Ángel Aso-Mollar1 , Eva Onaindia1
1
Valencian Research Institute for Artificial Intelligence (VRAIN),
Universitat Politècnica de València (UPV)
[email protected], [email protected]
arXiv:2403.08910v1 [cs.AI] 13 Mar 2024

Abstract is also known that some works attempt to overcome these

problems using, for instance, Reward Machines (Icarte et al.
There is a growing interest in the application of Reinforce-
2022).
ment Learning (RL) techniques to AI planning with the aim to
come up with general policies. Typically, the mapping of the In addition, it is common for papers that use RL with AI
transition model of AI planning to the state transition system planning to make use of the very action definition in plan-
of a Markov Decision Process is established by assuming a ning by applying it directly into Reinforcement Learning:
one-to-one correspondence of the respective action spaces. In to establish the mapping between the transition model of
this paper, we introduce the concept of meta-operator as the AI planning and the state transition system of a Markov
result of simultaneously applying multiple planning opera- Decision Process, it is assumed that there is a one-to-one
tors, and we show that including meta-operators in the RL ac- correspondence between their respective action spaces. This
tion space enables new planning perspectives to be addressed is what we refer to as the 1-1 mapping between planning
using RL, such as parallel planning. Our research aims to
and RL actions. In this paper, we propose a new perspec-
analyze the performance and complexity of including meta-
operators in the RL process, concretely in domains where sat- tive based on the concept of meta-operator, with which we
isfactory outcomes have not been previously achieved using decouple this 1-1 mapping often seen in the literature, there-
usual generalized planning models. The main objective of this fore allowing to apply several actions at the same time (in
article is thus to pave the way towards a redefinition of the RL parallel) in a concrete time step. Parallel plans increase the
action space in a manner that is more closely aligned with the efficiency of planning because increased parallelism leads to
planning perspective. a reduction of the plan length or the number of time points.
In this paper, we also investigate if general parallel policies
can be obtained with a generalized planning framework and
Introduction if such policies improve general sequential policies.
Integrating AI planning and Machine Learning algorithms By defining the concept of meta-operator as the applica-
is a hot research topic that has reported significant advances tion of several planning actions at the same time, we can
in many different scenarios, such as training general poli- force the RL training process to simulate parallelism. In ad-
cies with supervised or unsupervised Learning, (Ståhlberg, dition, we can also use meta-operators to further guide the
Bonet, and Geffner 2022a), (Ståhlberg, Bonet, and Geffner training process, alleviating the sparse reward problem by
2022b), Generalized Planning using Reinforcement Learn- establishing a small reward when the meta-operator contains
ing (RL), (Gehring et al. 2022), (Rivlin, Hazan, and Karpas more than one atomic action.
2020), (Francès et al. 2019), learning Action Models using We incorporate meta-operators in the context of Gener-
Reinforcement Learning (Ng and Petrick 2019), and many alized Planning and to this aim our approach is based on
others. a modified version of the architecture proposed in (Rivlin,
Specifically, strands of this discipline have integrated AI Hazan, and Karpas 2020), where Graph Neural Networks
Planning with RL usually to train, with synthesized experi- are used to generate a compact representation of the plan-
ence, heuristics that guide the search process of a planning ning states. We will report results showing that the inclusion
agent, through rectification of domain-independent heuris- of meta-actions allows almost a 100% coverage in domains
tics (Gehring et al. 2022) or through direct policy training where it is difficult to reach generalization with traditional
(Rivlin, Hazan, and Karpas 2020), for example. The training learning methods such as logistics or depots, as stated in
process is usually based on giving the agent a reward of 1 (Ståhlberg, Bonet, and Geffner 2022a).
when it reaches the goal of the planning problem and 0 oth- We tested our models in problem instances from the In-
erwise. This introduces the so-called sparse reward problem ternational Planning Competitions1 (IPC) and in randomly
wherein we observe that the agent rarely receives rewards generated ones, and we observed that the coverage using
from the environment, thus hindering its ability to learn. It meta-operators improves with respect to not using them. It
Copyright © 2023, Association for the Advancement of Artificial
1
Intelligence (www.aaai.org). All rights reserved. https://www.icaps-conference.org/competitions/
is also observed that the length of the plan, as expected, is executing actions that change its internal state, and these ac-
also reduced when we include meta-operators. tions are rewarded consequently. The main objective of the
This paper is structured as follows: several basic concepts agent is to improve its performance so that it can progres-
with which we rely our work on are going to be defined, sively maximize the received reward.
such as Planning, RL and their integration, including Gener- Formally, an MDP is defined as M = hS, A, R, P ri,
alized Planning. Next, the core element of this article will be where S is a set of states, A is a set of actions, R : S ×
defined: the meta-operator, and then will be integrated into A × S → R is a reward function that values how good or
the RL structure. Finally, several different experiments that bad it is to take an action at at a certain state st that leads
justify the use of meta-operators in RL will be conducted, to another state st+1 , R(st , at , st+1 ) = rt , and P r is a tran-
followed by a discussion of whether to use this mechanism sition probability function such as P r(s′ , s, a) = p(s′ |s, a),
and the benefits against other models that do not include this which is usually unknown.
approach. At each time step t, an agent takes an action in the cur-
rent state st among a set of applicable actions At ⊆ A in
Background st following a probability distribution called policy, repre-
sented by π : S → p(·| S = st ). The objective of RL is to
Planning reward decisions that reach the target as quickly as possible,
Classical planning is the problem of finding a sequence by prioritizing immediate rewards.
of actions that when applied to an initial state lead to the The RL optimization problem is formally defined as
achievement of a goal or set of goals. The two main compo- max Eπ Gt (1)
nents of a planning task are the domain and the problem. π
A planning domain is defined as D = hF , Ai, where P∞ k
where Gt = k=0 γ rt+k+1 is the sum of the discounted
F is a set of fluents that describe lifted properties of the reward of a trajectory sampled from the policy π from the
domain objects and their relations, and A is a set of ac- current state t to the infinite. Thus, formula (1) measures the
tion schemas in which every a ∈ A is defined by a triplet reward received by all possible trajectories multiplied by the
hPre(a), Add(a), Del(a)i, Pre(a), Add(a), Del(a) ⊆ F . probability of taking the decisions of the trajectories under
A planning problem instance linked to a domain D = the policy π, where γ ∈ [0, 1] is the discount factor of future
hF , Ai is defined as a tuple P = hF, O, I, Gi where F is rewards.
the set of facts that result from grounding the fluents F to
the objects of the problem P , and O, called operators, is the Integrating RL on top of planning
result of grounding the action schemas A. A state s ⊆ F is The integration of Planning and RL is a promising approach
a set of facts, and I, G ⊆ F are, respectively, the initial and for solving control problems. Planning is effective to reason
the goal state of the problem. over a long horizon but assumes access to a local schema; in
Every operator o ∈ O has its corresponding grounded contrast, RL is effective at learning policies and relative val-
triplet defined by hP re(o), Add(o), Del(o)i. P re(o) ⊆ F ues of states but fails to plan over long horizons (Eysenbach,
are the preconditions that must be true in a state for the op- Salakhutdinov, and Levine 2019). Combining planning and
erator to be applicable; that is, an operator o is applicable in RL can thus be regarded as learning a control policy for the
a state s ⊆ F if P re(o) ⊆ s. Add(o) ⊆ F are the effects planner in lieu of using a planning heuristic function.
of the operator that assert a positive literal after the appli- Combining Planning and RL enables addressing a vari-
cation of o; and Del(o) ⊆ F is the set of negative effects, ety of tasks like learning generalist policies to solve a large
i.e., the set of literals which become false after the opera- number of problems (Celorrio, Aguas, and Jonsson 2019),
tor is applied. Applying o ∈ O in s leads to a new state learning action models based on interaction (Moerland et al.
s′ = (s \ Del(o)) ∪ Add(o)). 2023), training intelligent heuristics that guide the agent in
A solution to a planning problem P = hF, O, I, Gi is a the control task (Gehring et al. 2022), or even training the
sequence of operators or plan ρ = ho1 , o2 , . . . , ok i, oi ∈ agent to solve concrete planning problems. Generally, the
O ∀i, such that the result of applying the operator sequence RL module is integrated inside the planning agent to help it
ho1 , o2 , . . . , ok i to the initial state I leads to a state s ⊆ F make a decision, although other approaches use it the oppo-
that satisfies G ⊆ s, that is to say, that the state s contains site way (Lee et al. 2022). In this work, we will follow the
every goal in G. first approach.
As the formulation of both RL and Planning use the con-
Reinforcement Learning cept of state and action, we must establish the equivalence
Reinforcement Learning (RL) is a computational approach between RL actions and planning operators, so that the plan-
to learning from environmental interaction (Sutton and ning and RL modules can communicate with each other.
Barto 1998). The main objective of RL is to learn a policy, It is usually assumed in the literature that for a planning
or behaviour, that maximizes the positive interaction that an problem P = hF, O, I, Gi the equivalence is established by
agent gets through time. simply considering the planning operators O as the RL ac-
RL scenarios are often modeled as finite Markov Decision tions A (A = O). States in RL (S) are abstract concepts that
Processes (MDP) (Puterman 1990). An MDP is a control simply represent situations that occur in the environment, so
process that stochastically models decision-making scenar- we can inherit the concept of planning state and apply it di-
ios; an agent continuously interacts with an environment by rectly to RL by making every s ∈ S as s ⊆ F ; i.e., we can
assume RL states as planning states. Hence, the notion of an Defining meta-operators
action being applicable to a state s in RL follows directly
from the above considerations and uses the same principle As we discussed in the previous section, a bijective corre-
of applicability of an operator in the planning state s. spondence is usually considered between the set of planning
operators and the set of RL actions when solving a planning
Since this one-to-one correspondence between actions problem P = hF, O, I, Gi of a domain D. That is, A = O.
and operators yields huge action spaces, some researchers We have also seen that we can transform this operator space
adopt certain considerations to reduce the number of possi- into a different one g(O). In this work, unlike PDDLGym
ble actions. For example, in PDDLGym (Silver and Chitnis that performs a reduction of the operator space, we will de-
2020), authors consider only a partial grounding of the ac- fine a transformation function to enrich it.
tion schema in order to reduce the number of possible ac-
tions. Particularly, they claim that certain operators contain For this purpose, we define the notion of a meta-operator
parameters that correspond to free objects (in terms of con- for a problem P = hF, O, I, Gi as a new operator that, given
trolling the agent), that is, objects whose properties can po- L > 1 and o1 , ..., oL ∈ O, which we will refer to as atomic
tentially change, leaving ungrounded the non-free agents. operators, is represented as
For example, in a transport domain in which we want to L
transport packages between cities using trucks, the pack-
M
oi
ages and trucks would be free objects because their loca-
i=1
tion changes, but the cities would not. In this scenario, an
action schema like (move ?t - truck ?c1 - city with the following considerations:
?c2 - city) will only bound the parameter truck to a L  S
concrete object as the two parameters city refer to static L L
• P re i=1 oi = i=1 P re(oi )
objects that only denote the structure of the movement net-
work.
L  S
L L
• Add i=1 oi = i=1 Add(oi )
We can think of this consideration as a transformation g
of the planning operator space, g(O), which is then used as
L  S
L L
the RL action space, A = g(O). Note that in PDDLGym this
• Del i=1 oi = i=1 Del(oi )

specification is hard-coded, while we propose an automated • No pair of actions oi , oj from all the atomic actions
method, as we will discuss later. o1 , ..., oL that form the meta-operator conflict with each
We already mentioned that the notion of a planning state is other. Two atomic operators oi and oj conflict with each
directly translated into an RL state. We need to take into ac- other if one of these two conditions hold:
count that sometimes, and depending on the RL method, the
planning states must be translated into some encoding that – ∃ p ∈ P re(oi ) such that p ∈ Del(oj ).
the RL method supports. For example, if we are using a NN – ∃ p ∈ Add(oi ) such that p ∈ Del(oj ).
to represent the policy, we need a vectorial representation of
the state. This representation can be obtained, for example, Broadly speaking, a meta-operator is nothing but a syn-
by using Graph Neural Networks (Zhou et al. 2020) or Neu- thetic operator resulting from the union of atomic ones that
ral Logic Machines (Dong et al. 2019), in conjunction with can be executed in any order. The resulting operator will
intermediate structures such as graphs. therefore inherit the union of its sets Add, P re and Del, al-
ways bearing in mind that all the atomic operators involved
Sometimes we will say that we apply a learned policy π to
can be executed at the same time, that is, that they do not
a planning problem P , meaning that the planning operator
conflict with each other.
corresponding to the RL action given by the policy is suc-
cessively applied from the initial state of the problem to the This means that they are inconsistent if they compromise
goal, and states will change accordingly through the plan- the consistency of the resulting state when applying the op-
ning operators application. This way, as mentioned above, erator, i.e., if the resulting state changes if the sequential or-
the planning states are mapped to the RL states and (hege- der of application of the atomic operators changes. These
monically) planning operators are mapped to RL actions. two considerations are equivalent to the notions of incon-
sistent effects and interference in the calculation of a mutex
realtion between two actions in GraphPlan (Blum and Furst
Generalized Planning 1997).
We define the set of meta-operators of degree L ∈ N in
Generalized Planning is a discipline in which we aim to a problem P of a domain D as the union of every possible
find general policies that explain a set of problems, usu- meta-operation:
ally of different size. Specifically, given a set of problems
{hFi , Oi , Ii , Gi i}N
i=1 , N > 0, rather than searching for a so- [ M
" L #
lution plan ρi = ho1 , o2 , . . . , ok i, oj ∈ Oi for each individ- L
O = oi
ual problem Pi by applying a policy πi previously trained oi ∈O i=1
for Pi , we aim to find a general policy π that when applied
to every problem of the set returns a solution plan to all of where oi are actions from O that do not interfere with each
them. other when defining a single meta-operator and O1 = O.
Algorithm 1: Calculate all applicable RL actions include more than one atomic operator, briefly alleviating
Input: the aforementioned problem.
A set of applicable planning operators O In that sense, we want to test how this inclusion of differ-
Degree of meta-operators L ent specific amounts of reward in meta-operators affects to
Output: the learning process, so we will do an analysis using differ-
A set of applicable RL actions A ent amounts of reward and we will see how well they train
1: A = O and how parallel generated plans are compared with each
2: N = ∅ other.
3: # Generate all conflicts We also want to test whether the inclusion of meta-
4: for a ∈ O do operators actually improves the coverage for problems of
5: for p ∈ P re(a) do domains we analyzed, compared to the coverage of a se-
6: for b ∈ O do quential model, trained with the same domain but without
7: if p ∈ Del(b) then using meta-operators (A = O).
8: N = N ∪ {(a, b), (b, a)} # Interference
9: for p ∈ Add(a) do Experiments
10: for b ∈ O do In this section, we present a series of experiments that sup-
11: if p ∈ Del(b) then port the inclusion of meta-operators in Generalized Plan-
12: N = N ∪{(a, b), (b, a)} # Inconsistent effects ning using RL. In particular, we are interested in two things:
13: # Combine pairs, triplets, until L, of operators (1) analyzing the impact of an extra reward when a meta-
14: # that do not contain any pair of operators in N . operator is applied in the learning process, and (2) checking
15: for i ∈ {2, ..., L} do whether the inclusion of meta-operators improves the results
16: A = A ∪ MakeMetaOperators(O, L, N ) in terms of coverage (number of solved problems).
17: return A Specifically, we conduct two experiments. Experiment 1
is designed to measure the degree of parallelism of the solu-
tion plans using different rewarding in meta-operators. Ex-
Including meta-operators in RL periment 2 evaluates the performance of our model against
two different defined datasets.
Meta-operators are then added to the RL action space, en-
riching and enabling the application of parallel planning op- Domains
erators within this sequential RL action space. We can there-
fore define a new transforming function gL as the union of We will use two domains that are widely used in the IPC
the base set with meta-operators of degree L: and also known for their complexity, logistics and depots,
and a third domain which is an extension of the well-known
L
[ blocksworld domain.
gL (O) = Oi
Multi-blocksworld. This domain is an extension of the
i=1
blocksworld domain that features a set of blocks on an in-
and train our RL algorithms using this consideration, i.e., finite table arranged in towers, with the objective of getting
A = gL (O). a different block configuration by moving the blocks with
This integration of meta-operators is calculated online, as robot arms. Blocks can be put on top of another block or on
in Algorithm 1, at each time step t and current state st , for all the table, and they can be grabbed from the table or from
applicable RL actions At = {o ∈ O : o is applicable at st }. another block. We have defined two robot arms.
We define that a meta-operator is applicable at state st if
Logistics. This domain features packages located at cer-
every atomic operator is also applicable at st and if opera-
tain points which must be transported to other locations by
tors do not interfere with each other, which is followed by
land or air. Ground transportation uses trucks and can only
definition.
happen between two locations that are within the same city,
We used the Generalized Planning RL training scheme
while air transportation is between airports, which are spe-
proposed in (Rivlin, Hazan, and Karpas 2020) to observe the
cial locations of the cities. The destination of a package is
effects of the inclusion of a meta-operator in domains that
either a location within the same city or in a different city.
often struggle to generalize in the literature, such as logistics
In general, ground transportation is required to take a pack-
or depots (Ståhlberg, Bonet, and Geffner 2022a). That archi-
age to the city’s airport (if the package is not at the airport).
tecture also uses GNNs for state representation and gives a
The package is then carried by air between cities, and finally
reward of 1 to the agent if it reaches the goal, and 0 in any
using ground transportation the package is delivered to the
other case, as usual.
final destination if its destination is not the arrival airport.
This last decision is highly criticized by the planning com-
munity because it introduces the so-called sparse rewards Depots. This domain consists of trucks that are used for
problem, that is, the agent receives information from the transporting crates between distributors, and hoists to han-
environment at very specific moments, thus hindering the dle the crates in pallets. Hoists are only available at cer-
learning process. The inclusion of meta-operators opens up tain locations and are static. Crates can be stacked/unstacked
the possibility of defining a certain reward for actions that onto a fixed set of pallets. Hoists do not store crates in any
particular order. This domain slightly resembles the multi- followed here, all policies are trained for 900 iterations, each
blocksword domain as there is a stacking operation, though with 100 episodes and up to 20 gradient update steps, using
crates do not need to be piled up in a specific order, and to Proximal Policy Optimization RL training algorithm, with a
the logistics domain as to the existence of agents that trans- discount factor of 0.99.
port crates from one point to another.
Experiment 1: Rewarding of meta-operators
Data generation In this experiment, we aim to observe how the reward
RL algorithms need a large number of instances in order to granted to the application of a meta-operator in the RL train-
converge. That is why, for the training process, it was nec- ing influences the learning process and the quality of the
essary to use automatic generators of planning problems. plans. We are interested in measuring the effect of meta-
For logistics and depots domains, we used generators of the operators in terms of the plan length or the number of time
AI Planning Github (Seipp, Torralba, and Hoffmann 2022), steps of the plan. To that end, we define the parallelism rate
while for the multi-blocksworld domain we created a new of a solution plan of a problem as:
generator based on the generator for the blocksworld domain
(Seipp, Torralba, and Hoffmann 2022). # parallel operators
Table 1 illustrates the size distribution of the problems # total plan timesteps
used in this work; it shows the number of objects of each
type involved in all three domains. We generated a dataset where # parallel operators is the number of meta-
of random problems out of the distributions shown in Table operators that appear in the plan by applying the learned pol-
1, which we will refer to as Dataset 1 (D1). icy to the problem, and # total plan timesteps is the total
number of timesteps of the plan. This is a measure of how
Dataset 1 (D1) It consists of problems uniformly sampled frequently parallel operators appear in the decisions made
from the test distribution of Table 1 and generated with the by the planner agent.
aforementioned generators. We generated 460 problems for We trained a series of models giving different reward val-
the multi-blocksworld domain, 792 problems for the logis- ues to meta-operators. This experiment can be thought of as
tics domain and 640 for the depots domain, as a result of a way of tuning the meta-operators reward, which can there-
creating ten problems for each configuration in the test dis- fore be regarded as a hyperparameter. Since we primarily
tribution. aim to find the most appropriate reward for the use of meta-
Additionally, we created a second collection of samples operators in this experiment, we decided to focus only on
that we will refer to as Dataset 2 (D2) from a renowed plan- the training distribution.
ning competition. We trained five models from the train distribution
of Table 1 with reward values to meta-operators of
Dataset 2 (D2) It consists of problems that were used in 0.0, 0.1, 0.01, 0.001 and 0.0001, respectively. Subsequently,
the IPC (specifically, in the IPC-2000 and IPC-2002). We the five models were run on a fixed sample, generating 10
used the 35 first instances for the blocksworld domain in problems for each element from the train distribution, and
IPC-2000, with a slight modification to introduce the two results were analyzed in order to obtain the average paral-
robot arms; the 30 first instances of logistics from IPC-2000; lelism rate of all plans.
and the 22 instances of depots from IPC-2002. This set of During the experiment execution, rewards and the num-
instances was chosen in order to compare our results with ber of parallel actions at each time step are monitored so as
those obtained in the work (Ståhlberg, Bonet, and Geffner to balance out the reward coming from parallel actions and
2022b). coming from achieving a solution plan. In other words, we
want to avoid situations in which parallel actions are just
Setup added for the sake of reward, which may deviate plans to-
We opted for using a L = 2 degree meta-operator to come wards a large number of parallel actions sacrificing reaching
up with a feasible extension of the action space. As the inclu- the objective.
sion of meta-operators increases the action space, we need The results of this experiment are shown in Table 2:
to find a balance between size and performance. Using two- all models are able to fulfill the aforementioned objective
degree meta-operators is sufficient to fulfill the two objec- (100% of coverage in training) except for the model that
tives mentioned at the beginning of this section, namely an- gives a reward of 0.1 to meta-operators (no results are shown
alyzing the impact of rewarding meta-operators and evaluat- because the model did not converge). Intuition tells us that
ing the coverage of the models. We will also evaluate how there are certain values that reward too much parallelism,
much does the action space rise with our approach compared even above reaching the problem goal itself, resulting in po-
to a sequential model. tentially infinite plans that execute parallel actions in a loop
The RL training was conducted on a machine with a (until the maximum episode time is reached). This means
Nvidia GeForce RTX 3090 GPU, a 12th Gen Intel(R) that a reward value of 0.1 for meta-operators outputs poli-
Core(TM) i9-12900KF CPU and Ubuntu 22.04 LTS operat- cies that yield more than ten parallel actions per plan, which
ing system, and the same hyperparameter configuration than exceeds the value given to reaching the goal, thus making
(Rivlin, Hazan, and Karpas 2020). A similar training process the algorithm converge to a situation in which no goal is
as the one proposed in (Rivlin, Hazan, and Karpas 2020) was reached but lots of meta-operators of degree L > 2 appear in
 Domain Train size Total objects train Validation/Test size Total objects test
Multi-blocksworld 5-6 blocks 5-6 10-11 blocks 10-100
2-4 airplanes 3-4 airplanes
2-4 cities 6-7 cities
Logistics 2-4 trucks 9-10 3-4 trucks 24-29
2-4 locations per city 6-7 locations per city
1-3 packages 6-7 packages
1-2 depots 5-6 depots
2-3 distributors 5-6 distributors
2-3 trucks 5-6 trucks
Depots 13-22 30-36
3-5 pallets 5-6 pallets
2-4 hoists 5-6 hoists
3-5 crates 5-6 crates

Table 1: Sizes used for the problem generation, in terms of general and specific objects.

Reward Multi-blocks Logistics Depots a problem size smaller than the problem size on which we
0.0 0.550 0.243 0.530 will test the results.
0.1 - - - All in all, it has been found that the amount of reward
0.01 0.701 0.851 0.857 given to meta-operators is significant in terms of quality and
0.001 0.559 0.582 0.381 convergence of plans.
0.0001 0.557 0.768 0.294
Experiment 2: Performance in Generalized
Table 2: Average parallelism rate for all models trained with Planning
specified reward for the application of meta-operators. In this experiment, we compare the original sequential
model with versions of the parallel model obtained with dif-
ferent reward values. We note that the aim is to test the per-
the plan. Ultimately, RL is about optimizing a reward func- formance of the models when dealing with new inputs of a
tion, and if adding meta-operators produces more reward, larger size. The trained policy for each domain is then an-
this will be the path taken by the model. alyzed as in the Generalized Planning literature by testing
According to Table 2, we observe that the model that gives the problems in datasets D1 and D2. Particularly, for each
the best results in terms of parallelism rate for all domains model, we measure the coverage and the average length of
within the plans generated is the 0.01 reward model. This the generated plans for the problems in D1 and D2.
indicates that, in order to obtain potentially better results, a Table 3 shows the results obtained with the sequential
balance must be established between the amount of reward (OR) model (Rivlin, Hazan, and Karpas 2020), the paral-
given to parallelism and the amount of reward given to the lel model trained with no reward (R=0.0), with a reward of
goal. 0.01 on meta-operators (R=0.01) and with a reward of 0.001
For example, a somewhat more conservative proposal, on meta-operators (R=0.001) for the International Planning
which we know for sure would not exceed the goal reward, is Competition (D2) and randomly generated (D1) datasets.
GOAL REW ARD With these experiments we aim to illustrate how the results
to establish a meta-operator reward of MAX IT ERAT ION S ,
where GOAL REW ARD is the reward given to reach- vary from one model to another depending on the reward, as
ing the goal (generally 1) and M AX IT ERAT ION S is stated in the previous section.
the maximum number of applications of the policy before The table is divided in two halves, one for each set of
stating that the goal cannot be achieved. Generally, this ap- problems. In the top part of each half we show results for
proach is excessively limiting and does not encourage par- coverage of the analyzed models with respect to the prob-
allelism. This is evident from Table 2, which shows that lems of each set, while in the bottom part of each half the
greater rewards lead to improved parallelism. average length of plans for each problem of the set is an-
In fact, the amount of reward provided in meta-operators alyzed. In the top part of each half we show within paren-
is also dependent on the average length of the plans we theses the total number of problems that compound that set,
want to test. That means, perhaps if the problems we want and then in each column the number of those for which the
to test have a larger average plan length than the ones we model under analysis has managed to reach the target. The
trained, it would be wiser to test with a model that has been bottom part of each half corresponds to the number of time
trained with a slightly lower reward in order to not “over- steps with which the models managed to solve each set of
flow” the reward and fall into the undesired scenario of poli- problems. We present the average number of actions taking
cies that produce parallel actions with no goal termination. into account only the solved problems.
This would be a problem that would probably occur in Gen- Results of Table 3 show that coverage from the models
eralized Planning, for example, where we train models with that use meta-operators improves with respect to the cover-
 Domain - D1 OR R = 0.0 R = 0.001 R = 0.01
Multi-blocks (460) 268 439 408 406
Logistics (792) 131 701 717 317
Depots (640) 287 572 640 552
Multi-blocks 79.99 76.43 65.92 92.84
Logistics 200.50 112.61 109.05 110.25
Depots 338.49 106.68 124.46 121.80
Domain - D2 OR R = 0.0 R = 0.001 R = 0.01
Multi-blocks (35) 2 35 34 34
Logistics (30) 11 26 28 27
Depots (22) 20 17 20 20
Multi-blocks 172.00 43.17 44.59 54.12
Logistics 136.64 115.92 120.64 121.00
Depots 127.45 83.59 110.20 114.25

Table 3: Results for datasets D1 and D2. The top part of the tables shows coverage out of the total number of instances shown
between parenthesis. The bottom part of the tables indicates the average plan length. OR is the original sequential model in
(Rivlin, Hazan, and Karpas 2020); R=0.0 is the model with no meta-operator reward; R=0.1 is the model with reward of 0.01,
and R=0.001 is the model with reward of 0.001.

Multi-blocks Logistics Depots ported significantly better coverage results than the model
Sequential 100 108 228 with R=0.01. In the Generalized Planning task there is a
Parallel 1140 3960 8519 variance in the size of problems tested, resulting also in
a variance in the length of its correspondent plans. As the
model R=0.01 gives a high reward to parallelism, if the size
Table 4: Action space or number of RL actions (planning
of the plans is too large, parallelism is being rewarded too
operators and, when makes sense, meta-operators) visited
much. For example, 11 meta-operators would already mask
during training.
the objective’s reward, which is 1, i.e., 11 · 0.01 = 1.1 > 1.

age of the sequential model. Moreover, it is observed that for Discussion

the logistics domain, which has been found in the literature Although there is not yet a standard neural network ar-
to be hard to address for RL approaches (Ståhlberg, Bonet, chitecture that suits perfectly the planning frame, the ap-
and Geffner 2022b), a very good result is obtained with re- proaches that leverage deep learning techniques that auto-
spect to that of the sequential model. matically extract structure from high-level data are mostly
Results are correlated to the enrichment of the action based on graph convolution to learn state embeddings like
space. Table 4 shows the size of the action space visited dur- Graph Neural Networks (Rivlin, Hazan, and Karpas 2020;
ing training of the sequential model and the parallel ones. Ståhlberg, Bonet, and Geffner 2022b) or first-order logic
There is only one result in Parallel because the action space neural models like Neural Logic Machines (Gehring et al.
is the same for all three models, only the amount of reward 2022).
given to the meta-operator application varies. We observe Overall, the aforementioned approaches show to be com-
that in the multi-blocks domain where there are only two petitive with baseline sequential models that use planning
agents (two robot-arms), the increase in the number of ac- heuristics or state-of-the-art implementations of classical
tions is much less significant than in logistics or depots do- planners in terms of coverage (number of solved problems).
main, in which many more agents (trucks, planes, etc.) co- More interestingly, some works report the inability of NN-
exist at the same time. based heuristics to outperform classical planning engines in
Plans also tend to be shorter in terms of average actions transport-like domains wherein a tight coupling between the
when applying meta-operators. In Table 3, results for R=0.0 different objects of a planning task exists (Rivlin, Hazan, and
tend to give short plan lengths, which makes sense because Karpas 2020; Ståhlberg, Bonet, and Geffner 2022b; Gehring
as a sparse reward problem plans tend to reach the goal faster et al. 2022).
thanks to the discount factor and reward only coming from Frameworks that combine AI planning and RL establish
reaching the goal. This sometimes even result in better cov- a one-to-one correspondence between operators in planning
erage, such as in multi-blocks, while in R=0.001 and R=0.01 and actions in RL. It is observed, as stated before, that low
the reward also comes from the meta-operators and not only performance usually arises in domains with a high number
from reaching the goal. of agents that need to collaborate with each other to reach
The use of a model with lower reward R=0.001 has re- the goal, also called tightly-coupled domains (Torreño, On-
aindia, and Sapena 2014), for example logistics or depots. On the other hand, we also wanted to hypothesize about
Specifically, speaking about logistics, in (Ståhlberg, why meta-operators also favor exploration in the state space
Bonet, and Geffner 2022b) authors prove that it is not possi- of planning. If we think about state graphs as in (Bonet and
ble to achieve satisfactory results using a GNN architecture. Geffner 2022), which are graphs that outline the whole struc-
The authors identify transitive relations without which the ture of actions and states that the agent can take at each mo-
locality of the objects is lost. For example, when a package ment, we can consider adding meta-operators to the RL ac-
is inside a truck and the truck is in a city, the information tion space as introducing extra edges to these state graphs.
about the city where the package is found fails to propagate In the end, training with RL is done on a trial-and-error
in a transitive manner, i.e., it fails to infer that the package basis, choosing actions randomly at first and seeing if it
is indeed in the city. Results within this perspective are im- brings some benefit. By including meta-operators we allow
proved by hard-coding extra predicates (derived atoms) and our planning agent to do in one sampling step what previ-
including them to guarantee the transitive relations. In the ously had to be done in more (as long as the actions can
mentioned example, a new atom is added to denote that a coexist in parallel, obviously). This helps the convergence
package is in a city. and thus the generalization of our methods.
Interestingly, we found that in tightly-coupled domains To illustrate this last affirmation, we want to give another
such as logistics, depots or multi-blocksworld test results example. Let’s assume an initial training iteration with a pol-
from our perspective are very satisfactory when including icy that is initialized with a uniform probability p to each RL
meta-operators. In fact, if we compare the same IPC in- action. If the agent decides to apply a meta-operator of de-
stances of the logistics domain from (Ståhlberg, Bonet, and gree L, we arrive to a state s with a probability of p, but
Geffner 2022b) with our approach we observe that, while in a sequential environment we would have to arrive to s
they have a coverage of 17 (28) without using the hard-coded (following the same path) with L different transitions, each
atoms, we obtain 26 (28). It should be emphasized that our one corresponding to the atomic operators that conform the
process is automatic, i.e., it is not necessary to manually en- meta-operator, with a pL probability. This summarizes why
rich any problem or domain to reach the goal. we think it is interesting to apply meta-operators.
It is interesting to think about why adding meta-operators
actually improves the training process when it comes to us- Conclusion and Future work
ing RL for planning. It is not only the fact that it allows In conclusion, we defined the notion of meta-operator as a
several atomic operators to coexist at the same time, decen- way to enrich the RL structure that has hegemonically been
tralizing as already mentioned in other sections the purely used to attack the planning problem. Furthermore, we saw
sequential nature of RL, but also the fact that when adding that it improves results from other sequentially trained mod-
the extra operators the algorithm converges faster and yields els, both with randomly generated problems and with prob-
better results. lems from the International Planning Competition. Finally,
We hypothesize that the inclusion of meta-operators we wanted to highlight the discussion that may arise from
opens a way of (virtual) communication between agents. this paper, with special emphasis on the fact that RL struc-
As stated in (Rivlin, Hazan, and Karpas 2020), problems in tures can be enriched to better match the true nature of plan-
tightly-coupled domains arise when certain resources need ning.
to be used by several agents at the same time. Since a pol- Finally, we are aware that the inclusion of meta-operators
icy is nothing more than the “brain” of the planning agent, makes the action space larger, so a balance between paral-
when several entities can be executed in parallel (e.g. mov- lelism and size must also be found so that training times are
ing different vehicles) or change their state (location, mode, not excessively long. For future work we would like to ex-
etc.) at the same time, the sequential nature of the RL may plore different ways of attacking RL problems with much
interfere with this notion of collaboration, thus obstructing larger, even continuous, action spaces in order to compu-
the convergence of RL methods for this type of problems. tationally be able to use meta-operators of any degree. We
By introducing parallel actions we allow several entities would also like to explore other different ways of represent-
to change their state at the same time, so that they can “pre- ing planning states and how this may influence what was
pare” for what is next to come. A clear example is in the analyzed and to try to explain how does tuning the meta-
logistics domain: if we have a package being transported operators reward affect in several different domains.
in a truck, which must then be picked up by another entity
that transports it elsewhere, in a sequential nature the truck Acknowledgments
would leave the package at one location, and then the other This work is supported by the Spanish AEI PID2021-
entity would have to pick it up and take it away. However,
127647NB-C22 project and Ángel Aso-Mollar is partially
how can one distinguish, given that the only known informa-
supported by the FPU21/04273.
tion is that the package is at one location, that the package is
to be picked up by the next entity and not by the same entity?
By introducing parallelism, the second entity could go to the References
exchange point before the package actually reaches that ex- Blum, A.; and Furst, M. L. 1997. Fast Planning Through
change point. We believe that our approach provides some Planning Graph Analysis. Artif. Intell., 90(1-2): 281–300.
sense of “entity” to the independent objects of a planning Bonet, B.; and Geffner, H. 2022. Language-Based Causal
problem and improves agent collaboration. Representation Learning.
Celorrio, S. J.; Aguas, J. S.; and Jonsson, A. 2019. A review Thiébaux, S.; Varakantham, P.; and Yeoh, W., eds., Proceed-
of generalized planning. Knowl. Eng. Rev., 34: e5. ings of the Thirty-Second International Conference on Au-
Dong, H.; Mao, J.; Lin, T.; Wang, C.; Li, L.; and Zhou, D. tomated Planning and Scheduling, ICAPS 2022, Singapore
2019. Neural Logic Machines. CoRR, abs/1904.11694. (virtual), June 13-24, 2022, 629–637. AAAI Press.
Ståhlberg, S.; Bonet, B.; and Geffner, H. 2022b. Learn-
Eysenbach, B.; Salakhutdinov, R.; and Levine, S. 2019. ing Generalized Policies without Supervision Using GNNs.
Search on the Replay Buffer: Bridging Planning and Re- In Kern-Isberner, G.; Lakemeyer, G.; and Meyer, T., eds.,
inforcement Learning. In Advances in Neural Information Proceedings of the 19th International Conference on Princi-
Processing Systems 32: Annual Conference on Neural Infor- ples of Knowledge Representation and Reasoning, KR 2022,
mation Processing Systems 2019, NeurIPS 2019, December Haifa, Israel. July 31 - August 5, 2022.
8-14, 2019, Vancouver, BC, Canada, 15220–15231.
Sutton, R. S.; and Barto, A. G. 1998. Reinforcement learn-
Francès, G.; Corrêa, A. B.; Geissmann, C.; and Pommeren- ing - an introduction. Adaptive computation and machine
ing, F. 2019. Generalized Potential Heuristics for Classi- learning. MIT Press. ISBN 978-0-262-19398-6.
cal Planning. In Kraus, S., ed., Proceedings of the Twenty-
Eighth International Joint Conference on Artificial Intel- Torreño, A.; Onaindia, E.; and Sapena, Ó. 2014. FMAP:
ligence, IJCAI 2019, Macao, China, August 10-16, 2019, Distributed cooperative multi-agent planning. Applied Intel-
5554–5561. ijcai.org. ligence, 41(2): 606–626.
Zhou, J.; Cui, G.; Hu, S.; Zhang, Z.; Yang, C.; Liu, Z.; Wang,
Gehring, C.; Asai, M.; Chitnis, R.; Silver, T.; Kaelbling, L.; Li, C.; and Sun, M. 2020. Graph neural networks: A
L. P.; Sohrabi, S.; and Katz, M. 2022. Reinforcement Learn- review of methods and applications. AI Open, 1: 57–81.
ing for Classical Planning: Viewing Heuristics as Dense Re-
ward Generators. In Kumar, A.; Thiébaux, S.; Varakan-
tham, P.; and Yeoh, W., eds., Proceedings of the Thirty-
Second International Conference on Automated Planning
and Scheduling, ICAPS 2022, Singapore (virtual), June 13-
24, 2022, 588–596. AAAI Press.
Icarte, R. T.; Klassen, T. Q.; Valenzano, R. A.; and McIlraith,
S. A. 2022. Reward Machines: Exploiting Reward Function
Structure in Reinforcement Learning. J. Artif. Intell. Res.,
73: 173–208.
Lee, J.; Katz, M.; Agravante, D. J.; Liu, M.; Tasse, G. N.;
Klinger, T.; and Sohrabi, S. 2022. Hierarchical Reinforce-
ment Learning with AI Planning Models.
Moerland, T. M.; Broekens, J.; Plaat, A.; and Jonker, C. M.
2023. Model-based Reinforcement Learning: A Survey.
Found. Trends Mach. Learn., 16(1): 1–118.
Ng, J. H. A.; and Petrick, R. P. A. 2019. Incremental
Learning of Planning Actions in Model-Based Reinforce-
ment Learning. In Kraus, S., ed., Proceedings of the Twenty-
Eighth International Joint Conference on Artificial Intel-
ligence, IJCAI 2019, Macao, China, August 10-16, 2019,
3195–3201. ijcai.org.
Puterman, M. L. 1990. Chapter 8 Markov decision pro-
cesses. In Stochastic Models, volume 2 of Handbooks in
Operations Research and Management Science, 331–434.
Elsevier.
Rivlin, O.; Hazan, T.; and Karpas, E. 2020. General-
ized Planning With Deep Reinforcement Learning. CoRR,
abs/2005.02305.
Seipp, J.; Torralba, Á.; and Hoffmann, J. 2022. PDDL Gen-
erators. https://doi.org/10.5281/zenodo.6382173.
Silver, T.; and Chitnis, R. 2020. PDDLGym: Gym Environ-
ments from PDDL Problems. CoRR, abs/2002.06432.
Ståhlberg, S.; Bonet, B.; and Geffner, H. 2022a. Learning
General Optimal Policies with Graph Neural Networks: Ex-
pressive Power, Transparency, and Limits. In Kumar, A.;