Automata-Theoretic Synthesis of Plans and Reactive Strategies

by

W
IE
Alberto Camacho
EV
PR

A thesis submitted in conformity with the requirements

for the degree of Doctor of Philosophy
Department of Computer Science
University of Toronto

© Copyright 2022 by Alberto Camacho

 Automata-Theoretic Synthesis of Plans and Reactive Strategies

Alberto Camacho
Doctor of Philosophy
Department of Computer Science
University of Toronto
2022

Abstract

Sequential decision making is a central problem in Artificial Intelligence (AI) with applications rang-

ing from the automated synthesis of plans for conversational agents to the synthesis of controllers for

power plants and autonomous vehicles. We are concerned with synthesizing strategies for sequential

decision making in discrete dynamical systems, directly from a logical specification. This problem

has been investigated within the fields of automated planning and reactive synthesis, utilizing differ-

W
ent formulations of the problem dynamics and objectives. By relating the two models, we exploit the

theory and practice from one to make significant advances in the theory and practice of the other.
IE
We start by revealing a direct correspondence between reactive synthesis and automated planning

with temporally extended goals in fully observable and non-deterministic (FOND) environments.
EV
Such a correspondence exists when goals and specifications are expressed in Linear Temporal Logic

(LTL), and it extends beyond LTL to all the regular and ω-regular specification languages, that

can be transformed into finite state automata. Finite state automata are a central theme in the

dissertation, and we use them as the computational normal form to derive theoretical results and
PR

algorithms. Another central theme is the use of FOND planners as a tool. FOND planners have

been highly optimized to exploit the problem structure, and also environment fairness assumptions.

However, such tools can only handle final-state goals. We develop theory and algorithms for FOND

planning with goals that are temporally extended, using several types of finite state automata to rep-

resent such goals. Our algorithms compile automata goals away, and make it possible to use existing

FOND planners for the purpose of planning with temporally extended goals. In the second part of

the thesis, we revisit specifications for reactive synthesis. We start by studying the crucial role of

environment assumptions in realizability, and argue that they should be interpreted as properties of

infinite-length traces, in general, even when programs terminate in finite time. We then recast spec-

ifications as automata games, and design principled methods to solve those games via compilation

to FOND planning. Our methods can handle specifications for terminating and non-terminating

programs, can incorporate environment assumptions, and can solve realizability and synthesis. Our

empirical evaluation suggests that FOND planners can be an effective tool for sequential decision

ii
making, and that exploiting program termination and environment fairness can improve the scala-

bility of algorithms. By studying planning and synthesis from an automata-theoretic perspective,

our results can be extended to a variety of temporal logics and other regular and ω-regular speci-

fication languages that can be transformed into automata. The work presented in this dissertation

establishes a new area of study for researchers in AI automated planning that holds the promise

to be transformative with respect to the field of reactive synthesis and to program synthesis more

generally.

W
IE
EV
PR

iii
 Acknowledgements
First and foremost, I would like to thank my supervisor, Sheila McIlraith. Sheila has given me ample
time and space to explore research topics. Eventually, she introduced me to the topics that ended
up being central to the development of my thesis. This work would have never been possible without
her guidance and support. Besides research, Sheila has been an example for how to be considerate
and appreciative of others. I am grateful to her for creating a friendly and intellectually stimulating
environment to work on challenging problems, and for assembling a great team of collaborators.
Besides Sheila, I am grateful to my internal committee—Fahiem Bacchus, Chris Beck, and Scott
Sanner. All of them have been a source of inspiration and their impact on me is hard to summarize.
I saw in Fahiem an example for how to be scientifically rigorous and humble. From Chris, I learned
to be more critical and give more relevance to experimental results. And Scott inspired me to think
out of the box and gain expertise in other fields of science, beyond the topics of my thesis. I also
thank my external committee member, Malte Helmert, for their extremely detailed and valuable
feedback. Malte has been an example of academic rigor and excellence to me.
I am grateful to all the people that have contributed to this dissertation in some way, including

W
the feedback from my committee, informal conversations, discussions in conferences, and all the
scientific work that set the basis for my thesis. Through a number of iterations, Sheila helped shape
the structure of my thesis into a form that tells a coherent story. I would also like to thank Toryn
IE
Klassen and Steven Shapiro who reviewed early drafts of my thesis.
I am fortunate to have crossed roads with many other people which have had a positive impact on
the way I think about science and about doing research. Hector Geffner introduced me to research
EV
in artificial intelligence when I was a master student in Barcelona. Way before that, I had very
dedicated math professors. I must mention Marta Berini, Modest Clavé, Jose M. Ferrán, and Josep
Grané. I also have to mention my grandfather Joaquı́n. He was very influential to me, and he would
have been very proud of my achievement with this dissertation.
I have shared countless coffees, lunches, and some birthday cakes with my office mates, including
PR

León Illanes, Toryn Klassen, Maayan Shvo, Rodrigo Toro Icarte, Pashootan Vaezipoor, and Rick
Valenzano. I have had the fortune to collaborate with Oscar Chen and Eleni Triantafillou, who were
undergraduate student researchers at the time. I have also shared many stories with people from
other research groups and from the University of Toronto, in general, including Ozan Erdem, Tyler
Lu, Karime Pereira, Nicolas Tanguy, Maria Toledo, and Zahra Hodjat.
I have had the privilege of having very talented collaborators. Christian Muise was instrumental
during the first years, and I have learned very good practices for doing research from him. Jorge
Baier brought a lot of expertise, and his scrutiny was crucial for advancing our research in the right
direction. I also enjoyed a lot working with Meghyn Bienvenu during her visits to the University of
Toronto. Her talent encouraged me to work harder and do better quality research.
I would also like to thank my friends from the planning community Tathagata Chakraborti,
Daniel Gnad, Guillem Francès, Jeremy Frank, Rick Freedman, Malte Helmert, Joerg Hoffman, An-
ders Jonsson, Erez Karpas, Michael Katz, Sarah Keren, Joseph Kim, Andrey Kolobov, Nir Lipovet-
zky, Hector Palacios, Sebastian Sardina, Javier Segovia, Shirin Sohrabi, Álvaro Torralba, and many,
many others with whom I have shared experiences.
Finally, I would also like to thank my family for their unconditional support, and for always
being there.

iv
Contents

1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Approach and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Outline and Contributions by Chapter . . . . . . . . . . . . . . . . . . . . . . . . . . 5

W
1.5 Organization of the Dissertation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2 Preliminaries 9
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
IE 9
2.2 Temporal Logics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.1 Linear Temporal Logics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.2.2 Temporal Operators in the Planning Domain Definition Language . . . . . . 12
EV
2.2.3 Other Temporal Logics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3.1 Finite-Word Automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.2 Infinite-Word Automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
PR

2.3.3 Operations on Automata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.3.4 Automata Transformations of Temporal Logics . . . . . . . . . . . . . . . . . 17
2.4 Planning with Temporally Extended Goals . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.1 Planning Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.2 Planning Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.3 Fairness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4.4 Solutions to FOND Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.5 Plans, Policies, Strategies, and Controllers . . . . . . . . . . . . . . . . . . . . 22
2.5 Synthesis of Reactive Systems from Specification . . . . . . . . . . . . . . . . . . . . 23
2.5.1 LTL Realizability and Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.5.2 LTLf Realizability and Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . 24

3 A Unified View of AI Planning and Reactive Synthesis 25

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.2 Differences Between Planning and Synthesis . . . . . . . . . . . . . . . . . . . . . . . 28
3.3 FOND Planning with LTL and LTLf Goals . . . . . . . . . . . . . . . . . . . . . . . 28
3.3.1 Planning Models and Solutions . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3.2 Measures of Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

v
 3.4 Game Structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
3.4.1 Game Structures and LTL Synthesis . . . . . . . . . . . . . . . . . . . . . . . 31
3.4.2 Solving Structured Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.5 Plan Synthesis and Structured Games . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.5.1 Structured Games as LTL FOND Planning . . . . . . . . . . . . . . . . . . . 34
3.5.2 LTL FOND Planning as Structured Games . . . . . . . . . . . . . . . . . . . 36
3.5.3 LTLf FOND Planning as Structured Games . . . . . . . . . . . . . . . . . . . 37
3.6 More Efficient LTL and LTLf Plan Synthesis . . . . . . . . . . . . . . . . . . . . . . 39
3.6.1 GR(1) Synthesis and Game Structures . . . . . . . . . . . . . . . . . . . . . . 40
3.6.2 Planning with GR(1) Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.6.3 Request-Response Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.6.4 Planning with PDDL3 Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.7 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.7.1 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.7.2 FOND Planning Problems Encoded as LTL Specifications . . . . . . . . . . . 44

W
3.7.3 LTL Synthesis versus FOND Planning . . . . . . . . . . . . . . . . . . . . . . 45
3.7.4 FOND Planners Exploit Structure . . . . . . . . . . . . . . . . . . . . . . . . 46
3.8 Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

4 Automata-Theoretic Plan Synthesis

4.1
IE
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
49
49
4.2 Background on Planning with Temporally Extended Goals . . . . . . . . . . . . . . . 52
EV
4.3 Planning with Automata Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
4.3.1 Complexity of Strong Planning . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.3.2 Complexity of Stochastic-Fair Planning . . . . . . . . . . . . . . . . . . . . . 56
4.4 Synthesis of Terminating Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.4.1 Compilations to Strong FOND Planning . . . . . . . . . . . . . . . . . . . . . 57
PR

4.4.2 Automata Decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

4.4.3 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
4.4.4 Computational Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
4.4.5 Special Case: FOND Planning with DFW Goals . . . . . . . . . . . . . . . . 60
4.4.6 Application: FOND Planning with LTLf, LDLf, and PLTL Goals . . . . . . . 61
4.5 Bounded Plan Synthesis and Realizability . . . . . . . . . . . . . . . . . . . . . . . . 62
4.5.1 Bounded Plan Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.5.2 Bounded Plan Realizability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.6 Synthesis of Non-Terminating Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
4.6.1 Compilations to Strong-Cyclic FOND Planning . . . . . . . . . . . . . . . . . 65
4.6.2 Automata Decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
4.6.3 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.6.4 Computational complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
4.6.5 Application: FOND Planning with LTL Goals . . . . . . . . . . . . . . . . . 68
4.7 Proofs of Unsolvability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
4.7.1 Compilations to Strong-Cyclic FOND Planning . . . . . . . . . . . . . . . . . 69
4.7.2 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

vi
 4.7.3 Computational complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
4.7.4 Application: FOND Planning with LTL Goals . . . . . . . . . . . . . . . . . 70
4.8 Plan Synthesis in Fair Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.8.1 Terminating Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
4.8.2 Non-Terminating Plans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
4.8.3 Application: Stochastic-Fair FOND Planning with LTL and LTLf Goals . . . 75
4.9 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
4.9.1 Computing Terminating Plans . . . . . . . . . . . . . . . . . . . . . . . . . . 77
4.9.2 Computing Non-Terminating Plans . . . . . . . . . . . . . . . . . . . . . . . . 80
4.10 Discussion and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83

5 Reactive Synthesis: Theory 86

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
5.2 Automata Games . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.2.1 Solutions to Terminating and Non-Terminating Games . . . . . . . . . . . . . 90

W
5.2.2 Certificates of Unrealizability . . . . . . . . . . . . . . . . . . . . . . . . . . . 91
5.2.3 Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.3 Realizability and Synthesis for Non-Terminating Games . . . . . . . . . . . . . . . . 92
5.3.1 Review on Bounded Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . .
IE 92
5.3.2 Bounded Realizability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.3.3 A Unified View of Realizability and Synthesis via Duality . . . . . . . . . . . 94
5.3.4 Application: LTL Realizability and Synthesis . . . . . . . . . . . . . . . . . . 96
EV
5.3.5 Particular Case: Safe and Co-Safe Specifications . . . . . . . . . . . . . . . . 96
5.4 Realizability and Synthesis for Terminating Games . . . . . . . . . . . . . . . . . . . 98
5.4.1 Review on LTLf Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.4.2 Certificates of Unrealizability . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
PR

5.5 Reactive Synthesis with Environment Assumptions . . . . . . . . . . . . . . . . . . . 101

5.5.1 Illustrative Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
5.5.2 Modeling Environment Assumptions in LTLf Synthesis . . . . . . . . . . . . . 102
5.5.3 Correspondence with LTLf Synthesis . . . . . . . . . . . . . . . . . . . . . . . 103
5.5.4 Correspondence with LTL Synthesis . . . . . . . . . . . . . . . . . . . . . . . 105
5.5.5 Efficient Algorithms for Constrained LTLf Synthesis . . . . . . . . . . . . . . 106
5.6 Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

6 Reactive Synthesis: Practice 113

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
6.2 Synthesis of Reactive Strategies for Terminating Games . . . . . . . . . . . . . . . . 115
6.2.1 Compilations to Strong FOND Planning . . . . . . . . . . . . . . . . . . . . . 116
6.2.2 Automata Decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6.2.3 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
6.2.4 Computational Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
6.2.5 Applications: Synthesis of LTLf and LDLf Specifications . . . . . . . . . . . . 119
6.3 Certificates of Unrealizability for Terminating Games . . . . . . . . . . . . . . . . . . 119
6.3.1 Compilations to Strong-Cyclic FOND Planning . . . . . . . . . . . . . . . . . 120

vii
 6.3.2 Automata Decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
6.3.3 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
6.3.4 Computational Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
6.3.5 Application: Certificates for LTLf, LDLf, and PLTL Specifications . . . . . . 122
6.4 Synthesis of Reactive Strategies for Non-Terminating Games . . . . . . . . . . . . . 123
6.4.1 Compilation of NkBW Games to Strong FOND Planning . . . . . . . . . . . 124
6.4.2 Compilations of UkCW Games to Strong-Cyclic FOND Planning . . . . . . . 125
6.4.3 Automata Decompositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.4.4 Correctness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
6.4.5 Computational Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6.5 Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.5.1 LTLf Realizability and Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.5.2 LTL Realizability and Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . 133
6.6 Summary and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

7 Concluding Remarks 137

W
7.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
7.2 Contributions by Chapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
7.3 Impact of Our Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
IE
7.4 Contributions to Related Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
7.5 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
EV
Bibliography 145
PR

viii
Chapter 1

Introduction

1.1 Motivation

W
Automated program synthesis is a central problem in computer science that is concerned with
producing a program that satisfies a prescribed high-level specification. The general problem of
synthesis from formal specification can be traced back to the work of Church (1957) on the application
IE
of formal logic to circuit synthesis. Techniques for program synthesis from logical specification
produce provably correct solutions, which is a compelling feature, with the added benefit that
program maintenance is reduced to the task of maintaining the specification of the problem. We
EV
are concerned with program synthesis for sequential decision making. In broad terms, a sequential
decision making problem involves the notion of an agent with a long-term objective and the ability to
interact over time with the rest of the world (commonly referred to as the environment). Sequential
decision making is a fundamental component in the development of intelligent autonomous systems.
We are concerned with two seemingly different models for sequential decision making: automated
PR

planning and reactive synthesis (planning and synthesis, for short). Planning has been applied to
robotics (e.g., (Cashmore et al., 2015)), business process trace alignment (e.g., (Giacomo et al.,
2016)), penetration testing (e.g., (Shmaryahu et al., 2018)), the development of dialogue agents
(e.g., (Chakraborti et al., 2019)), and rail network planning (e.g., (Li et al., 2021)), among others.
In comparison to planning, the popular benchmark problems for reactive synthesis usually specify
problems at a lower level of abstraction. Reactive synthesis has been applied to to the construction
of a reactive module for the control of electromechanical systems, memory buses, counters, logical
registers, and logical circuits (e.g., (Maoz and Ringert, 2015; Jacobs et al., 2019)).
Planning and synthesis share a common past, reflected in the early works of researchers such
as Green (1969), Nilsson (1969), Waldinger and Lee (1969) and Fikes and Nilsson (1971); however,
the subfields diverged as each became more specialized. Spurred by the success of STRIPS (Fikes
and Nilsson, 1971), automated planning emerged as a branch of Artificial Intelligence (AI) focused
on compact knowledge representation and effective algorithmic reasoning techniques for automated
plan generation. In contrast, reactive synthesis was often studied from a formal methods perspective.
Both models presume that a specification of the problem is given, but they differ with respect to
how the dynamics are conceived, the types of goals and objectives they address, and the form of the
solutions they yield.

1
CHAPTER 1. INTRODUCTION 2

Automated planning imposes strict assumptions to the dynamics and observability of the problem
setting (e.g., (Ghallab et al., 2004; Geffner and Bonet, 2013)). In a classical planning problem,
goals are final-state conditions, the world is fully observable, and the environment dynamics are
deterministic and Markovian—which means that the result of an action does not depend on the past
history of states. Solutions are action plans, or strategies that tell the agent which actions to execute
in order to satisfy the goal. For this reason, we also refer to automated planning as plan synthesis.
Over the years, non-classical planning models have emerged to accommodate the dynamics and
goals of many real-world problems that fall outside of the classical planning setting. In this regard,
so-called fully observable and non-deterministic (FOND) planning (Cimatti et al., 2003) breaks the
assumption that the dynamics are deterministic. Non-classical goals may refer to temporally extended
properties of the whole execution traces (e.g., safety and reachability), rather than solely on final
state conditions. Scientific advances in FOND planning had been mostly limited to final-state goals,
and FOND planning with temporally extended goals had remained largely unexplored.
Reactive synthesis can be traced back to the influential paper by Pnueli and Rosner (1989) on
synthesizing a reactive module from a logical specification. In contrast to planning, which provides

W
an actor-centric view of the dynamics of the problem setting, in synthesis the dynamics are typ-
ically conceived in terms of a game between the agent and the environment—both actors in the
problem setting. The game is played in turns. Each player controls their own set of variables,
IE
which collectively constitute a world state. Specifications comprise one formula that describes the
intended evolution of the world state. Such a formula embeds environment assumptions, dynamics,
and behavioral constraints that may be temporally extended and non-Markovian. The specification
is typically expressed in Linear Temporal Logic (LTL) (Pnueli, 1977), and other temporal logics that
EV
can be transformed into finite state automata—a mathematical abstraction of the specification of
the problem. Solutions are agent strategies that guarantee satisfaction of the specification.
Our contributions start by establishing a clear correspondence between FOND planning with
temporally extended goals and reactive synthesis. Finite state automata play a central role in such
PR

correspondence, that we exploit to augment the class of problems that can be modeled and solved.
The other central theme is the use of FOND planners, which can only handle final-state goals, as
a tool for solving FOND planning problems with temporally extended goals as well as for solving
specifications for reactive synthesis. We use several types of finite state automata to represent goals
and specifications. By doing so, our results extend to a myriad of popular, human-friendly, spec-
ification languages that yield equivalent automata representations. These include well-established
languages from software engineering, formal methods, and programming languages, such as regular
expressions and LTL. The unification of these tasks and the contributions to each individual subfield
take us an important step closer to the elusive aspiration of automated program synthesis.

Thesis Statement
FOND planning with temporally extended goals and reactive synthesis can be bridged
and unified through the lens of finite state automata. Automata can capture specifica-
tions of planning and synthesis problems across a diversity of languages, including Linear
Temporal Logic, and are a compelling computational normal form for the design of algo-
rithms. Planning technology can be a powerful tool capable of exploiting the structure
of automata to determine the existence of solutions and computing them.
CHAPTER 1. INTRODUCTION 3

1.2 Approach and Scope

Planning models usually presume that programs (i.e., plans) have to terminate in finite time. Accord-
ingly, variations of LTL interpreted over finite-length traces are used to specify temporally extended
goals, such as LTLf (De Giacomo and Vardi, 2013) and PDDL3 operators (Gerevini and Long, 2005;
Gerevini et al., 2009). In contrast, traditional specifications for reactive synthesis refer to non-
terminating programs, and they are commonly expressed in LTL. In our work, we will study models
and algorithms for planning and synthesis, for terminating and non-terminating programs.
Exploiting Finite State Automata. Finite state automata are a central theme in this dis-
sertation. An automaton is a mathematical abstraction that can be used to describe a broad class
of temporally extended properties of finite- and infinite-length traces (see Section 2.3 on page 12).
Automata are strictly more expressive than LTL and LTLf , and they also subsume other specification
languages commonly used in planning and synthesis. The field of reactive synthesis has studied the
theoretical properties of finite state automata to derive theoretical results and also algorithms. We
will adopt several types of automata as the computational normal form to represent temporally

W
extended goals (in planning), and specifications (for reactive synthesis).
Computationally Efficient Goals and Specifications. Because computing, manipulating,
and reasoning on finite state automata can become a computational bottleneck, a branch of research
IE
on reactive synthesis has focused on finding computationally efficient families of specifications, such
as the fragment of GR(1) formulae (Piterman et al., 2006; Bloem et al., 2012) and good-for-games
automata (e.g., (Henzinger and Piterman, 2006)). Following a similar rationale, we will identify
computationally efficient families of goals for planning, and specifications for reactive synthesis.
EV
Computing Terminating and Non-Terminating Plans. We will present algorithms for
FOND planning that can handle the full class of regular and ω-regular temporally extended goals,
which can be transformed into finite state automata. Our algorithms extend the automata-based
approach by Baier and McIlraith (2006a) for deterministic planning, and also adapt some of the
PR

techniques used in reactive synthesis (Schewe and Finkbeiner, 2007) to manipulate infinite-word
automata. Most of the work on planning with temporally extended goals had focused on computing
plans for deterministic domains that terminate in finite time (e.g., (Bacchus and Kabanza, 1996;
Baier and McIlraith, 2006a,c; Torres and Baier, 2015; Triantafillou et al., 2015)). Existing work in
this area had been very limited, and was limited to PDDL3 goals (Edelkamp, 2006), deterministic
domains (Patrizi et al., 2011), CTL (Pistore and Traverso, 2001; Pistore et al., 2001; Dal Lago et al.,
2002), and a subset of LTL goals in fair FOND domains (Patrizi et al., 2013).
FOND Planners as a Tool for FOND Planning with Temporally Extended Goals. We
will contribute algorithms for FOND planning with temporally extended goals, based on compila-
tions to FOND planning with final-state goals. By developing compilation-based approaches, we
will be able to take advantage of all the existing algorithmic advances in planning—in particular,
heuristics—for the purpose of planning with temporally extended goals. Heuristics are commonly
used in planning to guide the search for solutions (e.g., (Bonet and Geffner, 2001)). The develop-
ment of domain-general heuristics (that can be computed automatically and very efficiently from the
problem description) is a very active research area. More specific to FOND planning is the search
for solutions that presume that the environment is fair (Cimatti et al., 2003), because fairness often
makes the search for solutions easier in practice (Muise et al., 2012).
CHAPTER 1. INTRODUCTION 4

FOND Planners as a Tool for Realizabilizy and Synthesis. We will also contribute algo-
rithms for reactive synthesis via FOND planning. Given the prohibitive complexity of the problem
and the limited scalability of existing methods, it is worth exploring alternative approaches and
bringing algorithmic techniques from other fields. Noticeably, FOND planners usually perform a
search forwards (Mattmüller et al., 2010; Muise et al., 2012), which differs from traditional meth-
ods that use boolean satisfiability (SAT), satisfiability modulo theories (SMT), and binary decision
diagrams (BDDs) and backward reasoning (e.g., (Bohy et al., 2012; Zhu et al., 2017)).

1.3 Publications
The work presented in this dissertation is the synthesis of several publications that have been pre-
sented at premier conferences in Artificial Intelligence. These are, in chronological order:

• Alberto Camacho, Eleni Triantafillou, Jorge A. Baier, Christian Muise, and Sheila A. McIlraith.
Non-Deterministic Planning with Temporally Extended Goals: LTL over finite and infinite

W
traces. In Proceedings of the 31st AAAI Conference on Artificial Intelligence (AAAI-17).
AAAI Press, pages 3716-3724, 2017.
• Alberto Camacho, Jorge A. Baier, Christian Muise, and Sheila A. McIlraith. Finite LTL
IE
Synthesis as Planning. In Proceedings of the 28th International Conference on Automated
Planning and Scheduling (ICAPS-18). AAAI Press, pages 29-38, 2018.
• Alberto Camacho, Christian Muise, Jorge A. Baier, and Sheila A. McIlraith. LTL Realizability
EV
via Safety and Reachability Games. In Proceedings of the 27th International Joint Conference
on Artificial Intelligence (IJCAI-18), pages 4683-4691, 2018.
• Alberto Camacho, Christian Muise, Jorge A. Baier, and Sheila A. McIlraith. SynKit: LTL
Synthesis as a Service. In Proceedings of the 27th International Joint Conference on Artificial
PR

Intelligence (IJCAI-18), Demonstration track, pages 5817-5819, 2018.

• Alberto Camacho, Jorge A. Baier, Christian Muise, and Sheila A. McIlraith. Synthesizing Con-
trollers: On the Correspondence Between LTL Synthesis and Non-deterministic Planning. In
Advances in Artificial Intelligence - Proceedings of the 31st Canadian Conference on Artificial
Intelligence (AI 2018), pages 45-59, 2018.
• Alberto Camacho, Meghyn Bienvenu, and Sheila A. McIlraith. Finite LTL Synthesis with
Environment Assumptions and Quality Measures. In Proceedings of the 16th International
Conference on Knowledge Representation and Reasoning (KR-18), pages 4683-4691, 2018.
• Alberto Camacho, Meghyn Bienvenu, and Sheila A. McIlraith. Towards a Unified View of
AI Planning and Reactive Synthesis. In Proceedings of the 29th International Conference on
Automated Planning and Scheduling (ICAPS-19). AAAI Press, pages 58-67, 2019.
• Alberto Camacho and Sheila A. McIlraith. Strong Fully Observable Non-Deterministic Plan-
ning with LTL and LTLf Goals. In Proceedings of the 28th International Joint Conference on
Artificial Intelligence (IJCAI-19), pages 5523-5531.

The development of the doctoral work presented in this dissertation has led to a number of other
research publications that are not presented here. These further publications provide evidence of
CHAPTER 1. INTRODUCTION 5

the potential of our methods and findings to be applied in related research fields. These additional
publications are, in chronological order:

• Alberto Camacho, Christian Muise, and Sheila A. McIlraith. From FOND to Robust Probabilis-
tic Planning: Computing Compact Policies that Bypass Avoidable Deadends. In Proceedings of
the 26th International Conference on Automated Planning and Scheduling (ICAPS-16). AAAI
Press, pages 65-69, 2016.

• Alberto Camacho and Sheila A. McIlraith. Strong-Cyclic Planning when Fairness is Not a
Valid Assumption. In Proceedings of the Workshop on Knowledge-based Techniques for Prob-
lem Solving and Reasoning co-located with 25th International Joint Conference on Artificial
Intelligence (KnowProS@IJCAI), 2016.

• Alberto Camacho, Oscar Chen, Scott Sanner, and Sheila A. McIlraith. Non-Markovian Re-
wards Expressed in LTL: Guiding Search Via Reward Shaping. In Proceedings of the Tenth
International Symposium on Combinatorial Search (SoCS 2017), pages 159-160, 2017.

W
– A version of this paper was honored with an oral presentation at the 1st Workshop on Goal
Specifications for Reinforcement Learning (GoalsRL), co-located with ICML, IJCAI/, and
AAMAS, 2019.
IE
• Alberto Camacho and Sheila A. McIlraith. Learning Interpretable Models Expressed in Linear
Temporal Logic. In Proceedings of the 29th International Conference on Automated Planning
and Scheduling (ICAPS-19). AAAI Press, pages 621-630, 2019.
EV
• Alberto Camacho, Rodrigo Toro, Toryn Q. Klassen, Richard Valenzano, and Sheila A. McIl-
raith. LTL and Beyond: Formal Languages for Reward Function Specification in Reinforce-
ment Learning. In Proceedings of the 28th International Joint Conference on Artificial Intel-
ligence (IJCAI-19), pages 6065-6073, 2019.
PR

• Alberto Camacho and Sheila A. McIlraith. Towards Neural-Guided Reactive Synthesis for LTL
Specifications. In the 3rd Workshop on Knowledge Representation Meets Machine Learning
(KR2ML), co-located with NeurIPS, 2019.

– A version of this paper was presented at the 4th Multi-disciplinary Conference on Rein-
forcement Learning and Decision Making (RLDM), 2019.

1.4 Outline and Contributions by Chapter

The technical contributions presented in the dissertation are divided into two main blocks. In the
first part, we will focus on FOND planning with temporally extended goals. We will relate planning
and synthesis (Chapter 3), first using LTL as the language to represent goals and specifications.
Next, we will introduce FOND planning algorithms that take temporally extended goals in the
more general form of finite state automata, and that make use of standard FOND planners as a
tool (Chapter 4). In the second part we will shift of focus to reactive synthesis. We will recast
specifications as automata games, and present theoretical results (Chapter 5) and algorithms that
use FOND planners as a tool to solve those games (Chapter 6).
CHAPTER 1. INTRODUCTION 6

Part I: FOND Planning with Temporally Extended Goals

In the first part of the dissertation, we will present our contributions to FOND planning with
temporally extended goals. Our results advance the theory and practice with novel theoretical
foundations, algorithms, and an empirical evaluation of our techniques.
Chapter 3: “A Unified View of AI Planning and Reactive Synthesis”. We will establish
a bi-directional correspondence between reactive synthesis for LTL specifications, and the class of
FOND planning problems with temporally extended goals expressed in LTL (Camacho et al., 2019a).
Planning with temporally extended goals in FOND domains had remained largely unexplored until
the development of this thesis, but it became a more active area as we were advancing with novel
results. Almost in parallel with us, De Giacomo and Rubin (2018) introduced complexity results
that decouple the role of the goal (there, specified in LTLf ) within the combined complexity of the
problem. We will present novel complexity results for FOND planning with LTL goals interpreted
over infinite-length traces, and we will also replicate their results for LTLf goals by following a
different approach that bridges reactive synthesis and planning (Camacho et al., 2019a). Our study
of complexity will expose one of the advantages of planning models and algorithms: the compact

W
representation and handling of the dynamics of the problem. We will also exploit our complexity
results to identify two semantically rich families of goals for which the planning problem can be
solved with a reduced worst-case computational complexity: PDDL3 goals and GR(1) goals.
IE
Chapter 4: “Automata-Theoretic Plan Synthesis”. We will present complexity results and
algorithms for FOND planning with automata goal specifications. First, we will extend the com-
plexity results from Chapter 3 to the class of automata goals, which subsumes LTL and LTLf . Then,
EV
we will present the first algorithms for FOND planning with temporally extended goals which are
sound and complete for the whole class of automata goals. Our algorithms can compute plans (Ca-
macho et al., 2017b), and can also determine when a problem is unsolvable (Camacho and McIlraith,
2019b). In doing so, we take inspiration from existing work on plan solvability for classical planning
PR

(Eriksson et al., 2017, 2018b,a; Eriksson and Helmert, 2020), and also from our work on LTL realiz-
ability and synthesis (Camacho et al., 2018a,e). Our techniques address many open fronts for which
no algorithms existed. In comparison, the only other algorithm that computes non-terminating
plans requires the environment dynamics to be fair —i.e., to manifest all the non-determinism in the
limit—and it is limited to the subclass of LTL goal formulae that can be transformed into determin-
istic Büchi automata (Patrizi et al., 2013). When environment fairness is exploited algorithmically,
it makes it easier to compute plans in practice (Muise et al., 2012). We will also present algorithms
to compute plans that presume fairness, and that can handle automata goals. In our empirical
evaluation, we will see that fairness is also advantageous when goals are temporally extended, and
that it can be exploited to make problems easier to solve in practice.

Part II: Reactive Synthesis

In the second part of the dissertation we will focus on reactive synthesis. LTL synthesis has been
well studied theoretically since Pnueli and Rosner introduced the problem in 1989, but algorithms
have limited scalability due to the prohibitive worst-case complexity of the problem. With the
premise that terminating programs shall be easier to synthesize than for non-terminating programs,
De Giacomo and Vardi (2015) introduced the model of reactive synthesis for terminating programs—
CHAPTER 1. INTRODUCTION 7

using LTLf and Linear Dynamic Logic on finite traces (LDLf ) as possible specification languages.
Empirical evidence had later supported their premise (Zhu et al., 2017). Program termination brings
reactive synthesis closer to planning (Camacho et al., 2018c), and opens a window of opportunities
for research that we will explore.
Chapter 5: “Reactive Synthesis: Theory”. We will address two fundamental problems:
realizability and synthesis. Synthesis involves constructing a solution to the specification, whereas
realizability involves deciding whether a solution exists. Realizability allows for debugging faulty
specifications, and it can also be used to verify safety constraints and other properties of the system.
First, we will address realizabilty for traditional LTL specifications, i.e., the setting in which pro-
grams are non-terminating. As usual, we will adopt automata as the computational normal form. We
will introduce bounded realizability techniques that recast LTL realizability as a series of reachability
games (Camacho et al., 2018e). Bounded realizability complements so-called bounded synthesis, a
breakthrough technique that recasts LTL synthesis as a series of safety games (e.g., (Schewe and
Finkbeiner, 2007; Filiot et al., 2009)). We will show that bounded realizability and bounded synthe-
sis are dual methods, and we will provide a holistic view of realizability and synthesis as automata

W
games. We will then switch our focus to specifications for programs that terminate. We will reflect on
the importance of accounting for environment assumptions at the time of computing solutions—and
conversely, how neglecting them can make the specification unrealizable. In planning, environment
IE
assumptions are implicit to the model: they appear in the action dynamics, and also in the form of
environment fairness. We will argue that regular LTLf specifications cannot express certain types of
environment assumptions (Camacho et al., 2018c), which should be interpreted over (hypothetical)
EV
infinite-length traces. We will then introduce the model for constrained LTLf synthesis, that con-
strains the space of possible executions to only those in which environment assumptions (which can
be specified in LTL and interpreted over hypothetical infinite-length traces) are satisfied. We will
provide complexity results and also algorithms. As usual, we will look into special cases that can be
solved more efficiently.
PR

Chapter 6: “Reactive Synthesis: Practice”. We will contribute principled translations to

solve automata games via FOND planning. Our approach is the first that makes use of planning
technology. We will also present the first algorithms to compute certificates of unrealizability for LTLf
specifications (Camacho et al., 2018a). We will evaluate our methods empirically, by making use of
off-the-shelf FOND planners and heuristics. We will see that planning technology can be competitive
with more mature approaches that use SAT and BDDs (e.g., (Bohy et al., 2012; Zhu et al., 2017)),
despite the fact that our translations add some overhead, and that planning tools are not specialized
to the specific dynamics of reactive synthesis. While our planning-based approach does not achieve
a state-of-the-art performance, we expect to see improved performance by developing specialized
planning-based algorithms and identifying families of problems with advantageous computational
properties and structure. Planning technology holds promise to become an effective approach to
sequential decision making, and opportunities for future research on the topic abound.

1.5 Organization of the Dissertation

In the chapters that follow, we begin by providing the necessary background concepts on automated
planning, reactive synthesis, temporal logics, and finite state automata (Chapter 2). As we have
CHAPTER 1. INTRODUCTION 8

mentioned, our contributions are organized into two main blocks. In the first part, we will study
the correspondence between planning and synthesis (Chapter 3). Then, we will present algorithms
for planning with temporally extended goals in the form of finite state automata (Chapter 4).
Our methods are based on compilations to FOND planning with final-state goals. In the second
part, we will switch focus to reactive synthesis. First, we will study realizability and synthesis as
automata games (Chapter 5), and then we will present algorithms that solve those games by using
planning technology (Chapter 6). We will close the dissertation with a discussion of our contributions
and future research directions (Chapter 7). The contributions presented in the dissertation are
transformative with respect to the class of problems that can be modeled and solved. Planning
technology holds promise for being an effective approach for sequential decision making, both as a
model, and also as a tool to be used in different areas of research.

W
IE
EV
PR
Chapter 2

Preliminaries

2.1 Introduction

W
In this thesis, we will study two models for sequential decision making: automated planning for
artificial intelligence, and reactive synthesis. The purpose of this chapter is to review these models,
and introduce the notation that we will use throughout the dissertation. We start in Section 2.2 by
IE
reviewing a selection of temporal logics that are commonly used to describe temporally extended
goals in planning, and specifications for reactive synthesis. Temporal logics have a strong correspon-
dence with finite state automata, that we review in Section 2.3. Finally, we formalize the models
EV
for planning with temporally extended goals (Section 2.4) and reactive synthesis (Section 2.5).
A myriad of temporal logics and other high-level languages have been used to specify tempo-
rally extended goals in planning, and specifications for reactive synthesis. Among those, in this
dissertation we will mostly use Linear Temporal Logic interpreted over infinite- and finite-length
traces. We abbreviate them with LTL and LTLf , respectively. The syntax and semantics of LTL and
PR

LTLf are similar to natural language, and for this reason humans may find it convenient to specify
goals and other behavioral properties of a problem. On the other hand, automata representations
of the specification are more convenient to design algorithms. Given that many of the commonly
used specification languages for planning and reactive synthesis can be transformed into finite state
automata, we will design algorithms that exploit specifications in such form.
We will study planning in fully observable and non-deterministic (FOND) domains (Cimatti
et al., 2003). Whereas FOND planning for final-state goals has been well studied, FOND planning
with temporally extended goals is an area of research that had remained largely unexplored. In this
chapter, we formalize the models for FOND planning with temporally extended goals and describe
their type of solutions. We close this chapter by formalizing the model for reactive synthesis.
In the next chapter (Chapter 3) we will study the relationship between planning and synthesis,
where LTL is central to the specification of temporally extended goals and objectives. In the re-
maining of the dissertation we will place the focus on automata rather than LTL. We will contribute
to FOND planning with temporally extended goals (Chapter 4) and reactive synthesis (Chapter 5
and Chapter 6) with novel theoretical results and algorithms that exploit automata representations,
and leverage FOND planners (originally designed for final-state goals) as the core technology for
planning with temporally extended goals, and also for reactive synthesis.

9
CHAPTER 2. PRELIMINARIES 10

2.2 Temporal Logics

Linear Temporal Logic (LTL) was introduced by Pnueli (1977) for program verification. Since then, a
myriad of temporal logics and other formal languages have been used to formally describe temporally
extended properties of state traces in a variety of applications. In particular, temporal logics have
been used to specify goals and objectives in sequential decision-making models such as automated
planning (see Section 2.4) and reactive synthesis (see Section 2.5). Both models are central in the
development of this thesis. In what follows, we review a handful of popular temporal logics, making a
distinction between logics that are interpreted over finite- and infinite-length state traces. In Section
2.3, we review their correspondence with finite-state automata.

2.2.1 Linear Temporal Logics

LTL is a propositional modal logic commonly used to express temporally extended properties of state
trajectories, first developed for program verification (Pnueli, 1977). The syntax of LTL is defined
over a set of propositional variables AP , and includes the standard logical connectives (∧, ∨, ¬),

W
unary modal operator next ( ), and binary modal operator until ( U ). Formally,

ϕ := p | > | ⊥ | ¬ϕ | ϕ1 ∨ ϕ2 | ϕ1 ∧ ϕ2 | ϕ | ϕ1 U ϕ2
IE
where p ∈ AP . We review the semantics of LTL below. Other temporal operators are commonly
used, and are defined in terms of the basic operators. Namely, eventually (♦ϕ := > U ϕ), always
(ϕ := ¬♦¬ϕ), release (ϕ1 R ϕ2 := ¬(¬ϕ1 U ¬ϕ2 ))), and weak until (ϕ1 W ϕ2 := ϕ1 ∨ (ϕ1 U ϕ2 )).
EV
We denote by |ϕ| the size of LTL formula ϕ, i.e., its total number of symbols.

Infinite-Word Semantics

LTL formulae are evaluated over infinite traces, i.e., sequences π = s1 s2 . . . of infinite length where
PR

each si ⊆ AP is a subset of propositional variables and defines a propositional valuation. Intuitively,

propositional formulae tell what need to hold in the current time step; a formula ϕ states that ϕ
holds in the next time step; and ϕ1 U ϕ2 states that ϕ1 needs to hold until ϕ2 holds. Formally, we
say that an infinite trace π satisfies LTL formula ϕ (and we write π |= ϕ, for short) if π, 1 |= ϕ,
where for every natural number i ≥ 1:

• π, i |= p, for a propositional variable p ∈ AP , if p ∈ si ,

• π, i |= ¬ψ if it is not the case that π, i |= ψ,
• π, i |= (ψ ∧ χ) if π, i |= ψ and π, i |= χ,
• π, i |= ϕ if π, i + 1 |= ϕ,
• π, i |= ϕ U ψ if there exists a j ≥ i such that π, j |= ψ, and π, k |= ϕ for every i ≤ k < j.

Finite-Word Semantics

LTL interpreted over finite traces has received significant attention from the planning community
(e.g., (Baier and McIlraith, 2006a; Gerevini et al., 2009)). The use of LTL interpreted over finite
traces for planning goes back at least to 1996 with the work of Bacchus and Kabanza (1996). The
CHAPTER 2. PRELIMINARIES 11

use of LTL interpreted over finite traces with forward semantics has been used to express temporally
extended goals (e.g. (Baier and McIlraith, 2006c)) and preferences (e.g. (Bienvenu et al., 2011)).
De Giacomo and Vardi (2013) provided a formal description and named it LTLf . LTLf inherits the
syntax of LTL. Unlike LTL, the semantics of LTLf are evaluated over finite traces. As usual, we
denote with |π| the length of a trace π. We say that a finite trace π = s1 , . . . , sn satisfies LTLf
formula ϕ (π |= ϕ, for short) if π, 1 |= ϕ, where for every i ≥ 1:

• π, i |= p, for a propositional variable p ∈ P, if p ∈ si ,

• π, i |= ¬ψ if it is not the case that π, i |= ψ,
• π, i |= (ψ ∧ χ) if π, i |= ψ and π, i |= χ,
• π, i |= ϕ if i < |π| and π, i + 1 |= ϕ,
• π, i |= ϕ U ψ if there exists a i ≤ j ≤ |π| such that π, j |= ψ, and π, k |= ϕ for every i ≤ k < j,

The semantics of LTLf is similar to the semantics of LTL. One the main differences is how the
next ( ) operator evaluates on finite traces. In LTLf , a macro final := ¬ > is commonly used to

W
indicate the end of the trace. The weak next ( ) operator, defined by ϕ := ϕ ∨ final, indicates
what needs to hold in the next state in case it exists. In LTL, ϕ is equivalent to ϕ (and we write
ϕ ≡ ϕ). In other words, the traces that satisfy ϕ are the same that satisfy ϕ. It is important
IE
to notice that the equivalence is not true in LTLf . Similarly, ¬ ϕ 6≡ ¬ϕ in LTLf , and instead we
have ¬ ϕ ≡ final ∨ ¬ϕ or, using the weak-next operator, ¬ ϕ ≡ ¬ϕ. Note that the equivalence
¬ ϕ ≡ ¬ϕ holds in LTL and LTLf .
EV
Linear Temporal Logic of the Past

Linear Temporal Logic of the Past (also known as Past LTL, or simply PLTL) is a version of LTL
interpreted over finite traces (Emerson, 1990). The syntax of PLTL is analogous to that of LTLf ,
PR

with the difference that the semantics of temporal operators are interpreted backwards from the last
state. In this dissertation, the reference to PLTL follows Bacchus et al. (1996). PLTL has been used
to express non-Markovian rewards in MDPs (Bacchus et al., 1996, 1997), and to describe preferred
explanations in the context of dynamical diagnosis (Sohrabi et al., 2011).
PLTL has modal operators previously () and since (S)—whose semantics are the past versions
of the LTL operators next and until, respectively. We say that a finite trace π = s1 s2 . . . sn satisfies
ϕ (π |= ϕ, for short) iff π, n |= ϕ, where for every i ≤ n:

• π, i |= p, for a propositional variable p ∈ P, if p ∈ sn ,

• π, i |= ¬ψ if it is not the case that π, i |= ψ,
• π, i |= (ψ ∧ χ) if π, i |= ψ and π, i |= χ,
• π, i |= ϕ if i > 1 and π, i − 1 |= ϕ,
• π, i |= ϕSψ if there exists a 1 ≤ j ≤ i such that π, j |= ψ, and π, k |= ϕ for every j ≤ k < |π|,

For convenience, we write start ≡ ¬>. PLTL also includes the operator weak previously (),
where α ≡ α ∨ start. The operators eventually in the past () and always in the past () are
defined analogously to its respective LTL counterparts eventually and always.
CHAPTER 2. PRELIMINARIES 12

modal operator equivalent LTLf

(at-end θ) ♦(θ ∧ final)
(always θ) θ
(sometime θ) ♦θ
(sometime-after θ1 θ2 ) (θ1 → ♦θ2 )
(sometime-before θ1 θ2 ) θ2 R ¬θ1
(at-most-once θ) (¬θ) W (θ W (¬θ))
i
W
(within n θ) 0≤i≤n Wθ
(always-within n θ1 θ2 ) (θ1 → 0≤i≤n i θ2 )
i i
W V
* (hold-during n1 n2 θ) 0≤i≤n1 (θ ∧ final) ∨ n1 <i≤n2 (θ)
i
W
* (hold-after n θ) 0≤i≤n+1 (θ ∧ final)
Table 2.1: PDDL3 modal temporal operators and equivalent LTLf formulae. We provide corrected
LTLf formulae for operators tagged with ∗.

2.2.2 Temporal Operators in the Planning Domain Definition Language

W
The Planning Domain Definition Language specification, version 3.0 (PDDL3), standardized a set
of modal operators to express temporally extended goals and preferences (Gerevini and Long, 2005;
Gerevini et al., 2009). Table 2.1 shows the PDDL3 modal operators and their equivalent LTLf
IE
formulae as described in (Gerevini et al., 2009). We provided corrected versions of the LTLf formulae
for the operators tagged with an asterisk (∗). Whereas PDDL3 formulae can express many properties
that appear in practice, its expressiveness is somewhat limited. For example, arbitrary nesting of
EV
temporal operators is not allowed.

2.2.3 Other Temporal Logics

Other temporal logics exist and have been used for the description of temporally extended goals
PR

in planning. For example, Computational Tree Logic (CTL) (Emerson, 1990) is a branching-time
temporal logic that has been used to describe goals in FOND planning (Pistore et al., 2001). CTL
and LTL are non-equivalent languages (Lamport, 1980), and the logic called CTL∗ subsumes both
(Emerson and Halpern, 1983). Linear Dynamic Logic over Finite Traces (LDLf ) (De Giacomo and
Vardi, 2013) is a temporal logic with semantics interpreted over finite traces. LDLf is strictly more
expressive than LTLf (although, perhaps, less intuitive), and it that has been proposed for use as
specification language in reactive synthesis and to describe goals in automated planning. Before
LTLf was formally introduced (De Giacomo and Vardi, 2013), other versions of LTL interpreted over
finite traces were already used for describing temporally extended goals in planning, such as so-called
f-LTL (Baier and McIlraith, 2006c), which is equivalent to LTLf , and a lifted version of f-LTL which
uses first order logic, named f-FOLTL (Baier and McIlraith, 2006a).

2.3 Automata
Finite state automata are mathematical structures that can be used as acceptors to recognize lan-
guages. An automaton is a tuple A = hQ, Σ, q0 , δ, αi, where Q is a finite set of automaton states,
Σ is a finite alphabet of input symbols, q0 ∈ Q is the initial state, and δ : Q × Σ → 2Q is the
CHAPTER 2. PRELIMINARIES 13

> robot at ?R1

robot at ?R1

start q0 q1
>
Figure 2.1: The figure above is a representation of a finite-state automaton. Automaton states
are represented by circles, and accepting states are denoted with double circles. The initial state is
denoted with an inner arrow labeled with the word “start”, and transitions are represented by arrows
labeled with guards. The automaton above is an NBW automaton transformation of the LTL formula
♦robot at ?R1. The automaton is also an NFW automaton transformation of the LTLf formula
♦robot at ?R1 which, with finite semantics, is equivalent to LTLf formula ♦(robot at ?R1 ∧ end).

transition function. For our purposes, Σ = 2AP . Figure 2.1 illustrates a graphical representation of
a non-deterministic finite-state automaton.
The transition function is deterministic when the set δ(q, s) ⊆ Q is a singleton for each pair
(q, s) ∈ Q × Σ. In this case, it is usual to simply write δ : Q × Σ → Q. It is also convenient
to describe δ with a set of automaton transitions. In this thesis, a transition is a triplet (q, ψ, q 0 ),

W
where q, q 0 ∈ Q and ψ is a propositional formula over AP referred to as the guard of the transition.
Intuitively, the guard tells when a transition triggers. In the natural way, we say that a set of
transitions T describes δ if, for each q, q 0 ∈ Q and s ∈ Σ: q 0 ∈ δ(q, s) iff there exists (q, ψ, q 0 ) ∈ T
IE
such that s |= ψ. In this dissertation, we will usually express the guard ψ in a transition (q, ψ, q 0 ) as
W
the disjunction of subformulae, ψ = j∈J ψj (for example, in DNF form, although other arbitrary
forms such as ψ = {s ∈ Σ | q 0 ∈ δ(q, s)} are also valid). Then, we will decompose each transition
W
EV
(q, ψ, q 0 ) into the set of transitions (q, ψj , q 0 ), one for each j ∈ J.
The last element in the description of an automaton, α, serves to characterize the condition under
which a word is accepted. An automaton may accept a finite- or infinite-length word, depending on
their semantics. We denote with L(A) the language of an automaton A, that is, the set of words that
are accepted by A. In what follows, we review different classes of finite, and infinite word automata
PR

and their well-known correspondence with temporal logics (see, e.g., (Baier and McIlraith, 2006c;
Kupferman and Vardi, 2005)).

2.3.1 Finite-Word Automata

As the name suggests, finite-word automata are used to recognize finite words. More precisely, there
exists an equivalence between the languages recognized by finite-word automata and those that can
be defined by a regular expression (Yu, 1997)—so-called regular languages. We first review how we
check whether a finite word is in the language defined by an automaton. Then, we comment on the
special case in which the transition function of the automaton is deterministic.
Non-deterministic Finite-Word (NFW) automata. In NFW automata, the element α is a
subset of Q that denotes the set of accepting states. A run of an NFW automaton A on a finite word
w = s1 · · · sn ∈ Σ∗ is a maximal (and finite) sequence ρ = q0 · · · qm where qi ∈ δ(qi−1 , si ) for each
0 < i < |ρ|. By maximal we mean that m is the maximum integer, not greater than n, such that
δ(qi−1 , si ) 6= ∅ for each 0 < i < |ρ|. In other words, runs have to be progressed until the transition
function maps to the empty set, or m = n. Finally, a run ρ = q0 · · · qm is accepting if m = n and
qm ∈ α. An NFW automaton A accepts a finite word w ∈ Σ∗ if some run of A on w is accepting.
CHAPTER 2. PRELIMINARIES 14

¬request true true ¬response

request request

start q0 q1 start q0 q1

response ∧ ¬request
(a) NBW automaton for LTL formula ϕ (b) UCW automaton for LTL formula ϕ.
.
Figure 2.2: Automaton transformations of the LTL formula ϕ = (request → ♦response). In
English, the LTL formula tells that “each request must be eventually followed by some response”.

Deterministic Finite-Word (DFW) automata. DFW automata are a particular case of NFW
automata, with deterministic transition function. Note, the runs of a DFW automaton on a finite
word w are unique, and have length |w| + 1. NFW automata can be determinized into equivalent
DFW automata (i.e., automata that recognize the same language) in worst-case exponential time
with a worst-case exponential blowup in the number of states. The powerset construction is a

W
well-known mechanism to determinize NFW automata (Rabin and Scott, 1959).

2.3.2 Infinite-Word Automata IE

Infinite-word automata are used to recognize infinite words. There exists an equivalence between the
languages recognized by Non-Deterministic Büchi Automata, and those that can be defined by an
ω-regular expression—so-called ω-regular languages (see, e.g., (Thomas, 1990)). Such equivalence
EV
can be trivially extended to Universal co-Büchi Automata as well. In what follows, we review
infinite-word automata with different types of accepting conditions.
Non-deterministic Büchi Word (NBW) automata. In NBW automata, the element α is a
subset of Q and describes a set of accepting states. A run of an NBW automaton A on an infinite
PR

word w = s1 s2 · · · ∈ Σω is a maximal (perhaps finite) sequence ρ = q0 q1 · · · where qi ∈ δ(qi−1 , si )

for each 0 < i < |ρ|. Similar to how we defined maximality in finite-word automata, here by
maximal we mean that |ρ| is the maximum integer (or ∞, if such integer is unbounded) such that
δ(qi−1 , si ) 6= ∅ for each 0 < i < |ρ|. Before defining what are the accepting runs, we shall define
operators occ(ρ) : Q → N ∪ {0, ∞}, indexed by a run ρ, that return for each q ∈ Q the number
of occurrences of q in ρ. A run ρ of NBW automaton A on an infinite word w = s0 s1 · · · ∈ Σω is
accepting if occ(ρ)(q) = ∞ for some q ∈ α. In other words, a run ρ is accepting when some state in
α occurs infinitely often in ρ. Finally, A accepts w if some run of A on w is accepting.
Deterministic Büchi Word (DBW) automata. DBW automata are NBW automata with
deterministic transition functions. Whereas NFW automata can be always determinized into DFW
automata, NBW automata cannot be always determinized into DBW automata. In other words,
NBW automata are strictly more expressive than DBW automata (Landweber, 1969).
Universal co-Büchi Word (UCW) automata. UCW automata are the dual of NBW au-
tomata. In UCW automata, α ⊆ Q is a set of rejecting states. Runs are defined as in NBW
automata, but the acceptance condition is different. A run of UCW automaton A on an infinite
word w = s1 s2 · · · ∈ Σω is accepting if occ(ρ)(q) < ∞ for all q ∈ α. In other words, none of the
states in α occurs infinitely often in ρ. Finally, A accepts w if all runs of A on w are accepting.
CHAPTER 2. PRELIMINARIES 15

Automaton Can they be determinized?

NFW They can be determinized symbolically into DFW automata (Rabin and Scott, 1959).
NBW Sometimes, they can be determinized into DBW automata (Krishnan et al., 1994).
In general, they can be determinized into DPW automata (Piterman, 2007).
UkCW They can be determinized symbolically (Schewe and Finkbeiner, 2007).
NkBW They can be determinized symbolically (Camacho et al., 2018e).

Table 2.2: Some computational methods that make use of automata require deterministic automata.
The so-called Safra construction to determinize NBW into parity automata is involved and does
not scale well. Certain types of automata can be determinized symbolically with the powerset
construction, which is simpler and more scalable than the Safra construction.

Universal k-co-Büchi Word (UkCW) automata. UkCW automata are similar to UCW
automata. However, an UkCW automaton A, with k < ∞, accepts a word w when all runs of A on
w hit a number of states in α that is bounded by k. That is, Σq∈α occ(ρ)(q) < k in each run ρ.
Non-deterministic k-Büchi Word (NkBW) automata. NkBW automata are similar to

W
NBW automata. The difference is that an NkBW automaton A, with k < ∞, accepts a word w
when some run of A on w hit at least k states in α. That is, Σq∈α occ(ρ)(q) >= k for some run ρ.
Deterministic Parity Word (DPW) automata. DPW automata are automata with deter-
ministic transition function, where α : Q → N is a labeling function that assigns numbers (also called
IE
colors) to automaton states. The number of different colors that α can take is called the index of the
automaton. The runs of a DPW automaton are defined as with DBW automata. A run ρ = q0 q1 · · ·
of a DPW is accepting when the smallest color that appears infinitely often in α(q0 )α(q1 ) · · · is even.
EV

2.3.3 Operations on Automata

The techniques presented in this thesis make use of automata decompositions into sums and products
of other automata. These operations generate new automata that accept the union or intersection
PR

of the languages of each automaton. We also make use of automata complementation. We review
these concepts below. It is also common that algorithms rely on deterministic automata. Automata
determinizations typically incur into an exponential blowup. Some types of automata are amenable
to symbolic determinizations, that can be performed with an improved scalability (see Table 2.2).

Sum and Product of Automata

The sum and product of two automata A(1) and A(2) are the automata A(1) ⊕ A(2) and A(1) ⊗ A(2) ,
respectively, with components defined as follows. The semantics of the sum and product of automata
is commutative and associative, in the sense that the language of the resulting automaton is not
affected by the order of automata in binary operations, and parenthesizing. For simplicity, we
obviate writing parentheses in the sums and products of more than two automata.

(1)
A(1) = hQ(1) , Σ(1) , q0 , δ (1) , α(1) i
(2)
A(2) = hQ(2) , Σ(2) , q0 , δ (2) , α(2) i
(1) (2)
A(1) ⊕ A(2) := hQ(1) × Q(2) , Σ(1) ∪ Σ(2) , (q0 , q0 ), δ (1) × δ (2) , (α(1) × Q(2) ) ∪ (Q(1) × α(2) )i
(1) (2)
A(1) ⊗ A(2) := hQ(1) × Q(2) , Σ(1) ∪ Σ(2) , (q0 , q0 ), δ (1) × δ (2) , α(1) × α(2) i
 CHAPTER 2. PRELIMINARIES 16

In the equations above, the cross-product of two transition functions is defined in the natural way.
That is, δ (1) × δ (2) (q, s) := {(q1 , q2 ) | q1 ∈ δ (1) (q, s), q2 ∈ δ (2) (q, s))}. Likewise, the cross-product

of two sets α(1) and α(2) is the cartesian product α(1) × α(2) := (q1 , q2 ) | q1 ∈ α(1) and q2 ∈ α(2) .
The following properties hold.

Proposition 2.1. The sum automaton A(1) ⊕ A(2) of two NFW (resp. NBW) automata A(1) and
A(2) is an NFW (resp. NBW) automaton that accepts the union of the languages of A(1) and A(2) .

Proof. We proof the result for NFW automata. By definition, a finite word w is accepted by A(1) iff
there exists a run of A(1) on w that finishes in a state in α(1) , and this occurs iff there exists a run
of A(1) ⊕ A(2) on w that finishes in a state in α(1) × Q(2) . Likewise, w is accepted by A(2) iff there
exists a run of A(1) ⊕ A(2) on w that finishes in a state in Q(1) × α(2) . As such, the sum automaton
A(1) ⊕ A(2) accepts the union of the languages of A(1) and A(2) . The result for NBW automata
follows with an analogous reasoning.

Proposition 2.2. The product automaton A(1) ⊗ A(2) of two NFW automata A(1) and A(2) is an
NFW automaton that accepts the intersection of the languages of A(1) and A(2) .

W
Proof. By definition, a finite word w is accepted by A(1) iff there exists a run of A(1) on w that
finishes in a state in α(1) , and this occurs iff there exists a run of A(1) ⊗ A(2) on w that finishes in
IE
a state in α(1) × Q(2) . Likewise, w is accepted by A(2) iff there exists a run of A(1) ⊗ A(2) on w
that finishes in a state in Q(1) × α(2) . Because α(1) ⊗ α(2) ≡ (α(1) ⊗ Q(2) ) ∩ (Q(1) × α(2) ), the words
accepted by the product automaton A(1) ⊗ A(2) are all and only those accepted by the automata
EV
A(1) and A(2) .

Proposition 2.3. The sum automaton A(1) ⊕A(2) of two UCW automata A(1) and A(2) is an UCW
automaton that accepts the intersection of the languages of A(1) and A(2) .
PR

Proof. By definition, an infinite word w is accepted by A(1) iff there exists a run of A(1) on w that
hits a finite number of states in α(1) , and this occurs iff there exists a run of A(1) ⊕ A(2) on w that
hits a finite number of states in α(1) × Q(2) . Likewise, w is accepted by A(2) iff there exists a run of
A(1) ⊕ A(2) on w that hits a finite number of states in Q(1) × α(2) . As such, the words accepted by
both automata A(1) and A(2) are those for which there exists a run of A(1) ⊕ A(2) that hits a finite
number of states in (α(1) × Q(2) ) ∪ (Q(1) × α(2) )—that is, the words accepted by the sum automaton
A(1) ⊕ A(2) .

Complementation

The complementation of an infinite-word (resp. finite-word) automaton A is the task of constructing

an automaton that accepts the complement language that is accepted by A, i.e., the infinite (resp.
finite) words that are not in L(A). Some classes of automata are easy to complement. For example,
a DFW automaton can be complemented into another DFW automaton that flips accepting and
non-accepting states (see Proposition 2.4 on the next page). An NFW automaton can be comple-
mented by first determinizing ir, and then complementing the resulting automaton. The duality
between NBW and UCW automata can be exploited for complementation (see Proposition 2.5 on
the following page).

Reproduced with permission of copyright owner. Further reproduction prohibited without permission.