Planning, Execution & Learning

1. Linear & Non-Linear Planning

Reid Simmons
 Linear Planning
• Basic Idea
– Work on one goal until completely solved before moving on
to the next goal

• Planning Algorithm Maintains Goal Stack

• Implications
– No interleaving of goal achievement
– Efficient search if goals do not interact (much)

2
 Means-Ends Analysis
• Basic Idea
– Search only relevant aspects of problem
– What means (operators) are available to achieve the
desired ends (goal)

• Find difference between goal and current state

• Find operator to reduce difference
• Perform means-ends analysis on new subgoals

3
 GPS
• General Problem Solver [Newell, Simon, Ernst, 1960’s]
– Introduced concept of means-ends analysis
– Essentially linear planning using recursive procedure calls as
the goal-stack mechanism
• GPS Algorithm (initial-state, goals)
– If goals ⊆ initial-state then return (initial-state, [])
– Choose a difference d between initial-state and goals
– Choose an operator o to reduce the difference d
– If no applicable operators, then return (∅, [])
– (state, plan) = GPS(initial-state, preconditions(o))
– If state ≠ ∅ then
• (state, rest-plan) = GPS(apply(o, state), goals)
• plan = [plan; o; rest-plan]
– Return (state, plan)

4
 STRIPS
• STanford Research Institute Problem Solver
[Fikes, Nilsson, 1971]
– Same basic idea as GPS, but
– Solved the frame problem (“STRIPS assumption”)
– Introduced operator representation
– Operationalized notion of difference, subgoals, and operator
application
– Dealt (somewhat) with plan execution and learning

5
 STRIPS Algorithm
• STRIPS (initial-state, goals)
– state = initial-state; plan = []; stack = []
– Push goals on stack
– Repeat until stack is empty
• If top of stack is goal that matches state, then pop stack
• Else if top of stack is a conjunctive goal g, then
– Select an ordering for the subgoals of g, and push them on
stack
• Else if top of stack is a simple goal sg, then
– Choose an operator o whose add-list matches goal sg
– Replace goal sg with operator o
– Push the preconditions of o on the stack
• Else if top of stack is an operator o, then
– state = apply(o, state)
– plan = [plan; o]
6 Simmons, Veloso : Fall 2001
 STRIPS Meets the Blocks World
B B
A C A C
Table Table
Pickup_from_table(b) Pickup_from_block(b, c)
Pre: Block(b), Handempty Pre: Block(b), Handempty
Clear(b), On(b, Table) Clear(b), On(b, c), Block(c)
Add: Holding(b) Add: Holding(b), Clear(c)
Delete: Handempty,
Delete: Handempty,
On(b, Table)
On(b, c)
Putdown_on_table(b) Putdown_on_block(b, c)
Pre: Block(b), Holding(b) Pre: Block(b), Holding(b)
Add: Handempty, Block(c), Clear(c), b ≠ c
On(b, Table)
Add: Handempty, On(b, c)
Delete: Holding(b)
Delete: Holding(b), Clear(c)
7
 STRIPS Blocks-World Example
1. Stack State 2. Stack State
On(A, C) On(C, B) Clear(B) On(A, C) On(C, B) Clear(B)
Clear(C) Clear(C)
On(A, C)
On(C, A) On(C, A)
On(A, Table) On(C, B) On(A, Table)
On(B, Table) On(B, Table)
Handempty Handempty
A
C C
B A B
Goal Initial State 4. Stack State
On(A, C) On(C, B) Clear(B)
3. Stack State
Clear(C)
On(A, C) On(C, B) Clear(B) On(A, C)
On(C, A)
Clear(C) Put_Block(C, B) On(A, Table)
On(A, C) On(C, A) On(B, Table)
Put_Block(C, B) On(A, Table) Holding(C) Clear(B)
Handempty
On(B, Table) Holding(C)
Holding(C) Clear(B)
Handempty
Clear(B)

8
 STRIPS Blocks-World Example
5. Stack State 6. Stack State
On(A, C) On(C, B) Clear(B) On(A, C) On(C, B) Clear(B)
Clear(C) Clear(C)
Clear(C)
On(A, C) On(A, C)
On(C, A) On(C, A)
On(C, A)
Put_Block(C, B) On(A, Table) Put_Block(C, B) On(A, Table)
On(B, Table) On(B, Table)
Holding(C) Clear(B) Holding(C) Clear(B)
Handempty Handempty
Holding(C) Pick_Block(C)
Handempty Clear(C) On(C, ?b)

7. Stack State 8. Stack State

On(A, C) On(C, B) Clear(B) On(A, C) On(C, B) Clear(B)
Clear(B)
Clear(C) Clear(C)
On(A, C) On(A, C)
On(C,
On(C, A) On(A, Table)
Put_Block(C, B) On(A, Table) Put_Block(C, B) On(B, Table)
On(B, Table) Holding(C)
Holding(C)
Holding(C) Clear(B) Holding(C) Clear(B)
Handempty Clear(A)
Pick_Block(C)
[Pick(C)]
9
 STRIPS Blocks-World Example
9. Stack State 10. Stack State
On(A, C) On(C, B) Clear(B) On(A, C) On(C, B) Clear(C)
Clear(C) On(A, Table)
On(A, C) On(A, C)
On(A, Table) On(B, Table)
Put_Block(C, B) On(B, Table) Clear(A)
Holding(C)
Holding(C) Handempty
Clear(A) On(C, B)
[Pick(C)] [Pick(C); Put(C, B)]

11. Stack State 12. Stack State

On(A, C) On(C, B) Clear(C) On(A, C) On(C, B) Clear(C)
On(A, Table) On(A, Table)
Put_Block(A, C) Put_Block(A, C)
On(B, Table) On(B, Table)
Holding(A) Clear(C) Clear(A) Holding(A) Clear(C) Clear(A)
Handempty Handempty
Holding(A)
On(C, B) On(C, B)
Clear(C)
[Pick(C); Put(C, B)]
[Pick(C); Put(C, B)]

10
 STRIPS Blocks-World Example
13. Stack State 14. Stack State
On(A, C) On(C, B) Clear(C) On(A, C) On(C, B) Clear(C)
On(A, Table) On(A, Table)
Put_Block(A, C) Put_Block(A, C)
On(B, Table) On(B, Table)
Holding(A) Clear(C) Clear(A) Holding(A) Clear(C) Clear(A)
Handempty Handempty
Holding(A) Pick_Table(A)
On(C, B) On(C, B)
Handempty Clear(A)
On(A, Table)
[Pick(C); Put(C, B)]
[Pick(C); Put(C, B)]
15. Stack State 16. Stack State
On(A, C) On(C, B) Clear(C) On(A, C) On(C, B) Clear(C)
On(A, Table) On(B, Table)
Put_Block(A, C) On(B, Table) Put_Block(A, C) Clear(A)
Holding(A) Clear(C) Clear(A) Holding(A) Clear(C) On(C, B)
Handempty Holding(A)
Pick_Table(A)
On(C, B)

[Pick(C); Put(C, B)] [Pick(C); Put(C, B); PickT(A)]

11
 STRIPS Blocks-World Example
17. Stack State 18. Stack State
On(A, C) On(C, B) Clear(C)
Clear(C) On(A, C) On(C, B) On(B, Table)
On(B, Table) Clear(A)
Put_Block(A, C) Clear(A) On(C, B)
On(C, B) Handempty
Holding(A) On(A, C)

[Pick(C); Put(C, B); PickT(A)] [Pick(C); Put(C, B); PickT(A); Put(A, C)]

19. Stack State

On(B, Table)
Clear(A)
On(C, B)
Handempty
On(A, C)

[Pick(C); Put(C, B); PickT(A); Put(A, C)]

12
 Properties of Planning Algorithms
• Soundness
– A planning algorithm is sound if all solutions found are legal
plans
• All preconditions and goals are satisfied
• No constraints are violated (temporal, variable binding)
• Completeness
– A planning algorithm is complete if a solution can be found
whenever one actually exists
– A planning algorithm is strictly complete if all solutions are
included in the search space
• Optimality
– A planning algorithm is optimal if the order in which
solutions are found is consistent with some measure of plan
quality
13
 Linear Planning: Discussion
• Advantages
– Reduced search space, since goals are solved one at a time
– Advantageous if goals are (mainly) independent
– Linear planning is sound

• Disadvantages
– Linear planning may produce suboptimal solutions
(based on the number of operators in the plan)
– Linear planning is incomplete

14
 Suboptimal Plans
• Result of linearity, goal interactions and poor goal ordering
Load(o, p, loc) Unload(o, p, loc)
Pre: At(o, loc), At(p, loc) Pre: Inside(o, p), At(p, loc)
Add: Inside(o, p) Add: At(o, loc)
Delete: At(o, loc) Delete: Inside(o, p)
Fly(p, from, to)
Pre: At(p, from)
Add: At(p, to)
Delete: At(p, from)
• Initial State: At(Obj1, LocA), At(Obj2, LocA), At(747, LocA)
• Goals: At(Obj1, LocB), At(Obj2, LocB)
• Plan: [Load(Obj1, 747, LocA); Fly(747, LocA, LocB);
Unload(Obj1, 747, LocB); Fly(747, LocB, LocA);
Load(Obj2, 747, LocA); Fly(747, LocA, LocB); Unload(Obj2, 747, LocB)]

15
 STRIPS and the FedEx World
1. Stack State 2. Stack State
At(Obj1, LocB) At(Obj1, LocA) At(Obj1, LocB) At(Obj2, LocB) At(Obj1, LocA)
At(Obj2, LocB) At(Obj2, LocA) At(Obj2, LocB) At(Obj2, LocA)
At(747, LocA) At(747, LocA)
At(Obj1, LocB)

3. Stack State 4. Stack State

At(Obj1, LocB) At(Obj2, LocB) At(Obj1, LocA) At(Obj1, LocB) At(Obj2, LocB) At(Obj1, LocA)
At(Obj2, LocB) At(Obj2, LocA) At(Obj2, LocB) At(Obj2, LocA)
At(747, LocA) At(747, LocA)
Unload(Obj1, 747, LocB) Unload(Obj1, 747, LocB)
Inside(Obj1, 747) At(747, LocB) Inside(Obj1, 747) At(747, LocB)
Inside(Obj1, 747)
At(747, LocB)

16
 STRIPS and the FedEx World
5. Stack State 6. Stack State
At(Obj1, LocB) At(Obj2, LocB) At(Obj1, LocA) At(Obj1, LocB) At(Obj2, LocB) At(Obj1, LocA)
At(Obj2, LocB) At(Obj2, LocA) At(Obj2, LocB) At(Obj2, LocA)
At(747, LocA)
LocA) At(747, LocA)
LocA)
Unload(Obj1, 747, LocB) Unload(Obj1, 747, LocB)
Inside(Obj1, 747) At(747, LocB) Inside(Obj1, 747) At(747, LocB)
Inside(Obj1, 747) Inside(Obj1, 747)
Fly(747, ?loc, LocB) Fly(747, LocA, LocB)
At(747, ?loc)
7. Stack State 8. Stack State
At(Obj1, LocB) At(Obj2, LocB) At(Obj1, LocA) At(Obj1, LocB) At(Obj2, LocB) At(Obj1, LocA)
At(Obj2, LocB) At(Obj2, LocA) At(Obj2, LocB) At(Obj2, LocA)
At(747, LocB) At(747, LocB)
Unload(Obj1, 747, LocB) Unload(Obj1, 747, LocB)
Inside(Obj1, 747) At(747, LocB) Inside(Obj1, 747) At(747, LocB)
Inside(Obj1, 747) Load(Obj1, 747, ?loc) [Fly(747, LocA, LocB)]
[Fly(747, LocA, LocB)]
At(Obj1, ?loc) At(747, ?loc)
17
 STRIPS and the FedEx World
9. Stack State 10. Stack State
At(Obj1, LocB) At(Obj2, LocB) At(Obj1, LocA) At(Obj1, LocB) At(Obj2, LocB) At(Obj1, LocA)
At(Obj2, LocB) At(Obj2, LocA) At(Obj2, LocB) At(Obj2, LocA)
At(747, LocB)
LocA)
LocB) At(747, LocB)
Unload(Obj1, 747, LocB) Unload(Obj1, 747, LocB)
Inside(Obj1, 747) At(747, LocB) Inside(Obj1, 747) At(747, LocB)
Load(Obj1, 747, ?loc) [Fly(747, LocA, LocB)] Load(Obj1, 747, LocA)
LocB) [Fly(747, LocA, LocB)]
At(Obj1, ?loc) At(747, ?loc) At(Obj1,
At(Obj1, LocB)
LocA) At(747,
At(747, LocB)
LocA)
At(Obj1,
At(747, ?loc)
?loc) Unload(Obj1,
At(Obj1, 747,
LocB)
At(747, LocA) LocB)
At(Obj1,
At(747, ?loc)
?loc) Inside(Obj1, 747) At(747, LocB)

18
 STRIPS and the FedEx World
11. Stack State 12. Stack State
At(Obj1, LocB) At(Obj2, LocB) At(Obj1, LocA) At(Obj1, LocB) At(Obj2, LocB) At(Obj1, LocA)
At(Obj2, LocB) At(Obj2, LocA) At(Obj2, LocB) At(Obj2, LocA)
At(747, LocB)
LocB) LocB)
At(747, LocB)
Unload(Obj1, 747, LocB) Unload(Obj1, 747, LocB)
Inside(Obj1, 747) At(747, LocB) Inside(Obj1, 747) At(747, LocB)
Load(Obj1, 747, LocA) [Fly(747, LocA, LocB)] Load(Obj1, 747, LocA) [Fly(747, LocA, LocB)]
At(Obj1, LocA) At(747, LocA) At(Obj1, LocA) At(747, LocA)
Fly(747, ?loc, LocA) Fly(747, LocB, LocA)
At(747, ?loc)
13. Stack State
At(Obj1, LocB) At(Obj2, LocB) At(Obj2, LocA)
At(Obj2, LocB) At(747, LocA)
Inside(Obj1, 747)
Unload(Obj1, 747, LocB)
Inside(Obj1, 747) At(747, LocB) [Fly(747, LocA, LocB);
Fly(747, LocB, LocA);
Load(Obj1, 747, LocA)]
19
 STRIPS and the FedEx World
14. Stack State 15. Stack State
At(Obj1, LocB) At(Obj2, LocB) At(Obj2,
At(Obj2, LocA)
LocA) At(Obj1, LocB) At(Obj2, LocB) At(Obj2, LocA)
At(Obj2, LocB) At(747,
At(747, LocA)
LocA) At(Obj2, LocB) At(747, LocA)
LocA)
Inside(Obj1,
Inside(Obj1, 747) Inside(Obj1, 747)
Unload(Obj1, 747, LocB) Unload(Obj1, 747, LocB)
Inside(Obj1, 747) [Fly(747, LocA, LocB); Inside(Obj1, 747) [Fly(747, LocA, LocB);
At(747, LocB) Fly(747, LocB, LocA); At(747, LocB) Fly(747, LocB, LocA);
Load(Obj1, 747, LocA)] Load(Obj1, 747, LocA)]
At(747, LocB) Fly(747, ?loc, LocB)
Inside(Obj1, 747) At(747, ?loc)

16. Stack State

At(Obj1, LocB) At(Obj2, LocB) At(Obj2, LocA)
At(Obj2, LocB) At(747, LocB)
Inside(Obj1, 747)
Unload(Obj1, 747, LocB)
Inside(Obj1, 747) [Fly(747, LocA, LocB);
At(747, LocB) Fly(747, LocB, LocA);
Load(Obj1, 747, LocA);
Fly(747, LocA, LocB)]
20
 The “Sussman Anomaly”
1. Stack State 2. Stack State
On(A, B) On(B, C) Clear(B) On(A, B) On(B, C) Clear(B)
Clear(C) Clear(C)
On(A, B) On(B, C)
On(C, A) On(C, A)
On(B, C) On(A, Table) Put_Block(A, B) On(A, Table)
On(B, Table) On(B, Table)
Holding(A) Clear(B)
Handempty Handempty
A Holding(A)
B C Clear(B)
C A B
Goal Initial State

21
 The “Sussman Anomaly”
3. Stack State 4. Stack State
On(A, B) On(B, C) Clear(B) On(A, B) On(B, C) Clear(B)
Clear(C) Clear(C)
On(B, C) On(C, A) On(B, C) On(C, A)
Put_Block(A, B) On(A, Table) Put_Block(A, B) On(A, Table)
On(B, Table) On(B, Table)
Holding(A) Clear(B) Holding(A) Clear(B)
Handempty Handempty
Holding(A) Holding(A)
Pick_Table(A) Pick_Table(A)
Handempty Clear(A) On(A, Table) Handempty Clear(A) On(A, Table)
Clear(A) Pick_Block(C, A)
Handempty Handempty Clear(C) On(C, A)
On(A, Table)

22
 The “Sussman Anomaly”
5. Stack State 6. Stack State
On(A, B) On(B, C) Clear(B) On(A, B) On(B, C) Clear(B)
Clear(C) Clear(C)
On(B, C) On(B, C)
On(A, Table) On(A, Table)
Put_Block(A, B) On(B, Table) Put_Block(A, B) On(B, Table)
Holding(C) Holding(C)
Holding(A) Clear(B) Holding(A) Clear(B)
Holding(A) Holding(A)
Pick_Table(A) Pick_Table(A)
Handempty Clear(A) On(A, Table) Handempty Clear(A) On(A, Table)
Put_Table(C)
[Pick(C,A)] Holding(C)

[Pick(C,A)]

23
 The “Sussman Anomaly”
7. Stack State 8. Stack State
On(A, B) On(B, C) Clear(C) On(A, B) On(B, C) On(C, Table)
On(B, Table) Clear(B)
On(B, C) On(C, Table) [Pick(C,A); PutT(C); Clear(A)
PickT(A); Put(A, B);
[Pick(C,A); PutT(C); Clear(A) On(A, Table)
Pick(A, B); PutT(A);
PickT(A); Put(A, B)] On(A, B) On(B, C)
Handempty PickT(B); Put(B, C)] Handempty
A
B
B C
A C
9. Stack State 10. Stack State
On(A, B) On(B, C) On(C, Table) On(A, B) On(B, C) On(C, Table)
Clear(B) Clear(A)
On(A, B)
Clear(A) [Pick(C,A); PutT(C); On(B, C)
On(B, C) On(A, Table) PickT(A); Put(A, B); On(A, B)
[Pick(C,A); PutT(C); On(B, C) Pick(A, B); PutT(A); Handempty
PickT(A); Put(A, B); Handempty PickT(B); Put(B, C);
Pick(A, B); PutT(A); PickT(A); Put(A, B)]
PickT(B); Put(B, C)]
24
 Unsolvable Problems
• Result of linearity and poor irreversible actions
Load(o, p, loc) Unload(o, p, loc)
Pre: At(o, loc), At(p, loc) Pre: Inside(o, p), At(p, loc)
Add: Inside(o, p) Add: At(o, loc)
Delete: At(o, loc) Delete: Inside(o, p)
Fly(p, from, to)
Pre: At(p, from), Have-Fuel(p)
Add: At(p, to)
Delete: At(p, from), Have-Fuel(p)

• Initial State: At(Obj1, LocA), At(Obj2, LocA), At(747, LocA),

Have-Fuel(747)
• Goals: At(Obj1, LocB), At(Obj2, LocB)

25
 STRIPS and the UPS World
• I: Try Achieving Goal At(Obj1, LocB) First
– [Load(Obj1, LocA, 747); Fly(747, LocA, LocB);
Unload(Obj1, 747, LocB)]
– But, now cannot achieve Fly(747, LocB, LocA), since no
fuel!

• II: Try Achieving Goal At(Obj2, LocB) First

– [Load(Obj2, LocA, 747); Fly(747, LocA, LocB);
Unload(Obj2, 747, LocB)]
– But, now cannot achieve Fly(747, LocB, LocA), since no
fuel!

• Either way, the problem is unsolvable by STRIPS

26
 Non-Linear Planning
• Basic Idea
– Use goal set instead of goal stack
– Include in the search space all possible subgoal orderings
• Handles goal interactions by interleaving
• Advantages
– Non-linear planning is sound
– Non-linear planning is complete
– Non-linear planning may be optimal with respect to plan
length (depending on search strategy employed)
• Disadvantages
– Larger search space, since all possible goal orderings may
have to be considered
– Somewhat more complex algorithm; More bookkeeping
27
 Non-Linear Planning Algorithm
• NLP (initial-state, goals)
– state = initial-state; plan = []; goalset = goals; opstack = []
– Repeat until goalset is empty
• Choose a goal g from the goalset
• If g does not match state, then
– Choose an operator o whose add-list matches goal g
– Push o on the opstack
– Add the preconditions of o to the goalset
• While all preconditions of operator on top of opstack
are met in state
– Pop operator o from top of opstack
– state = apply(o, state)
– plan = [plan; o]

28
 Non-Linear FedEx World
1. Goals State Ops Plan
At(Obj1, LocB) At(Obj1, LocA) []
At(Obj2, LocB) At(Obj2, LocA)
At(747, LocA)

2. At(Obj1, LocB) At(Obj1, LocA) Unload(Obj2, 747, LocB) []

Inside(Obj2,
Inside(Obj2, 747) At(Obj2, LocA)
At(747, LocB) At(747, LocA)

3. At(Obj1, LocB) At(Obj1, LocA) Load(Obj2, 747, ?loc1) []

At(747, LocB) At(Obj2, LocA) Unload(Obj2, 747, LocB)
At(747, ?loc1) At(747, LocA)
At(Obj1, ?loc1)

4. At(Obj1, LocB) At(Obj1, LocA) Unload(Obj2, 747, LocB) [Load(Obj2, 747, LocA)]
At(747, LocB) At(747, LocA)
Inside(Obj2, 747)

29
 Non-Linear FedEx World
5. Goals State Ops Plan
At(Obj1, LocB) At(Obj1, LocA) Unload(Obj2, 747, LocB) [Load(Obj2, 747, LocA)]
At(747, LocB) At(747, LocA)
Inside(Obj2, 747)

6. At(747, LocB) At(Obj1, LocA) Unload(Obj1, 747, LocB) [Load(Obj2, 747, LocA)]
Inside(Obj1,
Inside(Obj1, 747) At(747, LocA) Unload(Obj2, 747, LocB)
Inside(Obj2, 747)

7. At(747, LocB) At(Obj1, LocA) Load(Obj1, 747, ?loc2) [Load(Obj2, 747, LocA)]
At(747, ?loc2) At(747, LocA) Unload(Obj1, 747, LocB)
At(Obj1, ?loc2) Inside(Obj2, 747) Unload(Obj2, 747, LocB)

8. At(747, LocB) At(747, LocA) Unload(Obj1, 747, LocB) [Load(Obj2, 747, LocA);
Inside(Obj2, 747) Unload(Obj2, 747, LocB) Load(Obj1, 747, LocA)]
Inside(Obj1, 747)

30
 Non-Linear FedEx World
9. Goals State Ops Plan
At(747, LocA) At(747, LocA) Fly(747, LocA, LocB) [Load(Obj2, 747, LocA);
Inside(Obj2, 747) Unload(Obj1, 747, LocB) Load(Obj1, 747, LocA)]
Inside(Obj1, 747) Unload(Obj2, 747, LocB)

10. At(747, LocB) Unload(Obj1, 747, LocB) [Load(Obj2, 747, LocA);

Inside(Obj2, 747) Unload(Obj2, 747, LocB) Load(Obj1, 747, LocA);
Inside(Obj1, 747) Fly(747, LocA, LocB)]

11. At(747, LocB) [Load(Obj2, 747, LocA);

At(Obj2, LocB) Load(Obj1, 747, LocA);
At(Obj1, LocB) Fly(747, LocA, LocB);
Unload(Obj1, 747, LocB);
Unload(Obj2, 747, LocB)]

31