1 INTRODUCTION

1
2 INTELLIGENT AGENTS

function TABLE -D RIVEN -AGENT( percept ) returns an action

persistent: percepts , a sequence, initially empty
table, a table of actions, indexed by percept sequences, initially fully specified

append percept to the end of percepts

action ← L OOKUP( percepts , table)
return action

Figure 2.3 The TABLE -D RIVEN -AGENT program is invoked for each new percept and returns an
action each time. It retains the complete percept sequence in memory.

function R EFLEX -VACUUM -AGENT( [location,status ]) returns an action

if status = Dirty then return Suck
else if location = A then return Right
else if location = B then return Left

Figure 2.4 The agent program for a simple reflex agent in the two-state vacuum environment. This
program implements the agent function tabulated in Figure ??.

function S IMPLE -R EFLEX -AGENT( percept ) returns an action

persistent: rules, a set of condition–action rules

state ← I NTERPRET-I NPUT( percept )

rule ← RULE -M ATCH(state, rules)
action ← rule.ACTION
return action

Figure 2.6 A simple reflex agent. It acts according to a rule whose condition matches the current
state, as defined by the percept.

2
 3

function M ODEL -BASED -R EFLEX -AGENT( percept ) returns an action

persistent: state, the agent’s current conception of the world state
model , a description of how the next state depends on current state and action
rules, a set of condition–action rules
action, the most recent action, initially none

state ← U PDATE -S TATE(state , action, percept , model)

rule ← RULE -M ATCH(state, rules)
action ← rule.ACTION
return action

Figure 2.8 A model-based reflex agent. It keeps track of the current state of the world, using an
internal model. It then chooses an action in the same way as the reflex agent.
3 SOLVING PROBLEMS BY
SEARCHING

function S IMPLE -P ROBLEM -S OLVING -AGENT( percept ) returns an action

persistent: seq, an action sequence, initially empty
state, some description of the current world state
goal , a goal, initially null
problem, a problem formulation

state ← U PDATE -S TATE(state , percept )

if seq is empty then
goal ← F ORMULATE -G OAL(state )
problem ← F ORMULATE -P ROBLEM(state, goal )
seq ← S EARCH( problem)
if seq = failure then return a null action
action ← F IRST(seq )
seq ← R EST(seq)
return action

Figure 3.1 A simple problem-solving agent. It first formulates a goal and a problem, searches for a
sequence of actions that would solve the problem, and then executes the actions one at a time. When
this is complete, it formulates another goal and starts over.

4
 5

function T REE -S EARCH( problem) returns a solution, or failure

initialize the frontier using the initial state of problem
loop do
if the frontier is empty then return failure
choose a leaf node and remove it from the frontier
if the node contains a goal state then return the corresponding solution
expand the chosen node, adding the resulting nodes to the frontier

function G RAPH -S EARCH( problem) returns a solution, or failure

initialize the frontier using the initial state of problem
initialize the explored set to be empty
loop do
if the frontier is empty then return failure
choose a leaf node and remove it from the frontier
if the node contains a goal state then return the corresponding solution
add the node to the explored set
expand the chosen node, adding the resulting nodes to the frontier
only if not in the frontier or explored set

Figure 3.7 An informal description of the general tree-search and graph-search algorithms. The
parts of G RAPH -S EARCH marked in bold italic are the additions needed to handle repeated states.

function B READTH -F IRST-S EARCH( problem) returns a solution, or failure

node ← a node with S TATE = problem.I NITIAL -S TATE, PATH -C OST = 0
if problem.G OAL -T EST(node.S TATE) then return S OLUTION(node)
frontier ← a FIFO queue with node as the only element
explored ← an empty set
loop do
if E MPTY ?( frontier ) then return failure
node ← P OP( frontier ) /* chooses the shallowest node in frontier */
add node.S TATE to explored
for each action in problem.ACTIONS(node.S TATE) do
child ← C HILD -N ODE( problem, node, action)
if child .S TATE is not in explored or frontier then
if problem.G OAL -T EST(child .S TATE) then return S OLUTION(child)
frontier ← I NSERT(child, frontier )

Figure 3.11 Breadth-first search on a graph.

6 Chapter 3. Solving Problems by Searching

function U NIFORM -C OST-S EARCH( problem) returns a solution, or failure

node ← a node with S TATE = problem.I NITIAL -S TATE, PATH -C OST = 0
frontier ← a priority queue ordered by PATH -C OST, with node as the only element
explored ← an empty set
loop do
if E MPTY ?( frontier ) then return failure
node ← P OP( frontier ) /* chooses the lowest-cost node in frontier */
if problem.G OAL -T EST(node.S TATE) then return S OLUTION(node)
add node.S TATE to explored
for each action in problem.ACTIONS(node.S TATE) do
child ← C HILD -N ODE( problem, node, action)
if child .S TATE is not in explored or frontier then
frontier ← I NSERT(child, frontier )
else if child.S TATE is in frontier with higher PATH -C OST then
replace that frontier node with child

Figure 3.13 Uniform-cost search on a graph. The algorithm is identical to the general graph search
algorithm in Figure ??, except for the use of a priority queue and the addition of an extra check in case
a shorter path to a frontier state is discovered. The data structure for frontier needs to support efficient
membership testing, so it should combine the capabilities of a priority queue and a hash table.

function D EPTH -L IMITED -S EARCH( problem, limit) returns a solution, or failure/cutoff

return R ECURSIVE -DLS(M AKE -N ODE(problem.I NITIAL -S TATE), problem, limit)

function R ECURSIVE -DLS(node, problem, limit) returns a solution, or failure/cutoff

if problem.G OAL -T EST(node.S TATE) then return S OLUTION(node)
else if limit = 0 then return cutoff
else
cutoff occurred ? ← false
for each action in problem.ACTIONS(node.S TATE) do
child ← C HILD -N ODE( problem, node, action)
result ← R ECURSIVE -DLS(child, problem, limit − 1)
if result = cutoff then cutoff occurred ? ← true
else if result 6= failure then return result
if cutoff occurred ? then return cutoff else return failure

Figure 3.16 A recursive implementation of depth-limited tree search.

 7

function I TERATIVE -D EEPENING -S EARCH( problem) returns a solution, or failure

for depth = 0 to ∞ do
result ← D EPTH -L IMITED -S EARCH( problem, depth )
if result 6= cutoff then return result

Figure 3.17 The iterative deepening search algorithm, which repeatedly applies depth-limited search
with increasing limits. It terminates when a solution is found or if the depth-limited search returns
failure, meaning that no solution exists.

function R ECURSIVE -B EST-F IRST-S EARCH( problem) returns a solution, or failure

return RBFS(problem, M AKE -N ODE(problem.I NITIAL -S TATE), ∞)

function RBFS(problem, node, f limit) returns a solution, or failure and a new f -cost limit
if problem.G OAL -T EST(node.S TATE) then return S OLUTION(node)
successors ← [ ]
for each action in problem.ACTIONS(node.S TATE) do
add C HILD -N ODE( problem, node, action) into successors
if successors is empty then return failure, ∞
for each s in successors do /* update f with value from previous search, if any */
s.f ← max(s.g + s.h, node.f ))
loop do
best ← the lowest f -value node in successors
if best .f > f limit then return failure, best .f
alternative ← the second-lowest f -value among successors
result , best .f ← RBFS(problem, best , min( f limit, alternative))
if result 6= failure then return result

Figure 3.24 The algorithm for recursive best-first search.

 BEYOND CLASSICAL
4 SEARCH

function H ILL -C LIMBING( problem) returns a state that is a local maximum

current ← M AKE -N ODE(problem.I NITIAL -S TATE)

loop do
neighbor ← a highest-valued successor of current
if neighbor.VALUE ≤ current.VALUE then return current .S TATE
current ← neighbor

Figure 4.2 The hill-climbing search algorithm, which is the most basic local search technique. At
each step the current node is replaced by the best neighbor; in this version, that means the neighbor
with the highest VALUE, but if a heuristic cost estimate h is used, we would find the neighbor with the
lowest h.

function S IMULATED -A NNEALING( problem, schedule) returns a solution state

inputs: problem, a problem
schedule, a mapping from time to “temperature”

current ← M AKE -N ODE(problem.I NITIAL -S TATE)

for t = 1 to ∞ do
T ← schedule(t)
if T = 0 then return current
next ← a randomly selected successor of current
∆E ← next .VALUE – current .VALUE
if ∆E > 0 then current ← next
else current ← next only with probability e∆E/T

Figure 4.5 The simulated annealing algorithm, a version of stochastic hill climbing where some
downhill moves are allowed. Downhill moves are accepted readily early in the annealing schedule and
then less often as time goes on. The schedule input determines the value of the temperature T as a
function of time.

8
 9

function G ENETIC -A LGORITHM( population, F ITNESS -F N) returns an individual

inputs: population, a set of individuals
F ITNESS -F N, a function that measures the fitness of an individual

repeat
new population ← empty set
for i = 1 to S IZE( population) do
x ← R ANDOM -S ELECTION( population, F ITNESS -F N)
y ← R ANDOM -S ELECTION( population , F ITNESS -F N)
child ← R EPRODUCE(x , y)
if (small random probability) then child ← M UTATE(child)
add child to new population
population ← new population
until some individual is fit enough, or enough time has elapsed
return the best individual in population, according to F ITNESS -F N

function R EPRODUCE(x , y) returns an individual

inputs: x , y, parent individuals

n ← L ENGTH(x ); c ← random number from 1 to n

return A PPEND(S UBSTRING(x , 1, c), S UBSTRING(y, c + 1, n))

Figure 4.8 A genetic algorithm. The algorithm is the same as the one diagrammed in Figure ??, with
one variation: in this more popular version, each mating of two parents produces only one offspring,
not two.

function A ND -O R -G RAPH -S EARCH(problem) returns a conditional plan, or failure

O R -S EARCH(problem.I NITIAL -S TATE, problem, [ ])

function O R -S EARCH(state , problem, path) returns a conditional plan, or failure

if problem.G OAL -T EST(state) then return the empty plan
if state is on path then return failure
for each action in problem.ACTIONS(state ) do
plan ← A ND -S EARCH(R ESULTS(state, action), problem, [state | path])
if plan 6= failure then return [action | plan]
return failure

function A ND -S EARCH(states , problem, path) returns a conditional plan, or failure

for each si in states do
plan i ← O R -S EARCH(si , problem, path)
if plan i = failure then return failure
return [if s1 then plan 1 else if s2 then plan 2 else . . . if sn−1 then plan n−1 else plan n ]

Figure 4.11 An algorithm for searching AND – OR graphs generated by nondeterministic environ-
ments. It returns a conditional plan that reaches a goal state in all circumstances. (The notation [x | l]
refers to the list formed by adding object x to the front of list l.)
10 Chapter 4. Beyond Classical Search

function O NLINE -DFS-AGENT(s ′ ) returns an action

inputs: s ′ , a percept that identifies the current state
persistent: result , a table indexed by state and action, initially empty
untried , a table that lists, for each state, the actions not yet tried
unbacktracked , a table that lists, for each state, the backtracks not yet tried
s, a, the previous state and action, initially null

if G OAL -T EST(s ′ ) then return stop

if s ′ is a new state (not in untried ) then untried [s ′ ] ← ACTIONS(s ′ )
if s is not null then
result [s, a] ← s ′
add s to the front of unbacktracked [s ′ ]
if untried [s ′ ] is empty then
if unbacktracked [s ′ ] is empty then return stop
else a ← an action b such that result [s ′ , b] = P OP(unbacktracked [s ′ ])
else a ← P OP(untried [s ′ ])
s ← s′
return a

Figure 4.21 An online search agent that uses depth-first exploration. The agent is applicable only in
state spaces in which every action can be “undone” by some other action.

function LRTA*-AGENT(s ′ ) returns an action

inputs: s ′ , a percept that identifies the current state
persistent: result , a table, indexed by state and action, initially empty
H , a table of cost estimates indexed by state, initially empty
s, a, the previous state and action, initially null

if G OAL -T EST(s ′ ) then return stop

if s ′ is a new state (not in H ) then H [s ′ ] ← h(s ′ )
if s is not null
result [s, a] ← s ′
H [s] ← min LRTA*-C OST(s, b, result [s, b], H )
b ∈ ACTIONS (s)
a ← an action b in ACTIONS(s ′ ) that minimizes LRTA*-C OST(s ′ , b, result [s ′ , b], H )
s ← s′
return a

function LRTA*-C OST(s, a, s ′ , H ) returns a cost estimate

if s ′ is undefined then return h(s)
else return c(s, a, s′ ) + H[s′ ]

Figure 4.24 LRTA*-AGENT selects an action according to the values of neighboring states, which
are updated as the agent moves about the state space.
5 ADVERSARIAL SEARCH

function M INIMAX -D ECISION(state) returns an action

return arg maxa ∈ ACTIONS(s) M IN -VALUE(R ESULT(state, a))

function M AX -VALUE(state ) returns a utility value

if T ERMINAL -T EST(state) then return U TILITY(state)
v ← −∞
for each a in ACTIONS(state) do
v ← M AX(v , M IN -VALUE(R ESULT(s, a)))
return v

function M IN -VALUE(state ) returns a utility value

if T ERMINAL -T EST(state) then return U TILITY(state)
v ←∞
for each a in ACTIONS(state) do
v ← M IN(v , M AX -VALUE(R ESULT(s, a)))
return v

Figure 5.3 An algorithm for calculating minimax decisions. It returns the action corresponding
to the best possible move, that is, the move that leads to the outcome with the best utility, under the
assumption that the opponent plays to minimize utility. The functions M AX -VALUE and M IN -VALUE
go through the whole game tree, all the way to the leaves, to determine the backed-up value of a state.
The notation argmaxa ∈ S f (a) computes the element a of set S that has the maximum value of f (a).

11
12 Chapter 5. Adversarial Search

function A LPHA -B ETA -S EARCH(state) returns an action

v ← M AX -VALUE(state, −∞, +∞)
return the action in ACTIONS(state ) with value v

function M AX -VALUE(state , α, β) returns a utility value

if T ERMINAL -T EST(state) then return U TILITY(state)
v ← −∞
for each a in ACTIONS(state) do
v ← M AX(v , M IN -VALUE(R ESULT(s,a), α, β))
if v ≥ β then return v
α ← M AX(α, v )
return v

function M IN -VALUE(state , α, β) returns a utility value

if T ERMINAL -T EST(state) then return U TILITY(state)
v ← +∞
for each a in ACTIONS(state) do
v ← M IN(v , M AX -VALUE(R ESULT(s,a) , α, β))
if v ≤ α then return v
β ← M IN(β, v )
return v

Figure 5.7 The alpha–beta search algorithm. Notice that these routines are the same as the
M INIMAX functions in Figure ??, except for the two lines in each of M IN -VALUE and M AX -VALUE
that maintain α and β (and the bookkeeping to pass these parameters along).
 CONSTRAINT
6 SATISFACTION
PROBLEMS

function AC-3( csp) returns false if an inconsistency is found and true otherwise
inputs: csp, a binary CSP with components (X, D, C)
local variables: queue, a queue of arcs, initially all the arcs in csp

while queue is not empty do

(Xi , Xj ) ← R EMOVE -F IRST(queue)
if R EVISE(csp, Xi , Xj ) then
if size of Di = 0 then return false
for each Xk in Xi .N EIGHBORS - {Xj } do
add (Xk , Xi ) to queue
return true

function R EVISE( csp, Xi , Xj ) returns true iff we revise the domain of Xi

revised ← false
for each x in Di do
if no value y in Dj allows (x ,y) to satisfy the constraint between Xi and Xj then
delete x from Di
revised ← true
return revised

Figure 6.3 The arc-consistency algorithm AC-3. After applying AC-3, either every arc is arc-
consistent, or some variable has an empty domain, indicating that the CSP cannot be solved. The
name “AC-3” was used by the algorithm’s inventor (?) because it’s the third version developed in the
paper.

13
14 Chapter 6. Constraint Satisfaction Problems

function BACKTRACKING-S EARCH(csp) returns a solution, or failure

return BACKTRACK({ }, csp)

function BACKTRACK(assignment, csp) returns a solution, or failure

if assignment is complete then return assignment
var ← S ELECT-U NASSIGNED -VARIABLE(csp)
for each value in O RDER -D OMAIN -VALUES(var , assignment, csp) do
if value is consistent with assignment then
add {var = value} to assignment
inferences ← I NFERENCE(csp, var , value)
if inferences 6= failure then
add inferences to assignment
result ← BACKTRACK(assignment, csp)
if result 6= failure then
return result
remove {var = value} and inferences from assignment
return failure

Figure 6.5 A simple backtracking algorithm for constraint satisfaction problems. The algo-
rithm is modeled on the recursive depth-first search of Chapter ??. By varying the functions
S ELECT-U NASSIGNED -VARIABLE and O RDER -D OMAIN -VALUES, we can implement the general-
purpose heuristics discussed in the text. The function I NFERENCE can optionally be used to impose
arc-, path-, or k-consistency, as desired. If a value choice leads to failure (noticed either by I NFERENCE
or by B ACKTRACK), then value assignments (including those made by I NFERENCE) are removed from
the current assignment and a new value is tried.

function M IN -C ONFLICTS(csp, max steps ) returns a solution or failure

inputs: csp, a constraint satisfaction problem
max steps , the number of steps allowed before giving up

current ← an initial complete assignment for csp

for i = 1 to max steps do
if current is a solution for csp then return current
var ← a randomly chosen conflicted variable from csp.VARIABLES
value ← the value v for var that minimizes C ONFLICTS(var , v , current , csp)
set var = value in current
return failure

Figure 6.8 The M IN -C ONFLICTS algorithm for solving CSPs by local search. The initial state may
be chosen randomly or by a greedy assignment process that chooses a minimal-conflict value for each
variable in turn. The C ONFLICTS function counts the number of constraints violated by a particular
value, given the rest of the current assignment.
 15

function T REE -CSP-S OLVER( csp) returns a solution, or failure

inputs: csp, a CSP with components X, D, C

n ← number of variables in X
assignment ← an empty assignment
root ← any variable in X
X ← T OPOLOGICAL S ORT(X , root)
for j = n down to 2 do
M AKE -A RC -C ONSISTENT(PARENT(Xj ), Xj )
if it cannot be made consistent then return failure
for i = 1 to n do
assignment [Xi ] ← any consistent value from Di
if there is no consistent value then return failure
return assignment

Figure 6.11 The T REE -CSP-S OLVER algorithm for solving tree-structured CSPs. If the CSP has a
solution, we will find it in linear time; if not, we will detect a contradiction.
7 LOGICAL AGENTS

function KB-AGENT( percept ) returns an action

persistent: KB, a knowledge base
t, a counter, initially 0, indicating time

T ELL(KB, M AKE -P ERCEPT-S ENTENCE( percept , t))

action ← A SK(KB , M AKE -ACTION -Q UERY(t))
T ELL(KB, M AKE -ACTION -S ENTENCE(action, t))
t ←t + 1
return action

Figure 7.1 A generic knowledge-based agent. Given a percept, the agent adds the percept to its
knowledge base, asks the knowledge base for the best action, and tells the knowledge base that it has in
fact taken that action.

16
 17

function TT-E NTAILS ?(KB , α) returns true or false

inputs: KB, the knowledge base, a sentence in propositional logic
α, the query, a sentence in propositional logic

symbols ← a list of the proposition symbols in KB and α

return TT-C HECK -A LL(KB, α, symbols, { })

function TT-C HECK -A LL(KB , α, symbols, model ) returns true or false

if E MPTY ?(symbols) then
if PL-T RUE ?(KB, model ) then return PL-T RUE ?(α, model )
else return true // when KB is false, always return true
else do
P ← F IRST(symbols)
rest ← R EST(symbols)
return (TT-C HECK -A LL(KB, α, rest , model ∪ {P = true})
and
TT-C HECK -A LL(KB , α, rest , model ∪ {P = false }))

Figure 7.8 A truth-table enumeration algorithm for deciding propositional entailment. (TT stands
for truth table.) PL-T RUE ? returns true if a sentence holds within a model. The variable model rep-
resents a partial model—an assignment to some of the symbols. The keyword “and” is used here as a
logical operation on its two arguments, returning true or false.

function PL-R ESOLUTION(KB, α) returns true or false

inputs: KB, the knowledge base, a sentence in propositional logic
α, the query, a sentence in propositional logic

clauses ← the set of clauses in the CNF representation of KB ∧ ¬α

new ← { }
loop do
for each pair of clauses Ci , Cj in clauses do
resolvents ← PL-R ESOLVE(Ci , Cj )
if resolvents contains the empty clause then return true
new ← new ∪ resolvents
if new ⊆ clauses then return false
clauses ← clauses ∪ new

Figure 7.9 A simple resolution algorithm for propositional logic. The function PL-R ESOLVE re-
turns the set of all possible clauses obtained by resolving its two inputs.
18 Chapter 7. Logical Agents

function PL-FC-E NTAILS ?(KB , q) returns true or false

inputs: KB, the knowledge base, a set of propositional definite clauses
q, the query, a proposition symbol
count ← a table, where count [c] is the number of symbols in c’s premise
inferred ← a table, where inferred [s] is initially false for all symbols
agenda ← a queue of symbols, initially symbols known to be true in KB

while agenda is not empty do

p ← P OP(agenda)
if p = q then return true
if inferred [p] = false then
inferred [p] ← true
for each clause c in KB where p is in c.P REMISE do
decrement count [c]
if count [c] = 0 then add c.C ONCLUSION to agenda
return false

Figure 7.12 The forward-chaining algorithm for propositional logic. The agenda keeps track of
symbols known to be true but not yet “processed.” The count table keeps track of how many premises
of each implication are as yet unknown. Whenever a new symbol p from the agenda is processed, the
count is reduced by one for each implication in whose premise p appears (easily identified in constant
time with appropriate indexing.) If a count reaches zero, all the premises of the implication are known,
so its conclusion can be added to the agenda. Finally, we need to keep track of which symbols have
been processed; a symbol that is already in the set of inferred symbols need not be added to the agenda
again. This avoids redundant work and prevents loops caused by implications such as P ⇒ Q and
Q ⇒ P.
 19

function DPLL-S ATISFIABLE ?(s) returns true or false

inputs: s, a sentence in propositional logic

clauses ← the set of clauses in the CNF representation of s

symbols ← a list of the proposition symbols in s
return DPLL(clauses, symbols, { })

function DPLL(clauses, symbols, model ) returns true or false

if every clause in clauses is true in model then return true
if some clause in clauses is false in model then return false
P , value ← F IND -P URE -S YMBOL(symbols, clauses, model )
if P is non-null then return DPLL(clauses, symbols – P , model ∪ {P =value})
P , value ← F IND -U NIT-C LAUSE(clauses, model )
if P is non-null then return DPLL(clauses, symbols – P , model ∪ {P =value})
P ← F IRST(symbols); rest ← R EST(symbols)
return DPLL(clauses, rest , model ∪ {P =true}) or
DPLL(clauses, rest , model ∪ {P =false}))

Figure 7.14 The DPLL algorithm for checking satisfiability of a sentence in propositional logic. The
ideas behind F IND -P URE -S YMBOL and F IND -U NIT-C LAUSE are described in the text; each returns a
symbol (or null) and the truth value to assign to that symbol. Like TT-E NTAILS ?, DPLL operates over
partial models.

function WALK SAT(clauses , p, max flips) returns a satisfying model or failure

inputs: clauses, a set of clauses in propositional logic
p, the probability of choosing to do a “random walk” move, typically around 0.5
max flips, number of flips allowed before giving up

model ← a random assignment of true/false to the symbols in clauses

for i = 1 to max flips do
if model satisfies clauses then return model
clause ← a randomly selected clause from clauses that is false in model
with probability p flip the value in model of a randomly selected symbol from clause
else flip whichever symbol in clause maximizes the number of satisfied clauses
return failure

Figure 7.15 The WALK SAT algorithm for checking satisfiability by randomly flipping the values of
variables. Many versions of the algorithm exist.
20 Chapter 7. Logical Agents

function H YBRID -W UMPUS -AGENT( percept ) returns an action

inputs: percept , a list, [stench ,breeze,glitter ,bump,scream]
persistent: KB, a knowledge base, initially the atemporal “wumpus physics”
t, a counter, initially 0, indicating time
plan, an action sequence, initially empty

T ELL(KB, M AKE -P ERCEPT-S ENTENCE( percept , t))

T ELL the KB the temporal “physics” sentences for time t
safe ← {[x , y] : A SK(KB , OK tx,y ) = true}
if A SK(KB , Glitter t ) = true then
plan ← [Grab] + P LAN -ROUTE(current , {[1,1]}, safe) + [Climb]
if plan is empty then
′
unvisited ← {[x , y] : A SK(KB, Ltx,y ) = false for all t′ ≤ t}
plan ← P LAN -ROUTE(current , unvisited ∩ safe, safe)
if plan is empty and A SK(KB, HaveArrow t ) = true then
possible wumpus ← {[x , y] : A SK(KB , ¬ Wx,y ) = false}
plan ← P LAN -S HOT(current , possible wumpus, safe)
if plan is empty then // no choice but to take a risk
not unsafe ← {[x , y] : A SK(KB , ¬ OK tx,y ) = false}
plan ← P LAN -ROUTE(current , unvisited ∩ not unsafe, safe)
if plan is empty then
plan ← P LAN -ROUTE(current , {[1, 1]}, safe) + [Climb]
action ← P OP(plan)
T ELL(KB, M AKE -ACTION -S ENTENCE(action, t))
t ←t + 1
return action

function P LAN -ROUTE(current ,goals,allowed) returns an action sequence

inputs: current , the agent’s current position
goals, a set of squares; try to plan a route to one of them
allowed , a set of squares that can form part of the route

problem ← ROUTE -P ROBLEM(current , goals,allowed )

return A*-G RAPH -S EARCH(problem)

Figure 7.17 A hybrid agent program for the wumpus world. It uses a propositional knowledge base
to infer the state of the world, and a combination of problem-solving search and domain-specific code
to decide what actions to take.
 21

function SAT PLAN( init, transition, goal , T max ) returns solution or failure
inputs: init, transition, goal , constitute a description of the problem
T max , an upper limit for plan length

for t = 0 to T max do
cnf ← T RANSLATE -T O -SAT( init, transition, goal , t)
model ← SAT-S OLVER(cnf )
if model is not null then
return E XTRACT-S OLUTION(model )
return failure

Figure 7.19 The SATP LAN algorithm. The planning problem is translated into a CNF sentence in
which the goal is asserted to hold at a fixed time step t and axioms are included for each time step up to
t. If the satisfiability algorithm finds a model, then a plan is extracted by looking at those proposition
symbols that refer to actions and are assigned true in the model. If no model exists, then the process is
repeated with the goal moved one step later.
8 FIRST-ORDER LOGIC

22
9 INFERENCE IN
FIRST-ORDER LOGIC

function U NIFY(x , y, θ) returns a substitution to make x and y identical

inputs: x , a variable, constant, list, or compound expression
y, a variable, constant, list, or compound expression
θ, the substitution built up so far (optional, defaults to empty)

if θ = failure then return failure

else if x = y then return θ
else if VARIABLE ?(x ) then return U NIFY-VAR(x , y, θ)
else if VARIABLE ?(y) then return U NIFY-VAR(y, x , θ)
else if C OMPOUND ?(x ) and C OMPOUND ?(y) then
return U NIFY(x .A RGS, y.A RGS, U NIFY(x .O P, y.O P, θ))
else if L IST ?(x ) and L IST ?(y) then
return U NIFY(x .R EST, y.R EST, U NIFY(x .F IRST, y.F IRST, θ))
else return failure

function U NIFY-VAR(var , x , θ) returns a substitution

if {var /val } ∈ θ then return U NIFY(val , x , θ)

else if {x/val } ∈ θ then return U NIFY(var , val , θ)
else if O CCUR -C HECK ?(var , x ) then return failure
else return add {var /x } to θ

Figure 9.1 The unification algorithm. The algorithm works by comparing the structures of the in-
puts, element by element. The substitution θ that is the argument to U NIFY is built up along the way and
is used to make sure that later comparisons are consistent with bindings that were established earlier. In
a compound expression such as F (A, B), the O P field picks out the function symbol F and the A RGS
field picks out the argument list (A, B).

23
24 Chapter 9. Inference in First-Order Logic

function FOL-FC-A SK(KB , α) returns a substitution or false

inputs: KB, the knowledge base, a set of first-order definite clauses
α, the query, an atomic sentence
local variables: new , the new sentences inferred on each iteration

repeat until new is empty

new ← { }
for each rule in KB do
( p1 ∧ . . . ∧ pn ⇒ q) ← S TANDARDIZE -VARIABLES(rule)
for each θ such that S UBST(θ, p1 ∧ . . . ∧ pn ) = S UBST(θ, p1′ ∧ . . . ∧ pn′ )
for some p1′ , . . . , pn′ in KB
q ′ ← S UBST(θ, q)
if q ′ does not unify with some sentence already in KB or new then
add q ′ to new
φ ← U NIFY(q ′ , α)
if φ is not fail then return φ
add new to KB
return false

Figure 9.3 A conceptually straightforward, but very inefficient, forward-chaining algorithm. On

each iteration, it adds to KB all the atomic sentences that can be inferred in one step from the impli-
cation sentences and the atomic sentences already in KB . The function S TANDARDIZE -VARIABLES
replaces all variables in its arguments with new ones that have not been used before.

function FOL-BC-A SK(KB , query ) returns a generator of substitutions

return FOL-BC-O R(KB , query , { })

generator FOL-BC-O R(KB, goal , θ) yields a substitution

for each rule (lhs ⇒ rhs) in F ETCH -RULES -F OR -G OAL(KB , goal ) do
(lhs, rhs) ← S TANDARDIZE -VARIABLES((lhs, rhs))
for each θ′ in FOL-BC-A ND(KB, lhs, U NIFY(rhs, goal , θ)) do
yield θ′

generator FOL-BC-A ND(KB, goals, θ) yields a substitution

if θ = failure then return
else if L ENGTH(goals) = 0 then yield θ
else do
first ,rest ← F IRST(goals), R EST(goals)
for each θ′ in FOL-BC-O R(KB, S UBST(θ, first ), θ) do
for each θ′′ in FOL-BC-A ND(KB, rest , θ′ ) do
yield θ′′

Figure 9.6 A simple backward-chaining algorithm for first-order knowledge bases.

 25

procedure A PPEND(ax , y, az , continuation )

trail ← G LOBAL -T RAIL -P OINTER()

if ax = [ ] and U NIFY(y, az ) then C ALL(continuation )
R ESET-T RAIL(trail )
a, x , z ← N EW-VARIABLE(), N EW-VARIABLE(), N EW-VARIABLE()
if U NIFY(ax , [a | x ]) and U NIFY(az , [a | z ]) then A PPEND(x , y, z , continuation )

Figure 9.8 Pseudocode representing the result of compiling the Append predicate. The function
N EW-VARIABLE returns a new variable, distinct from all other variables used so far. The procedure
C ALL(continuation) continues execution with the specified continuation.
10 CLASSICAL PLANNING

Init(At (C1 , SFO) ∧ At (C2 , JFK ) ∧ At(P1 , SFO) ∧ At (P2 , JFK )

∧ Cargo(C1 ) ∧ Cargo(C2 ) ∧ Plane(P1 ) ∧ Plane(P2 )
∧ Airport (JFK ) ∧ Airport(SFO))
Goal (At(C1 , JFK ) ∧ At(C2 , SFO))
Action(Load (c, p, a),
P RECOND : At(c, a) ∧ At (p, a) ∧ Cargo(c) ∧ Plane(p) ∧ Airport(a)
E FFECT: ¬ At (c, a) ∧ In(c, p))
Action(Unload (c, p, a),
P RECOND : In(c, p) ∧ At (p, a) ∧ Cargo(c) ∧ Plane(p) ∧ Airport(a)
E FFECT: At(c, a) ∧ ¬ In(c, p))
Action(Fly(p, from, to),
P RECOND : At(p, from) ∧ Plane(p) ∧ Airport(from) ∧ Airport(to)
E FFECT: ¬ At (p, from) ∧ At(p, to))

Figure 10.1 A PDDL description of an air cargo transportation planning problem.

Init(Tire(Flat) ∧ Tire(Spare) ∧ At(Flat, Axle) ∧ At (Spare, Trunk ))

Goal (At (Spare, Axle))
Action(Remove(obj , loc),
P RECOND : At (obj , loc)
E FFECT: ¬ At(obj , loc) ∧ At (obj , Ground ))
Action(PutOn(t, Axle),
P RECOND : Tire(t) ∧ At(t, Ground ) ∧ ¬ At (Flat, Axle)
E FFECT: ¬ At (t, Ground ) ∧ At (t, Axle))
Action(LeaveOvernight ,
P RECOND :
E FFECT: ¬ At (Spare, Ground ) ∧ ¬ At(Spare, Axle) ∧ ¬ At (Spare, Trunk )
∧ ¬ At(Flat, Ground ) ∧ ¬ At (Flat, Axle) ∧ ¬ At(Flat, Trunk ))

Figure 10.2 The simple spare tire problem.

26
 27

Init(On(A, Table) ∧ On(B, Table) ∧ On(C, A)

∧ Block (A) ∧ Block (B) ∧ Block (C) ∧ Clear (B) ∧ Clear (C))
Goal (On(A, B) ∧ On(B, C))
Action(Move(b, x, y),
P RECOND : On(b, x) ∧ Clear (b) ∧ Clear (y) ∧ Block (b) ∧ Block (y) ∧
(b6=x) ∧ (b6=y) ∧ (x6=y),
E FFECT: On(b, y) ∧ Clear (x) ∧ ¬On(b, x) ∧ ¬Clear (y))
Action(MoveToTable(b, x),
P RECOND : On(b, x) ∧ Clear (b) ∧ Block (b) ∧ (b6=x),
E FFECT: On(b, Table) ∧ Clear (x) ∧ ¬On(b, x))

Figure 10.3 A planning problem in the blocks world: building a three-block tower. One solution is
the sequence [MoveToTable(C, A), Move(B, Table, C), Move(A, Table, B)].

Init(Have(Cake))
Goal (Have(Cake) ∧ Eaten(Cake))
Action(Eat (Cake)
P RECOND : Have(Cake)
E FFECT: ¬ Have(Cake) ∧ Eaten(Cake))
Action(Bake(Cake)
P RECOND : ¬ Have(Cake)
E FFECT: Have(Cake))

Figure 10.7 The “have cake and eat cake too” problem.

function G RAPHPLAN( problem) returns solution or failure

graph ← I NITIAL -P LANNING -G RAPH( problem)
goals ← C ONJUNCTS(problem.G OAL)
nogoods ← an empty hash table
for tl = 0 to ∞ do
if goals all non-mutex in St of graph then
solution ← E XTRACT-S OLUTION(graph, goals, N UM L EVELS(graph), nogoods )
if solution 6= failure then return solution
if graph and nogoods have both leveled off then return failure
graph ← E XPAND -G RAPH(graph, problem)

Figure 10.9 The G RAPHPLAN algorithm. G RAPHPLAN calls E XPAND -G RAPH to add a level until
either a solution is found by E XTRACT-S OLUTION, or no solution is possible.
 PLANNING AND ACTING
11 IN THE REAL WORLD

Jobs({AddEngine1 ≺ AddWheels1 ≺ Inspect1 },

{AddEngine2 ≺ AddWheels2 ≺ Inspect2 })
Resources(EngineHoists(1), WheelStations(1), Inspectors(2), LugNuts(500))
Action(AddEngine1 , D URATION :30,
U SE :EngineHoists(1 ))
Action(AddEngine2 , D URATION :60,
U SE :EngineHoists(1 ))
Action(AddWheels1 , D URATION :30,
C ONSUME :LugNuts(20), U SE :WheelStations(1))
Action(AddWheels2 , D URATION :15,
C ONSUME :LugNuts(20), U SE :WheelStations(1))
Action(Inspect i , D URATION :10,
U SE :Inspectors(1))

Figure 11.1 A job-shop scheduling problem for assembling two cars, with resource constraints. The
notation A ≺ B means that action A must precede action B.

28
 29

Refinement (Go(Home, SFO),

S TEPS: [Drive(Home, SFOLongTermParking ),
Shuttle(SFOLongTermParking , SFO)] )
Refinement (Go(Home, SFO),
S TEPS: [Taxi (Home, SFO)] )

Refinement (Navigate([a, b], [x, y]),

P RECOND : a = x ∧ b = y
S TEPS: [ ] )
Refinement (Navigate([a, b], [x, y]),
P RECOND :Connected ([a, b], [a − 1, b])
S TEPS: [Left , Navigate([a − 1, b], [x, y])] )
Refinement (Navigate([a, b], [x, y]),
P RECOND :Connected ([a, b], [a + 1, b])
S TEPS: [Right , Navigate([a + 1, b], [x, y])] )
...

Figure 11.4 Definitions of possible refinements for two high-level actions: going to San Francisco
airport and navigating in the vacuum world. In the latter case, note the recursive nature of the refine-
ments and the use of preconditions.

function H IERARCHICAL -S EARCH( problem, hierarchy ) returns a solution, or failure

frontier ← a FIFO queue with [Act] as the only element
loop do
if E MPTY ?( frontier ) then return failure
plan ← P OP( frontier ) /* chooses the shallowest plan in frontier */
hla ← the first HLA in plan, or null if none
prefix ,suffix ← the action subsequences before and after hla in plan
outcome ← R ESULT(problem.I NITIAL -S TATE, prefix )
if hla is null then /* so plan is primitive and outcome is its result */
if outcome satisfies problem.G OAL then return plan
else for each sequence in R EFINEMENTS(hla, outcome, hierarchy ) do
frontier ← I NSERT(A PPEND( prefix , sequence , suffix ), frontier )

Figure 11.5 A breadth-first implementation of hierarchical forward planning search. The initial plan
supplied to the algorithm is [Act]. The R EFINEMENTS function returns a set of action sequences, one
for each refinement of the HLA whose preconditions are satisfied by the specified state, outcome.
30 Chapter 11. Planning and Acting in the Real World

function A NGELIC -S EARCH( problem, hierarchy , initialPlan) returns solution or fail

frontier ← a FIFO queue with initialPlan as the only element
loop do
if E MPTY ?( frontier ) then return fail
plan ← P OP( frontier ) /* chooses the shallowest node in frontier */
if R EACH+ (problem.I NITIAL -S TATE, plan) intersects problem.G OAL then
if plan is primitive then return plan /* R EACH+ is exact for primitive plans */
guaranteed ← R EACH− (problem.I NITIAL -S TATE, plan) ∩ problem.G OAL
if guaranteed 6={ } and M AKING -P ROGRESS(plan, initialPlan) then
finalState ← any element of guaranteed
return D ECOMPOSE(hierarchy , problem.I NITIAL -S TATE, plan, finalState)
hla ← some HLA in plan
prefix ,suffix ← the action subsequences before and after hla in plan
for each sequence in R EFINEMENTS(hla, outcome, hierarchy ) do
frontier ← I NSERT(A PPEND( prefix , sequence , suffix ), frontier )

function D ECOMPOSE(hierarchy , s0 , plan, sf ) returns a solution

solution ← an empty plan
while plan is not empty do
action ← R EMOVE -L AST(plan)
si ← a state in R EACH− (s0 , plan) such that sf ∈R EACH− (si , action)
problem ← a problem with I NITIAL -S TATE = si and G OAL = sf
solution ← A PPEND(A NGELIC -S EARCH(problem, hierarchy , action), solution)
sf ← si
return solution

Figure 11.8 A hierarchical planning algorithm that uses angelic semantics to identify and com-
mit to high-level plans that work while avoiding high-level plans that don’t. The predicate
M AKING -P ROGRESS checks to make sure that we aren’t stuck in an infinite regression of refinements.
At top level, call A NGELIC -S EARCH with [Act ] as the initialPlan.

Actors(A, B)
Init(At (A, LeftBaseline) ∧ At(B, RightNet ) ∧
Approaching (Ball, RightBaseline)) ∧ Partner (A, B) ∧ Partner (B, A)
Goal (Returned (Ball) ∧ (At(a, RightNet ) ∨ At (a, LeftNet ))
Action(Hit(actor , Ball),
P RECOND :Approaching (Ball, loc) ∧ At(actor , loc)
E FFECT:Returned (Ball))
Action(Go(actor , to),
P RECOND :At (actor , loc) ∧ to 6= loc,
E FFECT:At (actor , to) ∧ ¬ At (actor , loc))

Figure 11.10 The doubles tennis problem. Two actors A and B are playing together and can be in
one of four locations: LeftBaseline, RightBaseline, LeftNet , and RightNet . The ball can be returned
only if a player is in the right place. Note that each action must include the actor as an argument.
 KNOWLEDGE
12 REPRESENTATION

31
13 QUANTIFYING
UNCERTAINTY

function DT-AGENT( percept ) returns an action

persistent: belief state , probabilistic beliefs about the current state of the world
action, the agent’s action

update belief state based on action and percept

calculate outcome probabilities for actions,
given action descriptions and current belief state
select action with highest expected utility
given probabilities of outcomes and utility information
return action

Figure 13.1 A decision-theoretic agent that selects rational actions.

32
 PROBABILISTIC
14 REASONING

function E NUMERATION -A SK(X , e, bn) returns a distribution over X

inputs: X , the query variable
e, observed values for variables E
bn, a Bayes net with variables {X} ∪ E ∪ Y /* Y = hidden variables */

Q(X ) ← a distribution over X , initially empty

for each value xi of X do
Q(xi ) ← E NUMERATE -A LL(bn.VARS, exi )
where exi is e extended with X = xi
return N ORMALIZE(Q(X))

function E NUMERATE -A LL(vars, e) returns a real number

if E MPTY ?(vars) then return 1.0
Y ← F IRST(vars)
if Y has value y in e
then returnPP (y | parents(Y )) × E NUMERATE -A LL(R EST(vars), e)
else return y P (y | parents(Y )) × E NUMERATE -A LL(R EST(vars), ey )
where ey is e extended with Y = y

Figure 14.9 The enumeration algorithm for answering queries on Bayesian networks.

function E LIMINATION -A SK(X , e, bn) returns a distribution over X

inputs: X , the query variable
e, observed values for variables E
bn, a Bayesian network specifying joint distribution P(X1 , . . . , Xn )

factors ← [ ]
for each var in O RDER(bn.VARS) do
factors ← [M AKE -FACTOR(var , e)|factors ]
if var is a hidden variable then factors ← S UM -O UT(var , factors )
return N ORMALIZE(P OINTWISE -P RODUCT(factors ))

Figure 14.10 The variable elimination algorithm for inference in Bayesian networks.

33
34 Chapter 14. Probabilistic Reasoning

function P RIOR -S AMPLE(bn) returns an event sampled from the prior specified by bn
inputs: bn, a Bayesian network specifying joint distribution P(X1 , . . . , Xn )

x ← an event with n elements

foreach variable Xi in X1 , . . . , Xn do
x[i] ← a random sample from P(Xi | parents(Xi ))
return x

Figure 14.12 A sampling algorithm that generates events from a Bayesian network. Each variable
is sampled according to the conditional distribution given the values already sampled for the variable’s
parents.

function R EJECTION -S AMPLING(X , e, bn, N ) returns an estimate of P(X|e)

inputs: X , the query variable
e, observed values for variables E
bn, a Bayesian network
N , the total number of samples to be generated
local variables: N, a vector of counts for each value of X , initially zero

for j = 1 to N do
x ← P RIOR -S AMPLE(bn)
if x is consistent with e then
N[x ] ← N[x ]+1 where x is the value of X in x
return N ORMALIZE(N)

Figure 14.13 The rejection-sampling algorithm for answering queries given evidence in a Bayesian
network.
 35

function L IKELIHOOD-W EIGHTING(X , e, bn, N ) returns an estimate of P(X|e)

inputs: X , the query variable
e, observed values for variables E
bn, a Bayesian network specifying joint distribution P(X1 , . . . , Xn )
N , the total number of samples to be generated
local variables: W, a vector of weighted counts for each value of X , initially zero

for j = 1 to N do
x, w ← W EIGHTED -S AMPLE(bn, e)
W[x ] ← W[x ] + w where x is the value of X in x
return N ORMALIZE(W)

function W EIGHTED -S AMPLE(bn, e) returns an event and a weight

w ← 1; x ← an event with n elements initialized from e
foreach variable Xi in X1 , . . . , Xn do
if Xi is an evidence variable with value xi in e
then w ← w × P (Xi = xi | parents(Xi ))
else x[i] ← a random sample from P(Xi | parents(Xi ))
return x, w

Figure 14.14 The likelihood-weighting algorithm for inference in Bayesian networks. In

W EIGHTED -S AMPLE, each nonevidence variable is sampled according to the conditional distribution
given the values already sampled for the variable’s parents, while a weight is accumulated based on the
likelihood for each evidence variable.

function G IBBS -A SK(X , e, bn, N ) returns an estimate of P(X|e)

local variables: N, a vector of counts for each value of X , initially zero
Z, the nonevidence variables in bn
x, the current state of the network, initially copied from e

initialize x with random values for the variables in Z

for j = 1 to N do
for each Zi in Z do
set the value of Zi in x by sampling from P(Zi |mb(Zi ))
N[x ] ← N[x ] + 1 where x is the value of X in x
return N ORMALIZE(N)

Figure 14.15 The Gibbs sampling algorithm for approximate inference in Bayesian networks; this
version cycles through the variables, but choosing variables at random also works.
 PROBABILISTIC
15 REASONING OVER TIME

function F ORWARD -BACKWARD(ev, prior ) returns a vector of probability distributions

inputs: ev, a vector of evidence values for steps 1, . . . , t
prior , the prior distribution on the initial state, P(X0 )
local variables: fv, a vector of forward messages for steps 0, . . . , t
b, a representation of the backward message, initially all 1s
sv, a vector of smoothed estimates for steps 1, . . . , t

fv[0] ← prior
for i = 1 to t do
fv[i] ← F ORWARD(fv[i − 1], ev[i])
for i = t downto 1 do
sv[i] ← N ORMALIZE(fv[i] × b)
b ← BACKWARD(b, ev[i])
return sv

Figure 15.4 The forward–backward algorithm for smoothing: computing posterior probabilities of
a sequence of states given a sequence of observations. The F ORWARD and BACKWARD operators are
defined by Equations (??) and (??), respectively.

36
 37

function F IXED -L AG -S MOOTHING(et , hmm, d ) returns a distribution over Xt−d

inputs: et , the current evidence for time step t
hmm, a hidden Markov model with S × S transition matrix T
d , the length of the lag for smoothing
persistent: t, the current time, initially 1
f, the forward message P(Xt |e1:t ), initially hmm.P RIOR
B, the d-step backward transformation matrix, initially the identity matrix
et−d:t , double-ended list of evidence from t − d to t, initially empty
local variables: Ot−d , Ot , diagonal matrices containing the sensor model information

add et to the end of et−d:t

Ot ← diagonal matrix containing P(et |Xt )
if t > d then
f ← F ORWARD(f, et )
remove et−d−1 from the beginning of et−d:t
Ot−d ← diagonal matrix containing P(et−d |Xt−d )
B ← O−1t−d T
−1
BTOt
else B ← BTOt
t ←t + 1
if t > d then return N ORMALIZE(f × B1) else return null

Figure 15.6 An algorithm for smoothing with a fixed time lag of d steps, implemented as an online
algorithm that outputs the new smoothed estimate given the observation for a new time step. Notice
that the final output N ORMALIZE(f × B1) is just α f × b, by Equation (??).

function PARTICLE -F ILTERING(e, N , dbn) returns a set of samples for the next time step
inputs: e, the new incoming evidence
N , the number of samples to be maintained
dbn, a DBN with prior P(X0 ), transition model P(X1 |X0 ), sensor model P(E1 |X1 )
persistent: S , a vector of samples of size N , initially generated from P(X0 )
local variables: W , a vector of weights of size N

for i = 1 to N do
S [i] ← sample from P(X1 | X0 = S [i]) /* step 1 */
W [i] ← P(e | X1 = S[i]) /* step 2 */
S ← W EIGHTED -S AMPLE -W ITH -R EPLACEMENT(N , S , W ) /* step 3 */
return S

Figure 15.17 The particle filtering algorithm implemented as a recursive update operation with state
(the set of samples). Each of the sampling operations involves sampling the relevant slice variables
in topological order, much as in P RIOR -S AMPLE. The W EIGHTED -S AMPLE -W ITH -R EPLACEMENT
operation can be implemented to run in O(N ) expected time. The step numbers refer to the description
in the text.
16 MAKING SIMPLE
DECISIONS

function I NFORMATION -G ATHERING -AGENT( percept ) returns an action

persistent: D, a decision network

integrate percept into D

j ← the value that maximizes VPI (Ej ) / Cost(Ej )
if VPI (Ej ) > Cost(Ej )
return R EQUEST(Ej )
else return the best action from D

Figure 16.9 Design of a simple information-gathering agent. The agent works by repeatedly select-
ing the observation with the highest information value, until the cost of the next observation is greater
than its expected benefit.

38
17 MAKING COMPLEX
DECISIONS

function VALUE -I TERATION(mdp, ǫ) returns a utility function

inputs: mdp, an MDP with states S , actions A(s), transition model P (s′ | s, a),
rewards R(s), discount γ
ǫ, the maximum error allowed in the utility of any state
local variables: U , U ′ , vectors of utilities for states in S , initially zero
δ, the maximum change in the utility of any state in an iteration

repeat
U ← U ′; δ ← 0
for each state s in S do X
U ′ [s] ← R(s) + γ max P (s′ | s, a) U [s′ ]
a ∈ A(s)
s′
if |U [s] − U [s]| > δ then δ ← |U ′ [s] − U [s]|
′

until δ < ǫ(1 − γ)/γ

return U

Figure 17.4 The value iteration algorithm for calculating utilities of states. The termination condi-
tion is from Equation (??).

39
40 Chapter 17. Making Complex Decisions

function P OLICY-I TERATION(mdp) returns a policy

inputs: mdp, an MDP with states S , actions A(s), transition model P (s′ | s, a)
local variables: U , a vector of utilities for states in S , initially zero
π, a policy vector indexed by state, initially random

repeat
U ← P OLICY-E VALUATION(π, U , mdp)
unchanged ? ← true
for each state sX
in S do
P (s′ | s, a) U [s′ ] > P (s′ | s, π[s]) U [s′ ] then do
X
if max
a ∈ A(s)
s′ s′
P (s′ | s, a) U [s′ ]
X
π[s] ← argmax
a ∈ A(s)
s′
unchanged ? ← false
until unchanged ?
return π

Figure 17.7 The policy iteration algorithm for calculating an optimal policy.

function POMDP-VALUE -I TERATION(pomdp, ǫ) returns a utility function

inputs: pomdp, a POMDP with states S , actions A(s), transition model P (s′ | s, a),
sensor model P (e | s), rewards R(s), discount γ
ǫ, the maximum error allowed in the utility of any state
local variables: U , U ′ , sets of plans p with associated utility vectors αp

U ′ ← a set containing just the empty plan [ ], with α[ ] (s) = R(s)

repeat
U ←U′
U ′ ← the set of all plans consisting of an action and, for each possible next percept,
a plan in U with utility vectors computed according to Equation (??)
U ′ ← R EMOVE -D OMINATED -P LANS(U ′ )
until M AX -D IFFERENCE(U , U ′ ) < ǫ(1 − γ)/γ
return U

Figure 17.9 A high-level sketch of the value iteration algorithm for POMDPs. The
R EMOVE -D OMINATED -P LANS step and M AX -D IFFERENCE test are typically implemented as linear
programs.
18 LEARNING FROM
EXAMPLES

function D ECISION -T REE -L EARNING(examples, attributes , parent examples) returns a

tree

if examples is empty then return P LURALITY-VALUE(parent examples )

else if all examples have the same classification then return the classification
else if attributes is empty then return P LURALITY-VALUE(examples)
else
A ← argmaxa ∈ attributes I MPORTANCE(a, examples)
tree ← a new decision tree with root test A
for each value vk of A do
exs ← {e : e ∈ examples and e.A = vk }
subtree ← D ECISION -T REE -L EARNING(exs, attributes − A, examples)
add a branch to tree with label (A = vk ) and subtree subtree
return tree

Figure 18.4 The decision-tree learning algorithm. The function I MPORTANCE is described in Sec-
tion ??. The function P LURALITY-VALUE selects the most common output value among a set of
examples, breaking ties randomly.

41
42 Chapter 18. Learning from Examples

function C ROSS -VALIDATION -W RAPPER(Learner , k , examples ) returns a hypothesis

local variables: errT , an array, indexed by size, storing training-set error rates
errV , an array, indexed by size, storing validation-set error rates
for size = 1 to ∞ do
errT [size], errV [size] ← C ROSS -VALIDATION(Learner , size, k , examples)
if errT has converged then do
best size ← the value of size with minimum errV [size]
return Learner (best size, examples)

function C ROSS -VALIDATION(Learner , size, k , examples) returns two values:

average training set error rate, average validation set error rate

fold errT ← 0; fold errV ← 0

for fold = 1 to k do
training set , validation set ← PARTITION(examples, fold, k )
h ← Learner (size, training set )
fold errT ← fold errT + E RROR -R ATE(h, training set )
fold errV ← fold errV +E RROR -R ATE(h, validation set )
return fold errT /k , fold errV /k

Figure 18.7 An algorithm to select the model that has the lowest error rate on validation data by
building models of increasing complexity, and choosing the one with best empirical error rate on val-
idation data. Here errT means error rate on the training data, and errV means error rate on the
validation data. Learner (size, examples) returns a hypothesis whose complexity is set by the parame-
ter size, and which is trained on the examples . PARTITION(examples, fold, k) splits examples into two
subsets: a validation set of size N/k and a training set with all the other examples. The split is different
for each value of fold.

function D ECISION -L IST-L EARNING(examples) returns a decision list, or failure

if examples is empty then return the trivial decision list No

t ← a test that matches a nonempty subset examples t of examples
such that the members of examples t are all positive or all negative
if there is no such t then return failure
if the examples in examples t are positive then o ← Yes else o ← No
return a decision list with initial test t and outcome o and remaining tests given by
D ECISION -L IST-L EARNING(examples − examples t )

Figure 18.10 An algorithm for learning decision lists.

 43

function BACK -P ROP -L EARNING(examples, network ) returns a neural network

inputs: examples, a set of examples, each with input vector x and output vector y
network , a multilayer network with L layers, weights wi,j , activation function g
local variables: ∆, a vector of errors, indexed by network node

repeat
for each weight wi,j in network do
wi,j ← a small random number
for each example (x, y) in examples do
/* Propagate the inputs forward to compute the outputs */
for each node i in the input layer do
ai ← xi
for ℓ = 2 to L do
P j in layer ℓ do
for each node
in j ← i wi,j ai
aj ← g(in j )
/* Propagate deltas backward from output layer to input layer */
for each node j in the output layer do
∆[j] ← g ′ (in j ) × (yj − aj )
for ℓ = L − 1 to 1 do
for each node i in layer
P ℓ do
∆[i] ← g ′ (in i ) j wi,j ∆[j]
/* Update every weight in network using deltas */
for each weight wi,j in network do
wi,j ← wi,j + α × ai × ∆[j]
until some stopping criterion is satisfied
return network

Figure 18.23 The back-propagation algorithm for learning in multilayer networks.

44 Chapter 18. Learning from Examples

function A DA B OOST(examples, L, K ) returns a weighted-majority hypothesis

inputs: examples, set of N labeled examples (x1 , y1 ), . . . , (xN , yN )
L, a learning algorithm
K , the number of hypotheses in the ensemble
local variables: w, a vector of N example weights, initially 1/N
h, a vector of K hypotheses
z, a vector of K hypothesis weights

for k = 1 to K do
h[k ] ← L(examples, w)
error ← 0
for j = 1 to N do
if h[k ](xj ) 6= yj then error ← error + w[j]
for j = 1 to N do
if h[k ](xj ) = yj then w[j] ← w[j] · error /(1 − error )
w ← N ORMALIZE(w)
z[k ] ← log (1 − error )/error
return W EIGHTED -M AJORITY(h, z)

Figure 18.33 The A DA B OOST variant of the boosting method for ensemble learning. The al-
gorithm generates hypotheses by successively reweighting the training examples. The function
W EIGHTED -M AJORITY generates a hypothesis that returns the output value with the highest vote from
the hypotheses in h, with votes weighted by z.
19 KNOWLEDGE IN
LEARNING

function C URRENT-B EST-L EARNING(examples, h) returns a hypothesis or fail

if examples is empty then

return h
e ← F IRST(examples )
if e is consistent with h then
return C URRENT-B EST-L EARNING(R EST(examples), h)
else if e is a false positive for h then
for each h ′ in specializations of h consistent with examples seen so far do
h ′′ ← C URRENT-B EST-L EARNING(R EST(examples), h ′ )
if h ′′ 6= fail then return h ′′
else if e is a false negative for h then
for each h ′ in generalizations of h consistent with examples seen so far do
h ′′ ← C URRENT-B EST-L EARNING(R EST(examples), h ′ )
if h ′′ 6= fail then return h ′′
return fail

Figure 19.2 The current-best-hypothesis learning algorithm. It searches for a consistent hypothesis
that fits all the examples and backtracks when no consistent specialization/generalization can be found.
To start the algorithm, any hypothesis can be passed in; it will be specialized or gneralized as needed.

45
46 Chapter 19. Knowledge in Learning

function V ERSION -S PACE -L EARNING(examples) returns a version space

local variables: V , the version space: the set of all hypotheses

V ← the set of all hypotheses

for each example e in examples do
if V is not empty then V ← V ERSION -S PACE -U PDATE(V , e)
return V

function V ERSION -S PACE -U PDATE(V , e) returns an updated version space

V ← {h ∈ V : h is consistent with e}

Figure 19.3 The version space learning algorithm. It finds a subset of V that is consistent with all
the examples.

function M INIMAL -C ONSISTENT-D ET(E , A) returns a set of attributes

inputs: E , a set of examples
A, a set of attributes, of size n

for i = 0 to n do
for each subset Ai of A of size i do
if C ONSISTENT-D ET ?(Ai , E ) then return Ai

function C ONSISTENT-D ET ?(A, E ) returns a truth value

inputs: A, a set of attributes
E , a set of examples
local variables: H , a hash table

for each example e in E do

if some example in H has the same values as e for the attributes A
but a different classification then return false
store the class of e in H , indexed by the values for attributes A of the example e
return true

Figure 19.8 An algorithm for finding a minimal consistent determination.

 47

function F OIL(examples, target ) returns a set of Horn clauses

inputs: examples, set of examples
target , a literal for the goal predicate
local variables: clauses , set of clauses, initially empty

while examples contains positive examples do

clause ← N EW-C LAUSE(examples, target )
remove positive examples covered by clause from examples
add clause to clauses
return clauses

function N EW-C LAUSE(examples, target ) returns a Horn clause

local variables: clause, a clause with target as head and an empty body
l , a literal to be added to the clause
extended examples , a set of examples with values for new variables

extended examples ← examples

while extended examples contains negative examples do
l ← C HOOSE -L ITERAL(N EW-L ITERALS(clause), extended examples )
append l to the body of clause
extended examples ← set of examples created by applying E XTEND -E XAMPLE
to each example in extended examples
return clause

function E XTEND -E XAMPLE(example, literal) returns a set of examples

if example satisfies literal
then return the set of examples created by extending example with
each possible constant value for each new variable in literal
else return the empty set

Figure 19.12 Sketch of the F OIL algorithm for learning sets of first-order Horn clauses from exam-
ples. N EW-L ITERALS and C HOOSE -L ITERAL are explained in the text.
20 LEARNING
PROBABILISTIC MODELS

48
 REINFORCEMENT
21 LEARNING

function PASSIVE -ADP-AGENT(percept ) returns an action

inputs: percept , a percept indicating the current state s ′ and reward signal r ′
persistent: π, a fixed policy
mdp, an MDP with model P , rewards R, discount γ
U , a table of utilities, initially empty
Nsa , a table of frequencies for state–action pairs, initially zero
Ns ′ |sa , a table of outcome frequencies given state–action pairs, initially zero
s, a, the previous state and action, initially null

if s ′ is new then U [s ′ ] ← r ′ ; R[s ′ ] ← r ′

if s is not null then
increment Nsa [s, a] and Ns ′ |sa [s ′ , s, a]
for each t such that Ns ′ |sa [t, s, a] is nonzero do
P (t | s, a) ← Ns ′ |sa [t, s, a] / Nsa [s, a]
U ← P OLICY-E VALUATION(π, U , mdp)
if s ′ .T ERMINAL ? then s, a ← null else s, a ← s ′ , π[s ′ ]
return a

Figure 21.2 A passive reinforcement learning agent based on adaptive dynamic programming. The
P OLICY-E VALUATION function solves the fixed-policy Bellman equations, as described on page ??.

49
50 Chapter 21. Reinforcement Learning

function PASSIVE -TD-AGENT(percept ) returns an action

inputs: percept , a percept indicating the current state s ′ and reward signal r ′
persistent: π, a fixed policy
U , a table of utilities, initially empty
Ns , a table of frequencies for states, initially zero
s, a, r , the previous state, action, and reward, initially null

if s ′ is new then U [s ′ ] ← r ′
if s is not null then
increment N s [s]
U [s] ← U [s] + α(Ns [s])(r + γ U [s ′ ] − U [s])
if s .T ERMINAL ? then s, a, r ← null else s, a, r ← s ′ , π[s ′ ], r ′
′

return a

Figure 21.4 A passive reinforcement learning agent that learns utility estimates using temporal dif-
ferences. The step-size function α(n) is chosen to ensure convergence, as described in the text.

function Q-L EARNING -AGENT(percept ) returns an action

inputs: percept , a percept indicating the current state s ′ and reward signal r ′
persistent: Q, a table of action values indexed by state and action, initially zero
Nsa , a table of frequencies for state–action pairs, initially zero
s, a, r , the previous state, action, and reward, initially null

if T ERMINAL ?(s) then Q[s, None] ← r ′

if s is not null then
increment Nsa [s, a]
Q[s, a] ← Q[s, a] + α(Nsa [s, a])(r + γ maxa′ Q[s′ , a ′ ] − Q[s, a])
s, a, r ← s ′ , argmaxa′ f (Q[s ′ , a ′ ], Nsa [s ′ , a′ ]), r ′
return a

Figure 21.8 An exploratory Q-learning agent. It is an active learner that learns the value Q(s, a) of
each action in each situation. It uses the same exploration function f as the exploratory ADP agent,
but avoids having to learn the transition model because the Q-value of a state can be related directly to
those of its neighbors.
 NATURAL LANGUAGE
22 PROCESSING

function HITS(query ) returns pages with hub and authority numbers

pages ← E XPAND -PAGES(R ELEVANT-PAGES(query ))

for each p in pages do
p.AUTHORITY ← 1
p.H UB ← 1
repeat until convergence do
for each p in pages do
P
P ← i I NLINKi (p).H UB
p.AUTHORITY
p.H UB ← i O UTLINKi (p).AUTHORITY
N ORMALIZE(pages )
return pages

Figure 22.1 The HITS algorithm for computing hubs and authorities with respect to a query.
R ELEVANT-PAGES fetches the pages that match the query, and E XPAND -PAGES adds in every page
that links to or is linked from one of the relevant pages. N ORMALIZE divides each page’s score by the
sum of the squares of all pages’ scores (separately for both the authority and hubs scores).

51
23 NATURAL LANGUAGE
FOR COMMUNICATION

function CYK-PARSE(words , grammar ) returns P , a table of probabilities

N ← L ENGTH(words)
M ← the number of nonterminal symbols in grammar
P ← an array of size [M , N , N ], initially all 0
/* Insert lexical rules for each word */
for i = 1 to N do
for each rule of form (X → words i [p]) do
P [X , i, 1] ← p
/* Combine first and second parts of right-hand sides of rules, from short to long */
for length = 2 to N do
for start = 1 to N − length + 1 do
for len1 = 1 to N − 1 do
len2 ← length − len1
for each rule of the form (X → Y Z [p]) do
P [X , start , length] ← M AX(P [X , start , length],
P [Y , start , len1 ] × P [Z , start + len1 , len2 ] × p)
return P

Figure 23.4 The CYK algorithm for parsing. Given a sequence of words, it finds the most probable
derivation for the whole sequence and for each subsequence. It returns the whole table, P , in which
an entry P [X , start, len] is the probability of the most probable X of length len starting at position
start . If there is no X of that size at that location, the probability is 0.

52
 53

[ [S [NP-SBJ-2 Her eyes]

[VP were
[VP glazed
[NP *-2]
[SBAR-ADV as if
[S [NP-SBJ she]
[VP did n’t
[VP [VP hear [NP *-1]]
or
[VP [ADVP even] see [NP *-1]]
[NP-1 him]]]]]]]]
.]

Figure 23.5 Annotated tree for the sentence “Her eyes were glazed as if she didn’t hear or even
see him.” from the Penn Treebank. Note that in this grammar there is a distinction between an object
noun phrase (NP) and a subject noun phrase (NP-SBJ). Note also a grammatical phenomenon we have
not covered yet: the movement of a phrase from one part of the tree to another. This tree analyzes
the phrase “hear or even see him” as consisting of two constituent VP s, [VP hear [NP *-1]] and [VP
[ADVP even] see [NP *-1]], both of which have a missing object, denoted *-1, which refers to the NP
labeled elsewhere in the tree as [NP-1 him].
24 PERCEPTION

54
25 ROBOTICS

function M ONTE -C ARLO -L OCALIZATION(a, z , N , P (X ′ |X, v, ω), P (z|z ∗ ), m) returns

a set of samples for the next time step
inputs: a, robot velocities v and ω
z, range scan z1 , . . . , zM
P (X ′ |X, v, ω), motion model
P (z|z ∗ ), range sensor noise model
m, 2D map of the environment
persistent: S, a vector of samples of size N
local variables: W , a vector of weights of size N
S ′ , a temporary vector of particles of size N
W ′ , a vector of weights of size N

if S is empty then /* initialization phase */

for i = 1 to N do
S[i] ← sample from P (X0 )
for i = 1 to N do /* update cycle */
S ′ [i] ← sample from P (X ′ |X = S[i], v, ω)
W ′ [i] ← 1
for j = 1 to M do
z ∗ ← R AY C AST(j, X = S ′ [i], m)
W ′ [i] ← W ′ [i] · P (zj | z ∗ )
S ← W EIGHTED -S AMPLE -W ITH -R EPLACEMENT(N ,S ′ ,W ′ )
return S

Figure 25.9 A Monte Carlo localization algorithm using a range-scan sensor model with indepen-
dent noise.

55
26 PHILOSOPHICAL
FOUNDATIONS

56
27 AI: THE PRESENT AND
FUTURE

57
28 MATHEMATICAL
BACKGROUND

58
29 NOTES ON LANGUAGES
AND ALGORITHMS

generator P OWERS -O F -2() yields ints

i ←1
while true do
yield i
i ←2 × i

for p in P OWERS -O F -2() do

P RINT(p)

Figure 29.1 Example of a generator function and its invocation within a loop.

59