UNIT-2

3.Solving Problems by Searching

3.1 Problem-Solving Agents
3.2 Example Problems
3.3 Searching for Solutions
3.4 Uninformed Search Strategies
3.5 Informed (Heuristic) Search Strategies
3.6 Heuristic Functions

4. Beyond Classical Search:

4.1 Local Search Algorithms & Optimizarion Problems

4.2 Search in Continues Spaces

4.3 Searching with Deterministic Actions

4.4 Searching with partial Observations

4.5 Online Search and Unknown environments

3.1 PROBLEM SOLVING AGENTS:

Intelligent agents are supposed to maximize their performance measure. Imagine an agent in the city of
Arad, Romania, enjoying a touring holiday. The agent’sperformance measure contains many factors: it
wants to improve its suntan, improve its Romanian,take in the sights, enjoy the nightlife (such as it is),
avoid hangovers, and so on. The decision problem is a complex one involving many tradeoffs and careful
reading. And destination is Bucharest.
In that case, it makes sense for the agent to adopt the goal of getting to Bucharest.Courses of action.
Goal formulation, based on the current situation and the agent’s performance measure, is the first step in
problem solving.
Problem formulation is the process of deciding what actions and states to consider, given a goal.
 function SIMPLE-PROBLEM-SOLVING-AGENT( percept) returns an action
persistent: seq, an action sequence, initially empty
state, some description of the current world state
goal , a goal, initially null
problem, a problem formulation
state ← UPDATE-STATE(state, percept )
if seq is empty then
goal ← FORMULATE-GOAL(state)
problem ← FORMULATE-PROBLEM(state, goal )
seq ← SEARCH( problem)
if seq = failure then return a null action
action ← FIRST(seq)
seq ← REST(seq)
return action

Figure 3.1 A simple problem-solving agent. It first formulates a goal and a problem, searches
for a sequence of actions that would solve the problem, and then executes the actions one at
a time. When this is complete, it formulates another goal and starts over.

3.1.1 Well-defined problems and solutions

A problem can be defined formally by five components:
● initial state that the agent starts in. For example, the initial state for our agent in
Romania might be described as In(Arad).
● Actions:A description of the possible actions available to the agent. Given a particular state
s,ACTIONS(s) returns the set of actions that can be executed in s. We say that each of these actions
is applicable in s.
For example, from the state In(Arad), the applicable actions are {Go(Sibiu),Go(Timisoara),Go(Zerind)}.
● Transition model
• The goal test, which determines whether a given state is a goal state. Sometimes there
is an explicit set of possible goal states, and the test simply checks whether the given
state is one of them. The agent’s goal in Romania is the singleton set {In(Bucharest )}.
• A path cost function that assigns a numeric cost to each path. The problem-solving
agent chooses a cost function that reflects its own performance measure
3.1.2 Formulating problems
In the preceding section we proposed a formulation of the problem of getting to Bucharest in
terms of the initial state, actions, transition model, goal test, and path cost. This formulation
seems reasonable, but it is still a model—an abstract mathematical description—and not the real thing.
where the state of the world includes so many things: the traveling companions,
the current radio program, the scenery out of the window, the proximity of law enforcement
officers, the distance to the next rest stop, the condition of the road, the weather, and so on.
All these considerations are left out of our state descriptions because they are irrelevant to the
problem of finding a route to Bucharest. The process of removing detail from a representation
is called abstraction.

3.2 EXAMPLE PROBLEMS

The problem-solving approach has been applied to a vast array of task environments. We
list some of the best known here, distinguishing between toy and real-world problems. A toy problem is
intended to illustrate or exercise various problem-solving methods.
A real-world problem is one whose solutions people actually care about. Such problems tend not to
have a single agreed-upon description, but we can give the general flavor of their formulations.
3.2.1 Toy problems

⮚ The first example we examine is the vacuum world first introduced in Chapter 2. (See
Figure 2.2.) This can be formulated as a problem as follows:
• States: The state is determined by both the agent location and the dirt locations. The
agent is in one of two locations, each of which might or might not contain dirt. Thus,
there are 2 × 22 = 8 possible world states. A larger environment with n locations has
n ・ 2n states.
• Initial state: Any state can be designated as the initial state.
• Actions: In this simple environment, each state has just three actions: Left, Right, and
Suck. Larger environments might also include Up and Down.
• Transition model: The actions have their expected effects, except that moving Left in
the leftmost square, moving Right in the rightmost square, and Sucking in a clean square
have no effect. The complete state space is shown in Figure 3.3.
• Goal test: This checks whether all the squares are clean.
• Path cost: Each step costs 1, so the path cost is the number of steps in the path.

⮚ The 8-puzzle, an instance of which is shown in Figure 3.4, consists of a 3×3 board with
eight numbered tiles and a blank space. A tile adjacent to the blank space can slide into the
space. The object is to reach a specified goal state, such as the one shown on the right of the
figure. The standard formulation is as follows:
• States: A state description specifies the location of each of the eight tiles and the blank
in one of the nine squares.
• Initial state: Any state can be designated as the initial state. Note that any given goal
can be reached from exactly half of the possible initial states (Exercise 3.5).
• Actions: The simplest formulation defines the actions as movements of the blank space
Left, Right, Up, or Down. Different subsets of these are possible depending on where
the blank is.
• Transition model: Given a state and action, this returns the resulting state; for example,
if we apply Left to the start state in Figure 3.4, the resulting state has the 5 and the blank
switched.
• Goal test: This checks whether the state matches the goal configuration shown in Figure
3.4. (Other goal configurations are possible.)
• Path cost: Each step costs 1, so the path cost is the number of steps in the path.

⮚ The goal of the 8-queens problem is to place eight queens on a chessboard such that
no queen attacks any other. (A queen attacks any piece in the same row, column or diagonal.)
Figure 3.5 shows an attempted solution that fails: the queen in the rightmost column is
attacked by the queen at the top left.

• States: Any arrangement of 0 to 8 queens on the board is a state.

• Initial state: No queens on the board.
• Actions: Add a queen to any empty square.
• Transition model: Returns the board with a queen added to the specified square.
• Goal test: 8 queens are on the board, none attacked.
.
3.2.2 Real-world problems

1.The route-finding problem is defined in terms of specified locations and transitions along links
between them. Route-finding algorithms are used in a variety of applications. Others, such as routing
video streams in computer networks, military operations planning, and airline travel-planning systems,
involve much more complex specifications.
The airline travel problems that must be solved by a travel-planning Web site:
• States: Each state obviously includes a location (e.g., an airport) and the current time.
Furthermore, because the cost of an action (a flight segment) may depend on previous
segments, their fare bases, and their status as domestic or international, the state must
record extra information about these “historical” aspects.
• Initial state: This is specified by the user’s query.
• Actions: Take any flight from the current location, in any seat class, leaving after the
current time, leaving enough time for within-airport transfer if needed.
• Transition model: The state resulting from taking a flight will have the flight’s destination
as the current location and the flight’s arrival time as the current time.
• Goal test: Are we at the final destination specified by the user?
• Path cost: This depends on monetary cost, waiting time, flight time, customs and immigration
procedures, seat quality, time of day, type of airplane, frequent-flyer mileage awards, and so on.
However, that not all air travel goes according to plan. A really good system should include contingency
plans—such as backup reservations on alternate flights— to the extent that these are justified by the cost
and likelihood of failure of the original plan.
2. Touring problems are closely related to route-finding problems, for example, the problem “Visit every
city in Figure 3.2 at least once, starting and ending in Bucharest.” The state space, however, is quite
different. Each state must include not just the current location but also the set of cities the agent has
visited.
3.The traveling salesperson problem (TSP) is a touring problem in which each city must be visited
exactly once. The aim is to find the shortest tour.
4.A VLSI layout problem requires positioning millions of components and connections on a chip to
minimize area, minimize circuit delays, minimize stray capacitances, and maximize manufacturing yield.
The layout problem comes after the logical design phase and is usually split into two parts: cell layout
and channel routing. The aim is to place the cells on the chip so that they do not overlap and so that there
is room for the connecting wires to be placed between the cells. These search problems are extremely
complex, but definitely worth solving.
5.Robot navigation is a generalization of the route-finding problem described earlier.
Rather than following a discrete set of routes, a robot can move in a continuous space with
(in principle) an infinite set of possible actions and states. When the robot has arms and legs or
wheels that must also be controlled, the search space becomes many-dimensional.
6.Automatic assembly sequencing: In assembly
problems, the aim is to find an order in which to assemble the parts of some object. If the
wrong order is chosen, there will be no way to add some part later in the sequence without undoing some
of the work already done. Checking a step in the sequence for feasibility is a
difficult geometrical search problem closely related to robot navigation. Thus, the generation
of legal actions is the expensive part of assembly sequencing.

3.3 SEARCH SOLUTIONS:

Having formulated some problems, we now need to solve them. A solution is an action
sequence, so search algorithms work by considering various possible action sequences. The
possible action sequences starting at the initial state form a search tree with the initial state
at the root; the branches are actions and the nodes correspond to states in the state space of
the problem. Figure 3.6 shows the first few steps in growing the search tree for finding a route
from Arad to Bucharest. The root node of the tree corresponds to the initial state, In(Arad).
The first step is to test whether this is a goal state. (Clearly it is not, but it is important to
check so that we can solve trick problems like “starting in Arad, get to Arad.”) Then we
need to consider taking various actions. We do this by expanding the current state; that is,
applying each legal action to the current state, thereby generating a new set of states. In
this case, we add three branches from the parent node In(Arad) leading to three new child
nodes: In(Sibiu), In(Timisoara), and In(Zerind). Now we must choose which of these three
possibilities to consider further.
Suppose we choose Sibiu first.We check to see whether it is a goal state (it is not) and then expand it to
get In(Arad), In(Fagaras), In(Oradea), and In(RimnicuVilcea). We can then choose any of these four or
go back and choose Timisoara or Zerind.
3.3.1 Infrastructure for search algorithms
Search algorithms require a data structure to keep track of the search tree that is being constructed.
For each node n of the tree, we have a structure that contains four components:
• n.STATE: the state in the state space to which the node corresponds;
• n.PARENT: the node in the search tree that generated this node;
• n.ACTION: the action that was applied to the parent to generate the node;
• n.PATH-COST: the cost, traditionally denoted by g(n), of the path from the initial state
to the node, as indicated by the parent pointers.
.
Now that we have nodes, we need somewhere to put them. The frontier needs to be
stored in such a way that the search algorithm can easily choose the next node to expand
QUEUE according to its preferred strategy. The appropriate data structure for this is a queue. The

operations on a queue are as follows:

• EMPTY?(queue) returns true only if there are no more elements in the queue.
• POP(queue) removes the first element of the queue and returns it.
• INSERT(element , queue) inserts an element and returns the resulting queue.

3.3.2 Measuring problem-solving performance

Before we get into the design of specific search algorithms, we need to consider the criteria
that might be used to choose among them. We can evaluate an algorithm’s performance in
four ways:
• Completeness: Is the algorithm guaranteed to find a solution when there is one?
• Optimality: Does the strategy find the optimal solution, as defined on page 68?
• Time complexity: How long does it take to find a solution?
• Space complexity: How much memory is needed to perform the search?

3.4 UNIFORMED SEARCH STRATEGIES:

Uninformedsearch (also called blind search). The term means that the strategies have no additional
information about states beyond that provided in the problem definition.

3.4.1 Breadth-first search

Breadth-first search is a simple strategy in which the root node is expanded first, then all the
successors of the root node are expanded next, then their successors, and so on. In general,
all the nodes are expanded at a given depth in the search tree before any nodes at the next
level are expanded.
Breadth-first search is an instance of the general graph-search algorithm (Figure 3.7) in
which the shallowest unexpanded node is chosen for expansion. This is achieved very simply
by using a FIFO queue for the frontier. Thus, new nodes (which are always deeper than their
parents) go to the back of the queue, and old nodes, which are shallower than the new nodes,
get expanded first.
Now suppose that the solution is at depth d. In the worst case, it is the last node generated at that level.
Then the total number of nodes generated is
b + b2 +b3 +…..+b4+(b^d+1 – b) = O(b^d+1)

3.4.2 Uniform-cost search

By a simple extension, we can find an algorithm that is optimal
with any step-cost function. Instead of expanding the shallowest node,
uniform-cost search expands the node n with the lowest path cost g(n)

3.4.3 Depth-first search

Depth-first search always expands the deepest node in the current frontier of the search tree.
The progress of the search is illustrated in Figure 3.16. The search proceeds immediately
to the deepest level of the search tree, where the nodes have no successors. As those nodes
are expanded, they are dropped from the frontier, so then the search “backs up” to the next
deepest node that still has unexplored successors.

whereas breadth-first-search uses a FIFO queue, depth-first search uses a LIFO queue.
A LIFO queue means that the most recently generated node is chosen for expansion. This
must be the deepest unexpanded node because it is one deeper than its parent—which, in turn,
was the deepest unexpanded node when it was selected.
3.4.4 Depth-limited search
The embarrassing failure of depth-first search in infinite state spaces can be alleviated by
supplying depth-first search with a predetermined depth limit ℓ. That is, nodes at depth ℓ are
treated as if they have no successors. This approach is called depth-limited search.
The depth limit solves the infinite-path problem. Unfortunately, it also introduces an additional
source of incompleteness if we choose ℓ < d, that is, the shallowest goal is beyond the depth
limit. (This is likely when d is unknown.) Depth-limited search will also be nonoptimal if
we choose ℓ > d. Its time complexity is O(bℓ) and its space complexity is O(bℓ). Depth-first
search can be viewed as a special case of depth-limited search with ℓ=∞.

3.4.5 Iterative deepening depth-first search

Iterative deepening search (or iterative deepening depth-first search) is a general strategy,
often used in combination with depth-first tree search, that finds the best depth limit. It does
this by gradually increasing the limit—first 0, then 1, then 2, and so on—until a goal is found.
This will occur when the depth limit reaches d, the depth of the shallowest goal node. The
algorithm is shown in Figure 3.18. Iterative deepening combines the benefits of depth-first
and breadth-first search.
3.4.6 Bidirectional search
The idea behind bidirectional search is to run two simultaneous searches—one forward from
the initial state and the other backward from the goal—hoping that the two searches meet in
the middle (Figure 3.20). The motivation is that bd/2 + bd/2 is much less than bd, or in the
figure, the area of the two small circles is less than the area of one big circle centered on the
start and reaching to the goal.
Bidirectional search requires a method for computing predecessors. When all the actions in the state space
are reversible, the predecessors of x are just its successors. Other cases may require substantial ingenuity.
==============================================================

3.4.7 Comparing uninformed search strategies

3.5 INFORMED [HEURISTIC] SEARCH STRATEGIES:

The general approach we consider is called best-first search. Best-first search is an

instance of the general TREE-SEARCH or GRAPH-SEARCH algorithm in which a node is
selected for expansion based on an evaluation function, f(n). The evaluation function is
construed as a cost estimate, so the node with the lowest evaluation is expanded first.

h(n) = estimated cost of the cheapest path from the state at node n to a goal state.

3.5.1 Greedy best-first search

Greedy best-first search8 tries to expand the node that is closest to the goal, on the grounds
that this is likely to lead to a solution quickly. Thus, it evaluates nodes by using just the heuristic function;
that is, f(n) = h(n).
Let us see how this works for route-finding problems in Romania; we use the straightline
distance heuristic, which we will call hSLD. If the goal is Bucharest, we need to know the straight-line
distances to Bucharest, which are shown in Figure 3.22.
3.5.2 A* search: Minimizing the total estimated solution cost
The most widely known form of best-first search is called A∗ search (pronounced “A-star search”). It
evaluates nodes by combining g(n), the cost to reach the node, and h(n), the cost
to get from the node to the goal:f(n) = g(n) + h(n) .
Since g(n) gives the path cost from the start node to node n, and h(n) is the estimated cost
of the cheapest path from n to the goal, we have
f(n) = estimated cost of the cheapest solution through n .
Thus, if we are trying to find the cheapest solution, a reasonable thing to try first is the
node with the lowest value of g(n) + h(n). It turns out that this strategy is more than just
reasonable: provided that the heuristic function h(n) satisfies certain conditions, A∗ search is
both complete and optimal. The algorithm is identical to UNIFORM-COST-SEARCH except
that A∗ uses g + h instead of g.
A heuristic function, is a function that calculates an approximate cost to a problem (or
ranks alternatives). For example the problem might be finding the shortest driving distance to a
point. A heuristic cost would be the straight line distance to the point.
A* search:
3.5.3 Memory-bounded heuristic search

The simplest way to reduce memory requirements for A∗ is to adapt the idea of iterative
deepening to the heuristic search context, resulting in the iterative-deepening A∗ (IDA∗) algorithm.
Memory-bounded algorithms are RBFS and MA∗.
Recursive best-first search (RBFS) is a simple recursive algorithm that attempts to
mimic the operation of standard best-first search, but us only linear space
 Two algorithms that do this are MA∗ (memory-bounded A∗) and SMA∗ (simplified MA∗). SMA∗ MA* is—
well—simpler, so we will describe it. SMA∗ proceeds just like A∗, expanding the best leaf until memory
is full.
At this point, it cannot add a new node to the search tree without dropping an old one. SMA∗
always drops the worst leaf node—the one with the highest f-value. Like RBFS, SMA∗
then backs up the value of the forgotten node to its parent. In this way, the ancestor of a
forgotten subtree knows the quality of the best path in that subtree.

3.6 Heuristic functions:

■ 8-puzzle

■ The average solution cost for a randomly generated 8-puzzle instance is about 22 step
 ■ The branching factor is about 3

■ Exhaustive search to depth 22:

■ 3.1 x 1010 states.

■ A good heuristic function can reduce the search process.

■ Two commonly used heuristics

■ h1 = the number of misplaced tiles

■ h1(s)=8

■ h2 = the sum of the distances of the tiles from their goal positions (Manhattan
distance).

h2(s)=3+1+2+2+2+3+3+2=18

Admissible Heuristics:
■ A heuristic h(n) is admissible if for every node n,

h(n) ≤ h*(n),

where h*(n) is the true cost to reach the goal state from n.

■ An admissible heuristic never overestimates the cost to reach the goal, i.e., it is optimistic (or
at least, never pessimistic)

■ Example: hSLD(n) (never overestimates actual road distance)

■ Theorem:

If h(n) is admissible, A* using TREE-SEARCH is optimal

Dominance:

■ IF h2(n) ≥ h1(n) for all n (both admissible)

THEN h2 dominates h1

■ h2 is always better for search than h1

■ h2 guarantees to expand no more nodes than does h1

■ h2 almost always expands fewer nodes than does h1

■ Typical 8-puzzle search costs

 (average number of nodes expanded):

■ d=12 IDS = 3,644,035 nodes

A*(h1) = 227 nodes
A*(h2) = 73 nodes

■ d=24 IDS = too many nodes

A*(h1) = 39,135 nodes
A*(h2) = 1,641 nodes

-------------------------------------------------------------------------------------------------------------------------------

4. BEYOND CLASSICAL SEARCH:

4.1 Local search and optimization:

Local search:

• Use single current state and move to neighboring states.

Idea: start with an initial guess at a solution and

∙ incrementally improve it until it is one

Advantages:

∙ • Use very little memory

• Find often reasonable solutions in large or infinite state spaces.

Useful for pure optimization problems

.∙ • Find or approximate best state according to some objective function

• Optimal if the space to be searched

4.1.1 Hill Climbing:

o Hill climbing algorithm is a local search algorithm which continuously moves in the
direction of increasing elevation/value to find the peak of the mountain or best
solution to the problem. It terminates when it reaches a peak value where no
neighbor has a higher value.
o Hill climbing algorithm is a technique which is used for optimizing the mathematical
problems. One of the widely discussed examples of Hill climbing algorithm is
 Traveling-salesman Problem in which we need to minimize the distance traveled by
the salesman.
o It is also called greedy local search as it only looks to its good immediate neighbor
state and not beyond that.
o A node of hill climbing algorithm has two components which are state and value.
o Hill Climbing is mostly used when a good heuristic is available.
o In this algorithm, we don't need to maintain and handle the search tree or graph as it
only keeps a single current state.

o State-space Diagram for Hill Climbing:

o The state-space landscape is a graphical representation of the hill-climbing
algorithm which is showing a graph between various states of algorithm and
Objective function/Cost.
o On Y-axis we have taken the function which can be an objective function or cost
function, and state-space on the x-axis. If the function on Y-axis is cost then, the goal
of search is to find the global minimum and local minimum. If the function of Y-axis
is Objective function, then the goal of the search is to find the global maximum and
local maximum.

o
.

1. Local Maximum: A local maximum is a peak state in the landscape which is better than
each of its neighboring states, but there is another state also present which is higher than
the local maximum.
2. Plateau: A plateau is the flat area of the search space in which all the neighbor states of the
current state contains the same value, because of this algorithm does not find any best direction to
move. A hill-climbing search might be lost in the plateau area.

3. Ridges: A ridge is a special form of the local maximum. It has an area which is higher than its
surrounding areas, but itself has a slope, and cannot be reached in a single move.

4.1.2 Simulated Annealing:

A hill-climbing algorithm which never makes a move towards a lower value guaranteed to
be incomplete because it can get stuck on a local maximum. And if algorithm applies a
random walk, by moving a successor, then it may complete but not efficient. Simulated
Annealing is an algorithm which yields both efficiency and completeness.
In mechanical term Annealing is a process of hardening a metal or glass to a high
temperature then cooling gradually, so this allows the metal to reach a low-energy
crystalline state. The same process is used in simulated annealing in which the algorithm
picks a random move, instead of picking the best move. If the random move improves the
state, then it follows the same path. Otherwise, the algorithm follows the path which has a
probability of less than 1 or it moves downhill and chooses another path.

4.1.3 Local Beam Search

In this algorithm, it holds k number of states at any given time. At the start, these
states are generated randomly. The successors of these k states are computed with the
help of objective function. If any of these successors is the maximum value of the
objective function, then the algorithm stops.

4.1.4 Genetic algorithms

1.Start with k random states (the initial population)
2.New states are generated by either

1.“Mutation” of a single state or

2.“Sexual Reproduction”: (combining) two parent states (selected proportionally to their fitness)

Encoding used for the “genome” of an individual strongly affects the behavior of the search

Similar (in some ways) to stochastic beam search

Representation: Strings of genes

Each chromosome

•represents a possible solution

•made up of a string of genes

Each gene encodes some property of the solution

There is a fitness metric on phenotypes of chromosomes

•Evaluation of how well a solution with that set of properties solves the problem.

New generations are formed by

•Crossover: sexual reproduction

•Mutation: asexual reproduction

Encoding of a Chromosome

The chromosome encodes characteristics of the solution which it represents, often as a string of
binary digits.

Chromosome 1 1101100100110110

Chromosome 2 1101111000011110

Each set of bits represents some dimension of the solution.

4.2 Local Search in Continuous Spaces

● When the objective function is continuous. Then the graph of the function
creates a continuous subspace (like a surface) of the landscape.
● Gradient search: if the function is a scalar field of 3 variables, f(x, y, z) in R3,
then the gradient of this function:

● gives us the direction where the scalar field grows the most.

● The search moves from s=(x, y, z) to .

● A plateau is defined by
 4.3 Searching with nondeterministic actions:

In non-deterministic environments,the result of an action can vary.

Future percepts can specify which outcome has occurred.

Generalizing the transition function

𝑅𝐸𝑆𝑈𝐿𝑇 : 𝑆 × 𝐴 → 2𝑆 instead of 𝑅𝐸𝑆𝑈𝐿𝑇𝑆 : 𝑆 × 𝐴 →𝑆

search tree will be an and-or-tree.

Solution will be a sub-tree containing a contingency plan(nestedif-then-else statements)

4.3.1 The Erratic vaccum world:

Erratic vaccum world:

1. States
{1,2,…,8}

2. Actions
1. {Left,Right,Suck}
3. Goal
{7} or {8}

4. Non-deterministic:
1. When sucking a dirty square,it
cleans it and sometimes
cleans up
dirt in an adjacent square.

2. When sucking a clean

square,it sometimes
deposits dirt on the
carpet.
 Solution for AND-OR search problem is a sub-tree that:

specifies one action at each OR node

includes every outcome at each AND node

has a goal node at every leaf

Algorithms for searching AND-OR graphs

Depth first

BFS, best first,A*, …

4.3.2 And-OR Graph Search:

Cycles arise often in non-deterministic problems

Algorithm returns with failure when the current state is identical to one of ancestors

If there is a non-cyclic path, the earlier consideration of the state is sufficient

Termination is guaranteed in finite state spaces

Every path reaches a goal, a dead-end, or a repeated state

Slippery vacuum world: Left and Right actions sometimes fail (leaving the agent in the same location)

Solution?

Cyclic plan: keep on trying an action until it works.

[Suck, 𝐿1: Right, if state = 5 then 𝐿1 else Suck]

◻ Or equivalently [Suck, while state = 5 do Right, Suck]

--------------------------------------------------------------------------------------------------------------------

4.4 Searching with Partial Observations:

The agent does not always know its exact state.

Agent is in one of several possible states and thus an action may lead to one of several
possible outcomes

Belief state: agent’s current belief about the possible states, given the sequence of actions
and observations up to that point.