INTELLIGENT AGENTS

Chapter 2

AGENTS AND
ENVIRONMENTS
An agent is anything that can be viewed as
perceiving its environment through sensors and
acting upon that environment through actuators.
A human agent has eyes, ears, and other organs
for sensors and hands, legs, mouth, and other
body parts for actuators.
A robotic agent might have cameras and infrared
range finders for sensors and various motors for
actuators.
A software agent receives keystrokes, file contents,
and network packets as sensory inputs and acts on
the environment by displaying on the screen,
writing files, and sending network packets.

 Percept
agent's perceptual inputs at any given instant

percept sequence
percept sequence is the complete history of
everything the agent has ever perceived
an agent's choice of action at any given
instant can depend on the entire percept
sequence observed to date.
If we can specify the agent's choice of action
for every possible percept sequence, then we
have said more or less everything there is to
say about the agent.

AGENT FUNCTION
Mathematically speaking, we say that an
agent's behaviour is described by the agent
function that maps any given percept
sequence to an action.
agent function that describes any given agent
can be tabulated
a very large table-infinite for some agents
(unless we place a bound on the length of
percept sequences we want to consider)
Given an agent to experiment with, we can, in
principle, construct this table by trying out all
possible percept sequences and recording
which actions the agent does in response.

Cont..
Internally, the agent function for an
artificial agent will be implemented
by an agent program.
The agent function is an abstract
mathematical description; the agent
program
is
a
concrete
implementation, running on the
agent architecture.

Example
the vacuum-cleaner world shown in Figure 2.2.
This world is so simple that we can describe
everything that happens; it's also a made-up
world, so we can invent many variations. This
particular world has just two locations: squares
A and B. The vacuum agent perceives which
square it is in and1 whether there is dirt in the
square. It can choose to move left, move right,
suck up the dirt, or do nothing. One very simple
agent function is the following: if the current
square is dirty, then suck, otherwise move to the
other square
A partial tabulation of this agent function is
shown in Figure 2.3.

RATIONALITY (WISDOM)
A rational agent is one that does the
right thing--conceptually speaking,
every entry in the table for the agent
function is filled out correctly.
what does it mean to do the right thing?
the right action is the one that will cause the
agent to be most successful.
Therefore, we will need some way to
measure success.

Performance measures
A performance measure represents the
criterion for success of an agent's behaviour
When an agent is plunked down in an
environment, it generates a sequence of
actions according to the percepts it receives.
This sequence of actions causes the
environment to go through a sequence of
states.
If the sequence is desirable, then the agent
has performed well. (not one fixed measure
suitable for all agents)

 Consider the vacuum-cleaner agent from the

preceding section. We might propose to measure
performance by the amount of dirt cleaned up in a
single eight-hour shift.
A rational agent can maximize this performance
measure by cleaning up the dirt, then dumping it all on
the floor, then cleaning it up again, and so on.
A more suitable performance measure would reward
the agent for
having a clean floor. For example, one point could be
awarded for each clean square at each time step
(perhaps with a penalty for electricity consumed and
noise generated).
As a general rule, it is better to design performance
measures according to what one actually wants in the
environment, rather than according to how one thinks
the agent should behave.

Your Assignment
"clean floor" in the preceding paragraph is based
on average cleanliness over time. Yet the same
average cleanliness can be achieved by two
different agents, one of which does a mediocre
job all the time: while the other cleans
energetically but takes long breaks.
Which is preferable might seem to be a fine point of
janitorial science, but in fact it is a deep philosophical
question with far-reaching implications. Which is
better-a reckless life of highs and lows, or a safe but
humdrum existence? Which is better-an economy
where everyone lives in moderate poverty, or one in
which some live in plenty while others are very poor?

Rationality
What is rational at any given time
depends on four things:
The performance measure that defines
the criterion of success.
The agent's prior knowledge of the
environment.
The actions that the agent can perform.
The agent's percept sequence to date.

 This leads to a definition of a

rational agent:
For each possible percept sequence, a
rational agent should select an action
that is expected to maximize its
performance measure, given the
evidence provided by the percept
sequence and whatever built-in
knowledge the agent has.

Consider the simple vacuum-cleaner agent that cleans a

square if it is dirty and moves to the other square if not; this
is the agent function tabulated in Figure 2.3. Is this a
rational agent? That depends!
First, we need to say what the performance measure is,
what is known about the environment, and what sensors
and actuators the agent has. Let us assume the following:
The performance measure awards one point for each clean
square at each time step, over a "lifetime" of 1000 time steps.
The "geography" of the environment is known a priori (Figure
2.2) but the dirt distribution and the initial location of the
agent are not. Clean squares stay clean and sucking cleans the
current square. The Left and Right actions move the agent left
and right except when this would take the agent outside the
environment, in which case the agent remains where it is.
The only available actions are Left, Right, Suck, and NoOp (do
nothing).
The agent correctly perceives its location and whether that
location contains dirt.

We claim that under these circumstances the agent is

indeed rational; its expected performance is at least as high
as any other agent's

 One can see easily that the same agent would

be irrational under different circumstances. For
example, once all the dirt is cleaned up it will
oscillate needlessly back and forth; if the
performance measure includes a penalty of one
point for each movement left or right, the agent
will fare poorly. A better agent for this case
would do nothing once it is sure that all the
squares are clean. If clean squares can become
dirty again, the agent should occasionally
check and re-clean them if needed. If the
geography of the environment is unknown, the
agent will need to explore it rather than stick to
squares A and B.

Omniscience, learning, and

autonomy
An omniscient agent knows the actual outcome of its
actions and can act accordingly; but omniscience is
impossible in reality.
Consider the following example: I am walking along the
Champs Elysees one day and I see an old friend across
the street. There is no traffic nearby so, being rational, I
start to cross the street. Meanwhile, at 33,000 feet, a
cargo door falls off a passing airliner and before I make it
to the other side of the street I am flattened. Was I
irrational to cross the street? It is unlikely that my
obituary(an article in a newspaper about the life of
someone who has died recently) would read "Idiot
attempts to cross street.
This example shows that rationality is not the same as
perfection. Rationality maximizes expected performance,
while perfection maximizes actual performance

 Retreating from a requirement of perfection is

not just a question of being fair to agents. The
point is that if we expect an agent to do what
turns out to be the best action after the fact, it
will be impossible to design an agent to fulfil
this specification-unless we improve the
performance of crystal balls or time machines.
Our definition of rationality does not require
omniscience, then, because the rational choice
depends only on the percept sequence to date.
We must also ensure that we havent
unintentionally allowed the agent to engage in
decidedly under intelligent activities

 For example, if an agent does not look both ways

before crossing a busy road, then its percept
sequence will not tell it that there is a large truck
approaching at high speed. Does our definition of
rationality say that it's now OK to cross the road? Far
from it!
First, it would not be rational to cross the road given
this uninformative percept sequence: the risk of
accident from crossing without looking is too great.
Second, a rational agent should choose the "looking"
action before stepping into the street, because
looking helps maximize the expected performance.
Doing actions in order to modify future perceptssometimes called information gathering is an
important part of rationality

 Our definition requires a rational agent not only

to gather information, but also to learn as
much as possible from what it perceives. The
agent's initial configuration could reflect some
prior knowledge of the environment, but as the
agent gains experience this may be modified
and improved.
There are extreme cases in which the
environment is completely known a priori. In
such cases, the agent need not perceive or
learn; it simply acts correctly.
Of course, such agents are very fragile (weak).

 Consider the lowly dung beetle. After digging its nest and laying
its eggs, it fetches a ball of dung from a nearby heap to plug the
entrance. If the ball of dung is removed from its grasp en route,
the beetle continues on and pantomimes (Communication by
means of gesture and facial expression ) plugging the nest with
the non existent dung ball, never noticing that it is missing.
Evolution has built an assumption into the beetle's behavior, and
when it is violated, unsuccessful behavior results.
Slightly more intelligent is the sphex wasp. The female sphex will
dig a hole, go out and sting a caterpillar and drag it to the
burrow, enter the burrow again to check all is well, drag the
caterpillar inside, and lay its eggs. The caterpillar serves as a
food source when the eggs hatch. So far so good, but if an
entomologist moves the caterpillar a few inches away while the
sphex is doing the check, it will revert back to the "drag" step of
its plan, and will continue the plan without modification, even
after dozens of caterpillar-moving interventions. The sphex is
unable to learn that its innate(in-born) plan is failing, and thus
will not change it.

 Successful agents split the task of

computing the agent function into three
different periods:
when the agent is being designed, some of the
computation is done by its designers;
when it is deliberating (planning, calculating) on
its next action, the agent does more
computation;
and as it learns from experience, it does even
more computation to decide how to modify its
behavior.

AUTONOMY
an agent relies on the prior knowledge of its designer rather
than on its own percepts, we say that the agent lacks
autonomy.
A rational agent should be autonomous-it should learn what it
can to compensate for partial or incorrect prior knowledge.
For example, a vacuum-cleaning agent that learns to foresee
where and when additionaldirt will appear will do better than
one that does not. As a practical matter, one seldom requires
complete autonomy from the start: when the agent has had
little or no experience, it would have to act randomly unless
the designer gave some assistance. So, just as evolution
provides animals with enough built-in reflexes so that they can
survive long enough to learn for themselves, it would be
reasonable to provide an artificial intelligent agent with some
initial knowledge as well as an ability to learn. After sufficient
experience of its environment, the behavior of a rational agent
can become effectively independent of its prior knowledge.

Autonomy(independence,
self-rule)
A system is autonomous to the extent that
its own behavior is determined by its own
experience.
Therefore, a system is not autonomous if it
is guided by its designer according to a
priori decisions.
To survive, agents must have:
Enough built-in knowledge to survive.
The ability to learn.

Autonomy
Example
An alarm that goes off at a pre-specified time
is not autonomous
An alarm that goes off when smoke is sensed
is autonomous

A system without autonomy lacks flexibility

To the extent that an agent relies on the
prior knowledge of its designer rather than
on its own percepts, we say that the agent
lacks autonomy

 A rational agent should be autonomous-it

should learn what it can to compensate for
partial or incorrect prior knowledge.
For example, a vacuum-cleaning agent
that learns to foresee where and when
additional dirt will appear will do better
than one that does not.
As a practical matter, one seldom requires
complete autonomy from the start: when
the agent has had little or no experience, it
would have to act randomly unless the
designer gave some assistance.

 So, just as evolution provides animals with

enough built-in reflexes so that they can
survive long enough to learn for
themselves, it would be reasonable to
provide an artificial intelligent agent with
some initial knowledge as well as an ability
to learn. After sufficient experience of its
environment, the behavior of a rational
agent can become effectively independent
of its prior knowledge. Hence, the
incorporation of learning allows one to
design a single rational agent that will
succeed in a vast variety of environments.

Nature of Environments
task environments
"problems" to which rational agents are
the "solutions

task environments come in a variety

of flavours.
The flavour of the task environment
directly affects the appropriate
design for the agent program.

Specifying the task

environment
In designing an agent, the first step
must always be to specify the task
environment as fully as possible.
PEAS (Performance, Environment,
Actuators, Sensors)
Example vacuum cleaner

A more specific example:

Automated taxi driving system
Percepts: Video, sonar, speedometer, odometer, engine
sensors, keyboard input, microphone, GPS,
Actions: Steer, accelerate, brake, horn, speak/display,
Goals or performance measure: Maintain safety, reach
destination, maximize profits (fuel, tire wear), obey laws,
provide passenger comfort,
Environment: U.S. urban streets, freeways, traffic,
pedestrians, weather, customers,

Different aspects of driving may

require different types of agent
programs!

 First, what is the performance measure

to
which
we
would
like
our
automated driver to aspire? Desirable
qualities include getting to the correct
destination; minimizing fuel consumption
and wear and tear; minimizing the trip
time and/or cost; minimizing violations of
traffic laws and disturbances to other
drivers; maximizing safety and passenger
comfort; maximizing profits. Obviously,
some of these goals conflict, so there will
be tradeoffs involved.

 Next, what is the driving environment that

the taxi will face? Any taxi driver must
deal with a variety of roads, ranging from rural
lanes and urban alleys to 12-lane freeways. The
roads contain other traffic, pedestrians, stray
animals, road works, police cars, puddles, and
potholes. The taxi must also interact with
potential and actual passengers. There are also
some optional choices. The taxi might need to
operate in Southern California, where snow is
seldom a problem, or in Alaska, where it seldom
is not. It could always be driving on the right, or
we might want it to be flexible enough to drive
on the left when in Britain or Japan. Obviously,
the more restricted the environment, the easier
the design problem.

 The actuators available to an automated

taxi will be more or less the same as
those
available to a human driver: control over the
engine through the accelerator and control
over steering and braking. In ;addition, it will
need output to a display screen or voice
synthesizer to talk back to the passengers,
and perhaps some way to communicate with
other vehicles, politely or otherwise.

 To achieve its goals in the driving environment, the

taxi will need to know where it is, what else is on the
road, and how fast it is going. Its basic sensors
should therefore include one or more controllable
TV cameras, the speedometer, and the odometer. To
control the vehicle properly, especially on curves, it
should have an accelerometer; it will also need to
know the mechanical stxte of the vehicle, so it will
need the usual array of engine and electrical system
sensors. It might have instruments that are not
available to the average human driver: a satellite
global positioning system (GPS) to give it accurate
position information with respect to an electronic
map, and infrared or sonar sensors to detect
distances to other cars and obstacles. Finally, it will
need a keyboard or microphone for the passenger to
request a destination.

Examples of Agent Types and their PEAS

Descriptions

SOFTWARE AGENTS
SOFTBOTS
contrast, some software agents (or software

In
robots or softbots) exist in rich, un-limited domains.
Imagine a softbot designed to fly a flight simulator for a
large commercial airplane. The simulator is a very
detailed, complex environment including other aircraft
and ground operations, and the software agent must
choose from a wide variety of actions in real time.
imagine a softbot designed to scan Internet news
sources and show the interesting items to its
customers. To do well, it will need some natural
language processing abilities, it will need to learn what
each customer is interested in, and it will need to
change its plans dynamically-for example, when the
connection for one news source goes down or when a
new one comes online. The Internet is an environment
whose complexity rivals that of the physical world and
whose inhabitants include many artificial agents.

Properties of task
environments

Fully observable vs. partially

observable

If an agent's sensors give it access to the complete state

of the environment at each point in time, then we say
that the task environment is fully observable.
A task environment is effectively fully observable if the
sensors detect all aspects that are relevant to the choice
of action; relevance, in turn, depends on the
performance measure
An environment might be partially observable because
of noisy and inaccurate sensors or because parts of the
state are simplly missing from the sensor data-for
example, a vacuum agent with only a local dirt sensor
cannot tell whether there is dirt in other squares, and an
automated taxi cannot see what other drivers are
thinking.

Deterministic vs.
stochastic
If the next state
of the environment is completely

determined by the current state and the action

executed by the agent, then we say the environment
is deterministic; otherwise, it is stochastic
In principle, an agent need not worry about
uncertainty in a fully observable, deterministic
environment. If the environment is partially
observable, however, then it could appear to be
stochastic.
This is particularly true if the environment is
complex, making it hard to keep track of all the
unobserved aspects. Thus, it is often better to think
of an environment as deterministic or stochastic
from the point of view of the agent.

 Taxi driving is clearly stochastic in this

sense, because one can never predict the
behavior of traffic exactly; moreover, one's
tires blow out and one's engine seizes up
without warning. The vacuum world as we
described it is deterministic, but variations
can include stochastic elements such as
randomly appearing dirt and an unreliable
suction mechanism.

Strategic
If the environment is deterministic except for
the actions of other agents, we say that the
environment is startegic

Episodic vs. sequential

In an episodic task environment, the agent's
experience is divided into atomic episodes. Each
episode consists of the agent perceiving and then
performing a single action. Crucially, the next episode
does not depend on the actions taken in previous
episodes. In episodic environments, the choice of
action in each episode depends only on the episode
itself. Many classification tasks are episodic.
For example, an agent that has to spot defective parts
on an assembly line bases each decision on the
current part, regardless of previous decisions;
moreover, the current decision doesn't affect whether
the next part is defective

 In sequential environments, on the

other hand, the current decision
could affect all future decisions.
Chess
and
taxi
driving
are
sequential: in both cases, short-term
actions
can
have
long-term
consequences.
Episodic
environments are much simpler than
sequential environments because the
agent does not need to think ahead.

Static vs dynamic
If the environment can change while an
agent is deliberating (calculating, planning)
then we say the environment is dynamic for
that agent; otherwise, it is static.
Static environments are easy to deal with
because the agent need not keep looking at
the world while it is deciding on an action,
nor need it worry about the passage of time
Dynamic environments, on the other hand,
are continuously asking the agent what it
wants to do; if it hasn't decided yet, that
counts as deciding to do nothing

 If the environment itself does not change

with the passage of time but the agent's
performance score does, then we say the
environment is semi-dynamic.
Taxi driving is clearly dynamic: the other
cars and the taxi itself keep moving while
the driving algorithm dithers about what to
do next. Chess, when played with a clock,
is semi-dynamic. Crossword puzzles are
static.

Discrete vs. continuous

If the number of distinct percepts and actions
is limited, the environment is discrete,
otherwise it is continuous.
For example, a discrete-state environment such as a
chess game has a finite number of distinct states.
Chess also has a discrete set of percepts and actions.
Taxi driving is a continuousstate and continuous-time
problem: the speed and location of the taxi and of the
other vehicles sweep through a range of continuous
values and do so smoothly over time. Taxi-driving
actions are also continuous (steering angles, etc.).

Single agent vs. Multiagent

If one agent percepts the
environment then environment is
called single agent otherwise it will be
multi-agent environment
an agent solving a crossword puzzle
by itself is clearly in a single-agent
environment, whereas an agent
playing chess is in a two-agent
environment.

Characteristics of
environments
Accessible Deterministic Episodic Static
Solitaire
Backgammon
Taxi driving
Internet
shopping
Medical
diagnosis

Discrete

Characteristics of
environments
Solitaire
Backgammon
Taxi driving
Internet
shopping
Medical
diagnosis

Accessible Deterministic Episodic Static

Discrete

No

Yes

Yes

Yes

Yes

Characteristics of
environments
Accessible Deterministic Episodic Static

Discrete

No

Yes

Yes

Yes

Yes

Backgammon Yes

No

No

Yes

Yes

Solitaire

Taxi driving
Internet
shopping
Medical
diagnosis

Characteristics of
environments
Accessible Deterministic Episodic Static

Discrete

No

Yes

Yes

Yes

Yes

Backgammon Yes

No

No

Yes

Yes

Taxi driving

No

No

No

No

Solitaire

Internet
shopping
Medical
diagnosis

No

Characteristics of
environments
Accessible Deterministic Episodic Static

Discrete

No

Yes

Yes

Yes

Yes

Backgammon Yes

No

No

Yes

Yes

Taxi driving

No

No

No

No

No

Internet
shopping

No

No

No

No

No

Solitaire

Medical
diagnosis

Characteristics of
environments
Accessible Deterministic Episodic Static

Discrete

No

Yes

Yes

Yes

Yes

Backgammon Yes

No

No

Yes

Yes

Taxi driving

No

No

No

No

No

Internet
shopping

No

No

No

No

No

Medical
diagnosis

No

No

No

No

No

Solitaire

Lots of real-world domains fall into the hardest case!

The Structure of Agents

agent = architecture +program
agent program that implements the
agent function mapping percepts to
actions.

The program will run on some sort of

computing device with physical
sensors and actuators-we call this
the architecture

Agent programs
The agent programs take the current
percept as input from the sensors
and return an action to the actuators.
difference between the agent
program, which takes the current
percept as input, and the agent
function, which takes the entire
percept history.

Kinds of agent program

Simple reflex agents;

Model-based reflex agents;
Goal-based agents; and
Utility-based agents.

Simple reflex agents

Specific response to percepts, i.e.
condition-action rule
if new-boulder(large stone)-in-sight
then
move-towards-new-boulder
Pretty easy to implement
Requires no knowledge of past
Limited what you can do with it

A Simple Reflex Agent:

Schema

Model-based reflex
agents
Encode "internal state" of the world to remember the
past as contained in earlier percepts
Needed because sensors do not usually give the
entire state of the world at each input, so perception
of the environment is captured over time. "State"
used to encode different "world states" that generate
the same immediate percept.
Requires ability to represent change in the world; one
possibility is to represent just the latest state, but
then can't reason about hypothetical courses of action
Example: Rodney Brookss Subsumption Architecture

Agents that Keep Track of the

World

Brooks Subsumption
(include or place within
something large)
Architecture
Main idea: build complex, intelligent robots by

decomposing behaviors into a hierarchy of skills,

each completely defining a complete perceptaction cycle for one very specific task.
Examples: avoiding contact, wandering, exploring,
recognizing doorways, etc.
Each behavior is modeled by a finite-state machine
with a few states (though each state may
correspond to a complex function or module).
Behaviors are loosely coupled, asynchronous (no
timing requirement for transmission) interactions.

Goal-Based Agent
Choose actions so as to achieve a (given or computed)
goal.
A goal is a description of a desirable situation
Keeping track of the current state is often not enough -need to add goals to decide which situations are good
Deliberative (think about or discuss very carefully in
order to make decision) instead of reactive (response
to a problem)
May have to consider long sequences of possible
actions before deciding if goal is achieved -- involves
consideration of the future, what will happen if I
do...?

Agents with Explicit Goals

Utility-Based Agent
When there are multiple possible alternatives, how to
decide which one is best?
A goal specifies a crude (very simple or basic)
distinction between a happy and unhappy state, but
often need a more general performance measure
that describes "degree of happiness"
Utility function U: State --> Reals indicating a
measure of success or happiness when at a given
state
Allows decisions comparing choice between
conflicting goals, and choice between likelihood of
success and importance of goal (if achievement is
uncertain)

A Complete Utility-Based
Agent