Department of Computer Science & Engineering

Course: Artificial Intelligence and Machine Learning (18CS71)

Module 5
Classification using Instance-based Learning,
Reinforcement Learning

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Instance-based Learning
• Key idea: In contrast to learning methods that construct a general, explicit
description of the target function when training examples are provided, instance-
based learning constructs the target function only when a new instance must be
classified.

• Each time a new query instance is encountered, its relationship to the previously
stored examples is examined in order to assign a target function value for the new
instance.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Instance-based Learning
• Instance based learning includes nearest neighbor and locally weighted regression
methods that assume instances can be represented as points in a Euclidean space.

• It also includes case-based reasoning methods that use more complex, symbolic
representations for instances.

• Instance-based methods are sometimes referred to as "lazy" learning methods

because they delay processing until a new instance must be classified.

• A key advantage of this kind of delayed, or lazy, learning is that instead of estimating
the target function once for the entire instance space, these methods can estimate it
locally and differently for each new instance to be classified
Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Instance-based Learning
• Instance-based learning methods such as nearest neighbor and locally weighted
regression are conceptually straightforward approaches to approximating real-
valued or discrete-valued target functions.

• Learning in these algorithms consists of simply storing the presented training

data. When a new query instance is encountered, a set of similar related
instances is retrieved from memory and used to classify the new query instance

• Instance-based approaches can construct a different approximation to the target

function for each distinct query instance that must be classified

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Advantages of Instance-based learning
1. Training is very fast

2. Learn complex target function

3. Don’t lose information

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Disadvantages of Instance-based learning
• The cost of classifying new instances can be high.

• This is due to the fact that nearly all computation takes place at classification time
rather than when the training examples are first encountered.

• In many instance-based approaches, especially nearest-neighbor approaches, is

that they typically consider all attributes of the instances when attempting to
retrieve similar training examples from memory. If the target concept depends on
only a few of the many available attributes, then the instances that are truly most
"similar" may well be a large distance apart.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 k-NEAREST NEIGHBOR LEARNING
• The most basic instance-based method is the k-NEAREST NEIGHBOR algorithm.
• This algorithm assumes all instances correspond to points in the n-dimensional space Rn.
• The nearest neighbors of an instance are defined in terms of the standard Euclidean
distance.
• The arbitrary instance x be described by the feature vector

where ar(x) denotes the value of the rth attribute of instance x.

• Then the distance between two instances xi and xj is defined to be d ( xi , xj), where

• In nearest-neighbor learning the target function may be either discrete-valued or real-

valued.
Video Tutorial at https://www.youtube.com/@MaheshHuddar
 k-NEAREST NEIGHBOR LEARNING
• Let us first consider learning discrete-valued target functions of the form

• Where, V is the finite set {v1, . . . vs }

• The k- Nearest Neighbor algorithm for approximation a discrete-valued target function is
given below:

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 k-NEAREST NEIGHBOR LEARNING

Sl. No. Height Weight Target

1 150 50 Medium
2 155 55 Medium
3 160 60 Large
4 161 59 Large
5 158 65 Large
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 k-NEAREST NEIGHBOR LEARNING

Sl. No. Height Weight Target Distance

1 150 50 Medium 8.06
2 155 55 Medium 2.24
3 160 60 Large 6.71
4 161 59 Large 6.40
5 158 65 Large 11.05
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 k-NEAREST NEIGHBOR LEARNING

Nearest
Sl. No. Height Weight Target Distance
Points
1 150 50 Medium 8.06
2 155 55 Medium 2.24 1
3 160 60 Large 6.71 3
4 161 59 Large 6.40 2
5 158 65 Large 11.05
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 k-NEAREST NEIGHBOR LEARNING
• መ
The value 𝑓(𝑥𝑞) returned by this algorithm as its estimate of 𝑓(𝑥𝑞) is just
the most common value of 𝑓 among the 𝑘 training examples nearest to 𝑥𝑞.

• መ
If k = 1, then the 1- Nearest Neighbor algorithm assigns to 𝑓(𝑥𝑞) the value
𝑓(𝑥𝑖 ).

• Where xi is the training instance nearest to xq.

• For larger values of k, the algorithm assigns the most common value among the
k nearest training examples.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 k-NEAREST NEIGHBOR LEARNING
• Below figure illustrates the operation of the k-Nearest Neighbor algorithm for the case
where the instances are points in a two-dimensional space and where the target function
is Boolean valued.

• The positive and negative training examples are shown by “+” and “-” respectively.

• A query point 𝑥𝑞 is shown as well.

• The 1-Nearest Neighbor algorithm classifies 𝑥𝑞 as a positive example in this figure,

whereas the 5-Nearest
Video Tutorial at Neighbor algorithm classifies it as a negative example.
https://www.youtube.com/@MaheshHuddar
Distance-Weighted Nearest Neighbor Algorithm

Video Tutorial at https://www.youtube.com/@MaheshHuddar

Distance-Weighted Nearest Neighbor Algorithm

Sl. No. Height Weight Target

1 150 50 Medium
2 155 55 Medium
3 160 60 Large
4 161 59 Large
5 158 65 Large
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

Distance-Weighted Nearest Neighbor Algorithm

Sl. No. Height Weight Target Distance

1 150 50 Medium 8.06
2 155 55 Medium 2.24
3 160 60 Large 6.71
4 161 59 Large 6.40
5 158 65 Large 11.05
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

Distance-Weighted Nearest Neighbor Algorithm
Nearest
Sl. No. Height Weight Target Distance
Points
1 150 50 Medium 8.06
2 155 55 Medium 2.24 1
3 160 60 Large 6.71 3
4 161 59 Large 6.40 2
5 158 65 Large 11.05
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

Distance-Weighted Nearest Neighbor Algorithm
Nearest
Sl. No. Height Weight Target Distance 1/distance2
Points
1 150 50 Medium 8.06
2 155 55 Medium 2.24 0.45 1
3 160 60 Large 6.71 0.15 3
4 161 59 Large 6.40 0.16 2
5 158 65 Large 11.05
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 k-NEAREST NEIGHBOR LEARNING
• The K- Nearest Neighbor algorithm for approximation a real-valued target function is given
below

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 k-NEAREST NEIGHBOR LEARNING

Sl. No. Height Weight Target

1 150 50 1.5
2 155 55 1.2
3 160 60 1.8
4 161 59 2.1
5 158 65 1.7
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 k-NEAREST NEIGHBOR LEARNING

Sl. No. Height Weight Target Distance

1 150 50 1.5 8.06
2 155 55 1.2 2.24
3 160 60 1.8 6.71
4 161 59 2.1 6.40
5 158 65 1.7 11.05
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 k-NEAREST NEIGHBOR LEARNING

Nearest
Sl. No. Height Weight Target Distance
Points
1 150 50 1.5 8.06
2 155 55 1.2 2.24 1
3 160 60 1.8 6.71 3
4 161 59 2.1 6.40 2
5 158 65 1.7 11.05
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Distance-Weighted Nearest Neighbor Algorithm
• The refinement to the k-NEAREST NEIGHBOR Algorithm is to weight the
contribution of each of the k neighbors according to their distance to the query
point 𝑥𝑞, giving greater weight to closer neighbors.

• For example, in the k-Nearest Neighbor algorithm, which approximates discrete-

valued target functions, we might weight the vote of each neighbor according to
the inverse square of its distance from 𝑥𝑞.

• Distance-Weighted Nearest Neighbor Algorithm for approximation a discrete-

valued target functions

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Distance-Weighted Nearest Neighbor Algorithm
• Distance-Weighted Nearest Neighbor Algorithm for approximation a Real-valued target
functions

Video Tutorial at https://www.youtube.com/@MaheshHuddar

Distance-Weighted Nearest Neighbor Algorithm

Sl. No. Height Weight Target

1 150 50 1.5
2 155 55 1.2
3 160 60 1.8
4 161 59 2.1
5 158 65 1.7
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

Distance-Weighted Nearest Neighbor Algorithm

Sl. No. Height Weight Target Distance

1 150 50 1.5 8.06
2 155 55 1.2 2.24
3 160 60 1.8 6.71
4 161 59 2.1 6.40
5 158 65 1.7 11.05
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

Distance-Weighted Nearest Neighbor Algorithm
Nearest
Sl. No. Height Weight Target Distance
Points
1 150 50 1.5 8.06
2 155 55 1.2 2.24 1
3 160 60 1.8 6.71 3
4 161 59 2.1 6.40 2
5 158 65 1.7 11.05
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

Distance-Weighted Nearest Neighbor Algorithm

Nearest
Sl. No. Height Weight Target Distance 1/distance2
Points
1 150 50 1.5 8.06
2 155 55 1.2 2.24 0.45 1
3 160 60 1.8 6.71 0.15 3
4 161 59 2.1 6.40 0.16 2
5 158 65 1.7 11.05
6 157 54 ?

Video Tutorial at https://www.youtube.com/@MaheshHuddar

Suppose we have height, weight and T-shirt size of some customers and we
need to predict the T-shirt size of a new customer given only height and
weight information we have. Data including height, weight and T-shirt size
information is shown below
Height (in cms) Weight (in kgs) T Shirt Size
158 58 M
158 59 M
158 63 M
160 59 M
160 60 M
163 60 M
163 61 L
160 64 L
163 64 L
165 61 L
165 62 L
165 65 L
168 62 L
168 63 L
168 66 L
170 63 L

Video Tutorial at 170

https://www.youtube.com/@MaheshHuddar
64 L
170 68 L
New customer named XYZ' has height 161cm and
weight 61kg.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

Video Tutorial at https://www.youtube.com/@MaheshHuddar
 LOCALLY WEIGHTED REGRESSION
• Locally weighted regression is instance based learning algorithm.

• The phrase "locally weighted regression" is called

– local because the function is approximated based only on data near the query
point,

– weighted because the contribution of each training example is weighted by its

distance from the query point, and

– regression because this is the term used widely in the statistical learning
community for the problem of approximating real-valued functions.
Video Tutorial at https://www.youtube.com/@MaheshHuddar
 LOCALLY WEIGHTED REGRESSION
• Given a new query instance xq, the general approach in locally

weighted regression is to construct an approximation 𝑓መ that fits the

training examples in the neighborhood surrounding of xq.

መ 𝑞), which
• This approximation is then used to calculate the value 𝑓(𝑥

is output as the estimated target value for the query instance.

Video Tutorial at https://www.youtube.com/@MaheshHuddar
 LOCALLY WEIGHTED REGRESSION

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 LOCALLY WEIGHTED REGRESSION

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 LOCALLY WEIGHTED REGRESSION
• Consider locally weighted regression in which the target function f is approximated near
xq using a linear function of the form

• Where, ai(x) denotes the value of the ith attribute of the instance x

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 LOCALLY WEIGHTED REGRESSION
• Derived methods are used to choose weights that minimize the squared error summed
over the set D of training examples using gradient descent

• Which led us to the gradient descent training rule

• Where, η is a constant learning rate

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 LOCALLY WEIGHTED REGRESSION
Need to modify this procedure to derive a local approximation rather than a global
one. The simple way is to redefine the error criterion E to emphasize fitting the local
training examples. Three possible criteria are given below.
1. Minimize the squared error over just the k nearest neighbors:

2. Minimize the squared error over the entire set D of training examples, while weighting
the error of each training example by some decreasing function K of its distance from xq :

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 LOCALLY WEIGHTED REGRESSION
3. Combine 1 and 2:

• If we choose criterion three and re-derive the gradient descent rule, we obtain the
following training rule

• The differences between this new rule and the rule given by Equation (3) are that the
contribution of instance x to the weight update is now multiplied by the distance penalty
K(d(xq, x)), and that the error is summed over only the k nearest training examples.
Video Tutorial at https://www.youtube.com/@MaheshHuddar
 RADIAL BASIS FUNCTIONS
• One approach to function approximation that is closely related to distance-weighted
regression and also to artificial neural networks is learning with radial basis functions.
• In this approach, the learned hypothesis is a function of the form

• Where, each xu is an instance from X and where the kernel function Ku(d(xu, x)) is defined
so that it decreases as the distance d(xu, x) increases.
• Here k is a user provided constant that specifies the number of kernel functions to be
included.
• 𝑓መ is a global approximation to f (x), the contribution from each of the Ku(d(xu, x)) terms is
localized to a region nearby the point xu.
Video Tutorial at https://www.youtube.com/@MaheshHuddar
 RADIAL BASIS FUNCTIONS
• Choose each function Ku(d(xu, x)) to be a Gaussian function centred at the point xu with
some variance 𝜎u2

• The functional form of equ(1) can approximate any function with arbitrarily small error,
provided a sufficiently large number k of such Gaussian kernels and provided the width

• 𝜎2 of each kernel can be separately specified

• The function given by equ(1) can be viewed as describing a two layer network where the
first layer of units computes the values of the various Ku(d(xu, x)) and where the second
layer computes a linear combination of these first-layer unit values
Video Tutorial at https://www.youtube.com/@MaheshHuddar
Video Tutorial at https://www.youtube.com/@MaheshHuddar
 CASE-BASED REASONING
• Instance-based methods such as k-NEAREST NEIGHBOR and locally weighted
regression share three key properties.

• First, they are lazy learning methods in that they defer the decision of how to
generalize beyond the training data until a new query instance is observed.

• Second, they classify new query instances by analyzing similar instances while
ignoring instances that are very different from the query.

• Third, they represent instances as real-valued points in an n-dimensional

Euclidean space.
Video Tutorial at https://www.youtube.com/@MaheshHuddar
 CASE-BASED REASONING
• In CBR represent instances are not represented as real-valued points, but instead,
they use a rich symbolic representation and the methods used to retrieve similar
instances are correspondingly more elaborate.

• CBR has been applied to problems such as conceptual design of mechanical

devices based on a stored library of previous designs, reasoning about new legal
cases based on previous rulings, and solving planning and scheduling problems by
reusing and combining portions of previous solutions to similar problems

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 CASE-BASED REASONING
• In case-based reasoning, the training examples, the cases, are stored and
accessed to solve a new problem.

• To get a prediction for a new example, those cases that are similar, or close to, the
new example are used to predict the value of the target features of the new
example.

• This is at one extreme of the learning problem where, unlike decision trees and
neural networks, relatively little work must be done offline, and virtually all of the
work is performed at query time.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 CASE-BASED REASONING
Case-based reasoning consists of a cycle of the following four steps:

1. Retrieve - Given a new case, retrieve similar cases from the case
base.

2. Reuse - Adapt the retrieved cases to fit to the new case.

3. Revise - Evaluate the solution and revise it based on how well it

works.

4. Retain - Decide whether to retain this new case in the case base.
Video Tutorial at https://www.youtube.com/@MaheshHuddar
 CASE-BASED REASONING – Example
• A common example of a case-based reasoning system is a help desk that users call with
problems to be solved.
• Case-based reasoning could be used by the diagnostic assistant to help users diagnose problems
on their computer systems.
• When users give a description of a problem, the closest cases in the case base are retrieved. The
diagnostic assistant could recommend some of these to the user, adapting each case to the
user’s particular situation.
• An example of adaptation is to change the recommendation based on what software the user
has, what method they use to connect to the Internet, and the model of the printer.
• If one of the adapted cases works, that case is added to the case base, to be used when another
user asks a similar question.
• In this way, all of the common different cases will eventually be in the case base.
• If none of the cases found works, some other method is attempted to solve the problem,
perhaps by adapting other cases or having a human help diagnose the problem.
• WhenVideo
the Tutorial
problem at is finally solved, the solution is added to the case base.
https://www.youtube.com/@MaheshHuddar
 A prototypical example of a case-based reasoning
• The CADET system employs case-based reasoning to assist in the conceptual design
of simple mechanical devices such as water faucets.

• It uses a library containing approximately 75 previous designs and design fragments

to suggest conceptual designs to meet the specifications of new design problems.

• Each instance stored in memory (e.g., a water pipe) is represented by describing

both its structure and its qualitative function.

• New design problems are then presented by specifying the desired function and
requesting the corresponding structure.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A prototypical example of a case-based reasoning
• The problem setting is illustrated in below figure

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A prototypical example of a case-based reasoning
• The function is represented in terms of the qualitative relationships among the water-
flow levels and temperatures at its inputs and outputs.
• In the functional description, an arrow with a "+" label indicates that the variable at the
arrowhead increases with the variable at its tail. A "-" label indicates that the variable at
the head decreases with the variable at the tail.
• Here Qc refers to the flow of cold water into the faucet, Qh to the input flow of hot
water, and Qm to the single mixed flow out of the faucet.
• Tc, Th, and Tm refer to the temperatures of the cold water, hot water, and mixed water
respectively.
• The variable Ct denotes the control signal for temperature that is input to the faucet, and
Cf denotes the control signal for waterflow.
• The controls Ct and Cf are to influence the water flows Qc and Qh, thereby indirectly
influencing the faucet output flow Qm and temperature Tm.
Video Tutorial at https://www.youtube.com/@MaheshHuddar
A prototypical example of a case-based reasoning

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 A prototypical example of a case-based reasoning
• CADET searches its library for stored cases whose functional descriptions match
the design problem.

• If an exact match is found, indicating that some stored case implements exactly
the desired function, then this case can be returned as a suggested solution to
the design problem.

• If no exact match occurs, CADET may find cases that match various subgraphs of
the desired functional specification.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

Video Tutorial at https://www.youtube.com/@MaheshHuddar
 OVERVIEW OF REINFORCEMENT LEARNING
• Reinforcement learning (RL) is a branch of machine learning and imitates human beings.

• Humans observe the environment through senses such as eye and ear.

• The inputs are processed by brain.

• The brain then suggests the actions and acts voluntarily or involuntarily.

• In humans, learning happens in a real world called environment.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 OVERVIEW OF REINFORCEMENT LEARNING
• Take a kid for an example. How does the kid learn?

• The kid interacts with the environment and gains valuable experience.

• Experience leads to learning.

• For example, when a kid accidently burns his hand in fire, he learns to stay away from it in
the future and be careful.

• Similarly, the kid learns to do good things repeatedly by the encouragement or gifts given
by parents or teachers.

• Rewards can be positive or negative.

• Thus, kids gain experience and learn many things through rewards and punishments
Video Tutorial at https://www.youtube.com/@MaheshHuddar
(negative rewards).
OVERVIEW OF REINFORCEMENT LEARNING

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 OVERVIEW OF REINFORCEMENT LEARNING
• In computers, these scenarios can be simulated in many ways.

• As a kid executes many actions and receives a mix of positive and negative rewards that
lead to gaining experience and learning, a computer program or robot can learn through
experiences of simulated scenarios.

• Thus, reinforcement learning is a mathematical framework for learning.

• There are two types of reinforcement learning - positive and negative.

• Positive reinforcement learning is a recurrence of behaviour due to positive rewards.

• Rewards increase strength and the frequency of a specific behaviour.

• This encourages to execute similar actions that yield maximum reward.

Video Tutorial at https://www.youtube.com/@MaheshHuddar
 OVERVIEW OF REINFORCEMENT LEARNING
• Similarly, in negative reinforcement learning, negative rewards are used as a deterrent to
weaken the behaviour and to avoid it.

• In a maze game, there may a danger spot that may lead to loss.

• Negative rewards can be designed for such spots so that the agent does not visit that spot.

• Positive and negative rewards are simulated in reinforcement learning, say +10 for positive
reward and -10 for some danger or negative reward.

• Reinforcement learning is an example of semi-supervised learning technique and is used to

model sequential decision-making process.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Challenges of Reinforcement Learning
1. Reward design is a big challenge as in many games, as determining the rewards and its value is a challenge.

2. Absence of a model is a challenge - Games like chess have fixed board and rules. But, many games do not
have any fixed environment or rules. There is no underlying model as well. So, simulation must be done to
gather experience.

3. Partial observability of states - Many states are fully observable. Imagine a scenario in a weather forecasting
where the uncertainty or partial observability exists as complete information about the state is simply not
available.

4. Time consuming operations - More state spaces and possible actions may complicate the scenarios,
resulting in more time consumption.

5. Complexity- Many games like GO are complicated with much larger board configuration and many
possibilities of actions. So, labelled data is simply not available. This adds more complexity to the design of
Video Tutorial
reinforcement at
algorithms. https://www.youtube.com/@MaheshHuddar
REINFORCEMENT LEARNING AS MACHINE LEARNING
• Differences between Reinforcement Learning and Supervised Learning

Reinforcement Learning Supervised Learning

No supervisor and labelled dataset initially Presence of supervisor and labelled data
Decisions are dependent and are made Decisions are independent of each other and
sequentially based on input given in the training phase
Feedback is not instantaneous and delayed by Usually, the feedback is instantaneous once the
time model is created
Agent action affects the next input data Dependent of initial input or the input given at
start
No target values, only goal oriented Target class is predefined by the problem
Video Tutorial at https://www.youtube.com/@MaheshHuddar
Example: Chess, GO, Atari Games Example: Classifiers
REINFORCEMENT LEARNING AS MACHINE LEARNING
• Differences between Reinforcement Learning and Unsupervised Learning

Reinforcement Learning Unsupervised Learning

Mapping from input to output is present Not present
Get constant feedback from environment No feedback from environment

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 REINFORCEMENT LEARNING
Reinforcement learning addresses the question of how an autonomous agent that senses and
acts in its environment can learn to choose optimal actions to achieve its goals.

• Consider building a learning robot.

• The robot, or agent, has a set of sensors to observe the state of its environment, and a set
of actions it can perform to alter this state.

• Its task is to learn a control strategy, or policy, for choosing actions that achieve its goals.

• The goals of the agent can be defined by a reward function that assigns a numerical value
to each distinct action the agent may take from each distinct state.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 REINFORCEMENT LEARNING
• This reward function may be built into the robot, or known only to an external teacher

who provides the reward value for each action performed by the robot.

• The task of the robot is to perform sequences of actions, observe their consequences,

and learn a control policy.

• The control policy is one that, from any initial state, chooses actions that maximize the

reward accumulated over time by the agent.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 REINFORCEMENT LEARNING
Example:
• A mobile robot may have sensors such as a camera and sonars, and actions such as
"move forward" and "turn."

• The robot may have a goal of docking onto its battery charger whenever its battery level
is low.

• The goal of docking to the battery charger can be captured by assigning a positive reward
(Eg., +100) to state-action transitions that immediately result in a connection to the
charger and a reward of zero to every other state-action transition.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 REINFORCEMENT LEARNING

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 REINFORCEMENT LEARNING
Reinforcement Learning Problem
• An agent interacting with its environment.

• The agent exists in an environment described by some set of possible states S.

• Agent perform any of a set of possible actions A.

• Each time it performs an action A, in some state st the agent receives a real-valued
reward r, that indicates the immediate value of this state-action transition.

• This produces a sequence of states si, actions ai, and immediate rewards ri as shown in
the figure.

• The agent's task is to learn a control policy, 𝝅: S → A, that maximizes the expected sum of
Video Tutorial at
these rewards, with future https://www.youtube.com/@MaheshHuddar
rewards discounted exponentially by their delay.
 Reinforcement learning problem characteristics
1. Delayed reward: The task of the agent is to learn a target function 𝜋 that maps from the current state
s to the optimal action a = 𝜋 (s). In reinforcement learning, training information is not available in (s,
𝜋 (s)). Instead, the trainer provides only a sequence of immediate reward values as the agent
executes its sequence of actions. The agent, therefore, faces the problem of temporal credit
assignment: determining which of the actions in its sequence are to be credited with producing the
eventual rewards.

2. Exploration: In reinforcement learning, the agent influences the distribution of training examples by
the action sequence it chooses. This raises the question of which experimentation strategy produces
most effective learning. The learner faces a trade-off in choosing whether to favor exploration of
unknown states and actions, or exploitation of states and actions that it has already learned will yield
high reward.
Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Reinforcement learning problem characteristics
3. Partially observable states: The agent's sensors can perceive the entire state of the environment at
each time step, in many practical situations sensors provide only partial information. In such cases,
the agent needs to consider its previous observations together with its current sensor data when
choosing actions, and the best policy may be one that chooses actions specifically to improve the
observability of the environment.

4. Life-long learning: Robot requires to learn several related tasks within the same environment, using
the same sensors. For example, a mobile robot may need to learn how to dock on its battery charger,
how to navigate through narrow corridors, and how to pick up output from laser printers. This setting
raises the possibility of using previously obtained experience or knowledge to reduce sample
complexity when learning new tasks.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning
THE LEARNING TASK
• Consider Markov decision process (MDP) where the agent can perceive a set S of distinct
states of its environment and has a set A of actions that it can perform.
• At each discrete time step t, the agent senses the current state st, chooses a current action
at, and performs it.
• The environment responds by giving the agent a reward rt = r(st, at) and by producing the
succeeding state st+1 = δ(st, at).
• Here the functions δ(st, at) and r(st, at) depend only on the current state and action, and
not on earlier states or actions.
• The task of the agent is to learn a policy, 𝝅: S → A, for selecting its next action a, based on
Video Tutorial
the current at
observed state sthttps://www.youtube.com/@MaheshHuddar
; that is, 𝝅(st) = at.
 Reinforcement learning
How shall we specify precisely which policy π we would like the agent to learn?

1. One approach is to require the policy that produces the greatest possible cumulative
reward for the robot over time.

– To state this requirement more precisely, define the cumulative value Vπ (st) achieved by
following an arbitrary policy π from an arbitrary initial state st as follows:

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning
• Where, the sequence of rewards rt+i is generated by beginning at state st and by
repeatedly using the policy π to select actions.

• Here 0 ≤ γ ≤ 1 is a constant that determines the relative value of delayed versus

immediate rewards. if we set γ = 0, only the immediate reward is considered. As we set γ
closer to 1, future rewards are given greater emphasis relative to the immediate reward.

• The quantity Vπ (st) is called the discounted cumulative reward achieved by policy π from
initial state s. It is reasonable to discount future rewards relative to immediate rewards
because, in many cases, we prefer to obtain the reward sooner rather than later.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning
• Other definitions of total reward is finite horizon reward,

Considers the undiscounted sum of rewards over a finite number h of steps

• Another approach is average reward

Considers the average reward per time step over the entire lifetime of the agent.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning
• We require that the agent learn a policy π that maximizes Vπ (st) for all states s. such a
policy is called an optimal policy and denote it by π*

• Refer the value function Vπ*(s) an optimal policy as V*(s).

• V*(s) gives the maximum discounted cumulative reward that the agent can obtain starting
from state s.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning
Example:
• A simple grid-world environment is depicted in the diagram

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning
• To illustrate the operation of the Q learning algorithm, consider a single action taken by
an agent, and the corresponding refinement to 𝑄̂ shown in below figure

• The agent moves one cell to the right in its grid world and receives an
immediate reward of zero for this transition.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning
• Apply the training rule of Equation

• to refine its estimate Q for the state-action transition it just executed.

• According to the training rule, the new 𝑄̂ estimate for this transition is the sum
of the received reward (zero) and the highest 𝑄̂ value associated with the
resulting state (100), discounted by γ (.9).

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning
• The six grid squares in this diagram represent six possible states, or locations, for the
agent.

• Each arrow in the diagram represents a possible action the agent can take to move from
one state to another.

• The number associated with each arrow represents the immediate reward r(s, a) the
agent receives if it executes the corresponding state-action transition

• The immediate reward in this environment is defined to be zero for all state-action
transitions except for those leading into the state labelled G.

• The state G as the goal state, and the agent can receive reward by entering this state.
Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Reinforcement learning
• Once the states, actions, and immediate rewards are defined, choose a value for the
discount factor γ, determine the optimal policy π * and its value function V*(s).

• Let’s choose γ = 0.9. The diagram at the bottom of the figure shows one optimal policy for
this setting.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• Suppose we have 5 rooms in a building connected by doors as shown in the figure below.
We'll number each room 0 through 4. The outside of the building can be thought of as one
big room (5). Notice that doors 1 and 4 lead into the building from room 5 (outside).

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• We can represent the rooms on a graph, each room as a node, and each door as a link.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• The goal room is number 5
• The doors that lead immediately to the goal have an instant reward of 100. Other doors
not directly connected to the target room have zero reward.
• Each arrow contains an instant reward value, as shown below:

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• Each room, including outside, a "state", and the agent's movement from one room to
another will be an "action".
• In our diagram, a "state" is depicted as a node, while "action" is represented by the arrows.
• We have an agent in Room 2 and we want the agent to learn to reach outside the house
(5).

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• We can put the state diagram and the instant reward values into the following reward
table, "matrix R". The -1's in the table represent null values (i.e.; where there isn't a link
between nodes). For example, State 0 cannot go to State 1.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

Reinforcement learning - Illustrative Example

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• Q-Learning Algorithm
• To understand how the Q-learning algorithm works, we’ll go through a few episodes step
by step.
• The rest of the steps skipped are
• Learning rate = 0.8 and the initial state as Room 1.
• Initialize matrix Q as a zero matrix:

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• Look at the second row (state 1) of matrix R.
• There are two possible actions for the current state 1: go to state 3, or go to state 5.
• By random selection, we select to go to 5 as our action.

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• Now let’s imagine what would happen if our agent were in state 5.
• Look at the sixth row of the reward matrix R (i.e. state 5).
• It has 3 possible actions: go to state 1, 4 or 5.
• Q(state, action) = R(state, action) + Gamma * Max[Q(next state, all actions)]
• Q(1, 5) = R(1, 5) + 0.8 * Max[Q(5, 1), Q(5, 4), Q(5, 5)] = 100 + 0.8 * 0 = 100

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• Now let’s imagine what would happen if our agent were in state 5.
• Look at the sixth row of the reward matrix R (i.e. state 5).
• It has 3 possible actions: go to state 1, 4 or 5.
• Q(state, action) = R(state, action) + Gamma * Max[Q(next state, all actions)]
• Q(1, 5) = R(1, 5) + 0.8 * Max[Q(5, 1), Q(5, 4), Q(5, 5)] = 100 + 0.8 * 0 = 100

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• The next state, 5, now becomes the current state.

• Because 5 is the goal state, we’ve finished one episode.

• Our agent’s brain now contains an updated matrix Q as:

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• For the next episode, we randomly choose the initial state – say 3 (can go to 1, 2 & 4)

• Now we imagine that we are in state 1.

Video Tutorial at https://www.youtube.com/@MaheshHuddar
 Reinforcement learning - Illustrative Example
• Now we imagine that we are in state 1.
• Look at the second row of reward matrix R (i.e. state 1).
• It has 2 possible actions: go to state 3 or state 5.
• Then, we compute the Q value:
• Q(state, action) = R(state, action) + Gamma * Max[Q(next state, all actions)]
• Q(3, 1) = R(3, 1) + 0.8 * Max[Q(1, 3), Q(1, 5)] = 0 + 0.8 * Max(0, 100) = 80

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• The result of the computation is Q(3, 1) = 80
• because the reward is zero

Video Tutorial at https://www.youtube.com/@MaheshHuddar

 Reinforcement learning - Illustrative Example
• If our agent learns more through further episodes, it will finally reach convergence values
in matrix Q like:

• Tracing the best sequences of states is as simple as following the links with the highest
values at each state.

Video Tutorial at https://www.youtube.com/@MaheshHuddar