UNIT IV

Function Approximation Methods : Getting started with the function approximation

methods, Revisiting risk minimization, gradient descent from Machine Learning, Gradient
MC and Semi-gradient TD(0) algorithms, Eligibility trace for function approximation, After
states, Control with function approximation, Least squares, Experience replay in deep Q-
Networks.

Getting started with the function approximation methods

 Function approximation is a crucial concept in reinforcement learning (RL) that involves
representing complex and continuous functions, typically the value function or policy,
using a parameterized model. In RL, agents interact with an environment, receive
feedback in the form of rewards, and learn to make decisions that maximize their
cumulative reward over time. Function approximation is employed to handle situations
where the state or action space is too large to be explicitly represented, making it
infeasible to store or compute values for every possible state-action pair.
 Motivation:
 Tabular Methods: Traditional RL algorithms use tables to store values for each
state and action pair. This approach is simple but suffers from the curse of
dimensionality, becoming impractical for large state spaces.
 Function Approximation: Introduces a parameterized function to approximate
the value function or Q-function. This reduces memory requirements and
enables learning for continuous state spaces.
Methods:
 Linear Function Approximation:
 Represents the value function as a linear combination of features extracted from
the state:
V(s) = wT * phi(s)
 where w are weights, phi(s) is a vector of features, and ^T denotes the
transpose.
 Advantages: Simple to implement and interpret, computationally efficient.
 Disadvantages: Limited expressiveness for complex problems.
 Neural Networks:
 More powerful and flexible than linear models, capable of capturing non-linear
relationships in the state space.
 Different network architectures can be used, such as Multi-Layer Perceptrons
(MLPs) and Convolutional Neural Networks (CNNs).
 Advantages: High capacity for complex environments, can learn intricate
relationships between features.
 Disadvantages: More complex to train and interpret, can be computationally
expensive.
 Temporal Difference (TD) Learning:
 TD learning combines ideas from Monte Carlo methods and dynamic
programming. It updates the value function based on the difference between the
predicted value and the observed reward plus the estimated value of the next
state.
 TD methods, such as Q-learning and SARSA, are more computationally
efficient than pure Monte Carlo methods.
 Benefits of Function Approximation:
 Scalability: Handles large and continuous state spaces efficiently.
 Generalization: Learns from past experiences and applies knowledge to similar
situations.
 Compactness: Requires less memory compared to storing values for every state-
action pair.
 Function approximation is a fundamental aspect of reinforcement learning, enabling
agents to generalize their knowledge from observed experiences to unvisited states and
actions. The choice of the method depends on the specific characteristics of the problem,
such as the size of the state and action spaces, the complexity of the underlying function,
and the available computational resources.

Revisiting risk minimization

 Revisiting risk minimization refers to the idea of incorporating risk-sensitive
considerations into the learning process. Traditional RL algorithms often focus on
maximizing expected rewards without explicitly accounting for the uncertainty or risk
associated with different actions or policies.
 Risk-sensitive RL aims to address this limitation by taking into account not only the
expected return but also the variance or uncertainty of the outcomes.
 Risk Minimization in RL: Risk can be defined in various ways, such as:
 Variance: The spread of potential outcomes around the expected return.
 Worst-case scenario: The minimum feasible reward or maximum possible loss.
Several approaches have been proposed for incorporating risk minimization into RL
algorithms:
1. Risk Measures: Risk-sensitive RL often involves the use of risk measures to quantify
and manage uncertainty. Common risk measures include variance, conditional value at
risk (CVaR), and entropy. These measures capture different aspects of risk and provide a
more nuanced understanding of the distribution of possible outcomes.
2. Regularization techniques: Penalize high-variance policies during learning to encourage
stability.
3. Constrained optimization: Set constraints to limit the maximum potential loss or ensure
a minimum level of expected reward.
4. Distributional RL: Distributional RL is an approach that explicitly models the entire
distribution of returns for each state-action pair rather than just estimating the expected
value.
5. Exploration and Exploitation Trade-off: Balancing exploration and exploitation is a
fundamental challenge in RL. Risk-sensitive RL algorithms may place more emphasis on
exploring uncertain or unexplored regions of the state-action space, even if the expected
rewards are not maximized, to gain a better understanding of the environment's dynamics.
6. Safe Reinforcement Learning: Safe RL is a subfield that explicitly addresses the need
for agents to operate within certain safety constraints. It involves incorporating safety
considerations into the learning process to avoid catastrophic outcomes, especially in
real-world applications where the consequences of incorrect actions can be severe.
Benefits of Risk Minimization:
 Improved stability: Agents become less affected by noise and uncertainty in the
environment.
 More realistic behaviour: Encourages exploration and avoids unnecessary risk, leading to
more robust and adaptable agents.

Gradient Descent from Machine Learning

 Gradient descent is a fundamental optimization algorithm widely used in machine
learning, including reinforcement learning (RL). Its primary purpose is to iteratively
update the parameters of a model in the direction that minimizes a specified objective
function. In the context of machine learning and RL, this objective function is typically a
loss function that quantifies the difference between the predicted and actual values.
Gradient Descent in Machine Learning:
1. Objective Function: In machine learning, there is a model with parameters that need to
be adjusted to make accurate predictions. The objective function, also known as the loss
function or cost function, measures the difference between the predicted and true values.
2. Gradient Calculation: The gradient of the objective function is computed with respect to
the model parameters. The gradient is a vector that points in the direction of the steepest
increase in the objective function. It provides information on how the parameters should
be adjusted to decrease the loss.
3. Parameter Update: The model parameters are updated in the opposite direction of the
gradient. This involves multiplying the gradient by a learning rate, which determines the
size of the steps taken during the update. The learning rate is a hyperparameter that needs
to be carefully chosen to balance convergence speed and stability.
In RL, the application of gradient descent is often seen in two main contexts:
1. Policy Gradient Methods: In policy gradient methods, the objective is to directly
optimize the policy of an agent to maximize the expected cumulative reward. The policy
is parameterized by certain weights, and the gradient of the expected return with respect
to these weights is calculated.
 The gradient is then used to update the policy parameters through gradient ascent (as RL
typically involves maximizing rewards). The update process involves moving the policy
parameters in the direction that increases the likelihood of actions leading to higher
returns.
2. Value Function Approximation: When using value functions (e.g., state-value or
action-value functions) for estimating the expected return, gradient descent is employed
to update the parameters of the value function approximator. This is common in
algorithms like Q-learning or deep Q-networks (DQN).
The objective is to minimize the mean squared error between the predicted values and the
actual returns. The gradient of this error is computed with respect to the parameters of the
value function, and the parameters are updated in the direction that decreases the error.
Challenges of Gradient-based RL:
 Higher variance and instability compared to value-based methods.
 Requires careful design of the policy gradient estimator to avoid bias and noise.
 Can be computationally expensive for complex policies and large environments.

Gradient MC and Semi-gradient TD(0) algorithms

Gradient MC
"Gradient Monte Carlo" typically refers to a class of reinforcement learning algorithms that
combine ideas from Monte Carlo methods and gradient-based optimization. In the context of
reinforcement learning, Monte Carlo methods estimate the value of a state-action pair by
averaging the returns observed in sampled episodes. On the other hand, gradient-based
methods use the gradient of an objective function to update the parameters of a policy or
value function.
Working of Gradient MC
 Objective Function: The objective in reinforcement learning is often to maximize the
expected cumulative reward. This involves finding a policy or value function that yields
high expected returns.
 Monte Carlo Estimation: In Monte Carlo methods, the value of a state-action pair is
estimated by running episodes, collecting returns, and averaging them. This provides an
unbiased estimate of the expected return.
 Gradient Descent: Gradient-based methods aim to adjust the parameters of the policy or
value function to improve performance. The gradient of the expected return with respect
to the parameters is computed.
 Update Rule: The parameters are updated in the direction of the gradient, typically using
a gradient ascent or descent update rule. This adjusts the policy or value function to
increase the probability of actions that lead to higher returns or improve the accuracy of
value predictions.
 Repeat: Steps 2-4 are repeated iteratively, with new samples from episodes used to
update the estimates and adjust the parameters further.
Semi-gradient TD(0)
Similar to Gradient Monte Carlo, the objective is to estimate and optimize the value function
or policy. However, Semi-gradient TD(0) focuses on using Temporal Difference (TD)
learning, which combines elements of Monte Carlo methods and dynamic programming.
Steps:
1. TD(0) Update: Temporal Difference (TD) error is calculated at each time step,
representing the difference between the estimated value of a state and the immediate
reward plus the estimated value of the next state.
The TD error is used to update the value function estimates. Unlike Monte Carlo
methods, TD methods update the value function at each time step, allowing for online
learning.
2. Semi-gradient Update: While true gradient methods would update the parameters of a
value function approximator using the gradient of the true value function, semi-gradient
methods use an estimate of the gradient. This simplification allows for more practical
implementations and is often used with function approximators like neural networks.
3. Repeat: Steps 1-2 are repeated as the agent interacts with the environment. Over time,
the value function or policy is refined to better approximate the true values.

 An example of a Semi-gradient TD(0) algorithm is SARSA (State-Action-Reward-State-

Action), which is an on-policy TD learning algorithm. Semi-gradient methods are
commonly used in situations where using true gradients is impractical or computationally
expensive, and they provide a compromise between efficiency and simplicity.

Eligibility trace for function approximation

 An eligibility trace is a technique that improves the efficiency of updating the value
function. It addresses the issue of credit assignment, which refers to the problem of
determining which states and actions contributed to the observed reward.
 Without eligibility traces, the value function is updated only for the state visited at the
time the reward is received. This can be inefficient for complex tasks with long delays
between actions and rewards.
 Eligibility traces address this issue by keeping track of the states visited recently and
gradually reducing their contribution to the value function update. This allows the value
function to incorporate the influence of past states on the current reward, even if they are
not directly connected to the reward in the immediate next state.
Here's how eligibility trace works:
 When a state is visited, its corresponding eligibility trace is activated or increased.
 As the agent transitions through subsequent states, the eligibility trace of each visited
state decays gradually based on a decay factor.
 When a reward is received, the value function is updated for all states with non-zero
eligibility traces, weighted by their respective traces. The weight represents the
importance of each state in contributing to the current reward.
Benefits of using eligibility traces:
 Improved credit assignment: Enables accurate attribution of rewards to previously visited
states, leading to faster and more efficient learning.
 Stability: Reduces variance in the value function updates, leading to a more stable
learning process.
 Better generalization: Allows the value function to capture the long-term effects of
actions in complex environments.
Overall, eligibility traces offer a powerful tool for improving the efficiency and effectiveness
of RL algorithms with function approximation. However, careful consideration of their
benefits and drawbacks, as well as proper selection and parameter tuning, are crucial for
successful implementation.

After states
In reinforcement learning (RL), "after states" refer to a concept where the state
representation used for learning and decision-making is not the current state but rather the
state that occurs after taking a specific action. This concept is more common in certain types
of RL problems, such as games or board games, where the state representation after an action
can be more informative for learning and decision-making.
1. State Representation: In a typical RL problem, the agent makes decisions based on the
current state of the environment. The state is a representation of the relevant information
about the system at a particular point in time.
2. After states: After states are representations of the state that occur after the agent takes a
specific action in the current state. Instead of representing the current state, the learning
algorithm considers the state that results from the action.
3. Application in Games: After states are commonly used in game-playing scenarios,
particularly in board games like chess or checkers. In these games, the state after an
action often provides a more informative representation for learning and decision-making
than the current state.
4. Challenges: The use of after states is not always applicable or beneficial. In certain RL
problems, the current state might be more informative, and using after states could lead to
a loss of valuable information.
The definition and representation of afterstates depend on the specific problem and the
nature of the environment. Designing an effective afterstate representation requires
careful consideration of the problem's dynamics.

Control with function approximation

Control with function approximation refers to the use of function approximators to represent
the value function or policy, instead of using a table. This is essential for handling large or
continuous state and action spaces that are impractical or impossible to represent with a
tabular approach.
Function approximation is used to represent value functions or policies in situations where it
is impractical to store or compute values for all possible states or state-action pairs. Common
function approximators include linear models, non-linear models like neural networks, and
other function approximation techniques.
Challenges of Tabular Approaches
 Curse of dimensionality: As the size of the state and action spaces increases, the number
of table entries grows exponentially, making it computationally expensive and impractical
to store and update.
 Generalization: Learning from specific state-action pairs might not generalize well to
similar but unseen situations.
 Function Approximation:
o Introduces a parameterized function to approximate the value function (V) or
policy (π).
o This function can be based on various techniques, such as:

Control with Function Approximation:

 Policy gradient methods: Policy gradient methods are a class of control algorithms that
directly parameterize and update the policy of the agent. These methods use gradients to
adjust the policy parameters in the direction that increases the expected cumulative
reward.
 Actor-critic methods: Actor-critic methods combine elements of policy gradient
methods and value-based methods. The actor (policy) is updated using policy gradients,
and the critic (value function) is updated to evaluate the actions taken by the policy. Both
the actor and critic can be approximated using function approximation.
 Q-Learning and SARSA with Function Approximation: Q-learning and SARSA are
popular control algorithms in RL. When using function approximation, these algorithms
estimate the action-value function (Q-function) and update the parameters to improve the
estimates. For example, in deep Q-networks (DQN), a neural network is used to
approximate the Q-function.

Least squares
 The method of least squares plays a significant role in various reinforcement learning
(RL) algorithms for estimating value functions and policies. It provides a powerful tool
 for fitting a function to a set of data points, which helps the agent learn from experience
and improve its performance.
 Specifically, least squares methods can be applied to estimate the parameters of a value
function or policy when the state or action spaces are too large to be explicitly
represented.
Problem setting:
 State space (S): Set of all possible states the environment can be in.
 Action space (A): Set of all possible actions the agent can take.
 Reward function (R): Defines the reward received by the agent for taking an action in a
given state.
 Value function (V): Represents the expected future reward of being in a state and
following the policy.
 Policy (π): Defines the probability of the agent taking an action in a given state.
Least squares application:
 Data generation: The agent interacts with the environment and collects data, consisting of
state-action pairs and corresponding rewards.
 Function approximation: A function approximator (e.g., linear function, neural network)
is used to represent the value function or policy.
 Least squares formulation: The objective is to minimize the squared difference between
the predicted values (by the function approximator) and the actual rewards received by
the agent.
Types of least squares in RL:
 Least-squares temporal difference (LSTD): Estimates the value function based on TD
errors, which represent the difference between the predicted and actual rewards.
 Least-squares policy iteration (LSPI): Combines least squares with policy iteration to find
an optimal policy.
 Least-squares Q-learning (LSTD-Q): A variant of LSTD that estimates the Q-
function, which represents the expected future reward of taking a specific action in a
given state.
Challenges of using least squares in RL:
 Sensitivity to data: Relies heavily on the quality and quantity of data for accurate
estimation.
 Overfitting: Can overfit to the training data, leading to poor performance in unseen
situations.
 Computational cost

Experience replay in deep Q-Networks.

 Experience replay is a critical technique used in Deep Q-Networks (DQNs) to improve
training efficiency and stability. It addresses the issue of temporal dependency in the
 training data, where consecutive samples are highly correlated, leading to biased and
unstable learning.
Deep Q-Network (DQN):
 DQN is a popular algorithm for training a Q-learning agent that utilizes a neural network
to approximate the Q-function, which represents the expected cumulative reward for
taking an action in a given state. The DQN algorithm involves interacting with the
environment, collecting experiences, and updating the Q-network to improve the action-
value estimates.
The Problem:
 In standard Q-learning, the agent updates its Q-function based on the most recent
experiences.
 This can lead to overfitting and instability, as the agent prioritizes recent rewards and may
not learn effectively from older experiences.
 Additionally, using correlated data can inflate the variance of the updates, further
hindering learning.
Experience Replay Solution:
 Stores the agent's experiences in a replay memory, consisting of tuples of state, action,
reward, next state, and terminal flag.
 During training, the agent samples mini-batches from the replay memory to update its Q-
function.
 This has several benefits:
o Reduces temporal dependence: By decoupling the training data from the order in
which it was experienced, the agent learns more robustly.
o Improves data efficiency: The same experience can be used for multiple
updates, making better use of the collected data.
o Increases training stability: By introducing randomness through sampling, the
updates become less prone to noise and overfitting.
 Overall, experience replay is a fundamental technique that significantly improves the
performance and stability of Deep Q-Networks and other RL algorithms with function
approximation. It enables efficient learning, reduces bias, and promotes exploration,
leading to better generalization and performance in real-world tasks.