QUESTION BANK-UNIT V

Problem Solving
41 Design an RL agent to navigate a grid world using Fitted Q-learning with function
approximation.
To design an RL agent to navigate a grid world using Fitted Q-learning with function
approximation, you can follow these steps:
Step 1: Initialize Parameters
Initialize the parameters (weights and biases) of the Q-function approximation model. For
simplicity, we'll use a table to represent the Q-function.
Step 2: Interaction with the Environment
The agent interacts with the grid world environment, moving from one state to another and
taking actions according to its current policy.
Step 3: Observe State and Take Action
At each time step t, the agent observes the current state s_t (a grid cell in the grid world). It
selects an action a_t (e.g., up, down, left, right) based on its current Q-function
approximation.
Step 4: Receive Reward and Next State
After taking action a_t in state s_t, the agent receives a reward r_t based on the environment's
rules (e.g., +10 for reaching the goal, -1 for hitting an obstacle).
The agent transitions to the next state s_{t+1} based on the action a_t and the environment
dynamics.
Step 5: TD Target Calculation
Calculate the Temporal Difference (TD) target, which represents the target Q-value the Q-
function approximation should aim to approximate.
In Fitted Q-learning, the TD target is the observed reward plus the estimated maximum Q-
value of the next state-action pairs using the Q-function approximation: TD_target = r_t + γ
* max_a Q(s_{t+1}, a)
Step 6: Collect Data for Q-function Approximation
Collect a dataset (D) of state-action pairs (s, a) and their corresponding TD targets
(TD_target) over multiple time steps or episodes.
Step 7: Fitted Q-function Update
Use the collected dataset (D) to update the Q-function approximation model (table) by fitting
it to the TD targets. For each state-action pair (s, a) in the dataset, update the Q-Value in the
table to minimize the Mean Squared Error (MSE) between the predicted Qvalue and the TD
target: Q(s, a) = Q(s, a) + α * (TD_target - Q(s, a))
where α is the learning rate, controlling the step size of the update.
Step 8: Update Parameters
As we are using a table as the Q-function approximation model, there are no parameters
(weights) to update.
Step 9: Repeat
Repeat Steps 2 to 8 for multiple time steps or episodes, allowing the agent to learn and
update the Q-function approximation based on its interactions with the environment.
Step 10: Convergence and Evaluation
 Monitor the performance of the Fitted Q-learning algorithm and the convergence of the Q-
function approximation.
Evaluate the learned Q-function or the corresponding policy on test scenarios to assess
the agent's performance in the grid world environment.
In this example, Fitted Q-learning would involve updating the Q-values in the table based
on the observed rewards and the estimated maximum Q-values of the next states.
The agent would continue exploring the grid world environment, gradually improving its
Qfunction approximation to make better decisions and navigate to the goal state while
avoiding obstacles efficiently.
Here is an example implementation of Fitted Q-learning with function approximation
in Python:
import numpy as np
from sklearn.linear_model import SGDRegressor
class FittedQAgent:
def __init__(self, env, n_episodes=1000, gamma=0.99, epsilon=0.1, alpha=0.01):
self.env = env
self.n_episodes = n_episodes
self.gamma = gamma
self.epsilon = epsilon
self.alpha = alpha
self.Q = SGDRegressor(learning_rate='constant')
self.Q.partial_fit(np.zeros((env.observation_space.n,env.action_space.n)),
np.zeros((env.observation_space.n,)))

def fit(self):
for i in range(self.n_episodes):
s = self.env.reset()
done = False
while not done:
if np.random.rand() < self.epsilon:
a = self.env.action_space.sample()
else:
a = np.argmax(self.Q.predict([s])[0])
s_prime, r, done, _ = self.env.step(a)
self.Q.partial_fit([s], [self.Q.predict([s])[0] + self.alpha * (r + self.gamma *
np.max(self.Q.predict([s_prime])[0])-self.Q.predict([s])[0][a])*
np.eye(self.env.action_space.n)[a]])
s = s_prime

def predict(self, s):

return np.argmax(self.Q.predict([s])[0])
This implementation uses stochastic gradient descent to fit a linear model to the Q-function.
The fit method trains the agent using Fitted Q-learning with function approximation, and the
predict method returns the action with the highest Q-value for a given state.
42 Implement the Deep Q-Network (DQN) algorithm to solve a continuous action space
problem.
The Deep Q-Network (DQN) algorithm is a variant of Q-learning that can be used to solve
problems with continuous state spaces. However, it is not directly applicable to problems
with continuous action spaces. One approach to extend DQN to continuous action spaces is
to use an actor-critic architecture, such as the Deep Deterministic Policy Gradient (DDPG)
algorithm.
Introduction:
⮚ It is a powerful model-free reinforcement learning (RL) algorithm that combines Q-
learning with deep neural networks to handle high-dimensional state spaces
efficiently.
⮚ It was introduced by Mnih et al. in their paper "Playing Atari with Deep
Reinforcement Learning" in 2013.
⮚ DQN allows RL agents to learn directly from raw pixel inputs, making it suitable for
complex tasks in environments with large state spaces.

Steps:
Step 1: Initialize Deep Q-Network
Initialize the Deep Q-Network architecture, typically using convolutional layers for image
processing followed by fully connected layers to approximate the Q-function.
Step 2: Initialize Target Network
Create a target network with the same architecture as the Deep Q-Network.
This target network is used to calculate the TD target during updates and remains fixed for
a certain number of steps before being updated again.
Step 3: Initialize Replay Memory
Create a replay memory buffer to store experiences of the agent. Each experience is
represented as a tuple (state, action, reward, next_state, done).
Step 4: Set Hyper parameters
Set hyper parameters such as the learning rate (alpha), discount factor (gamma), exploration
rate (epsilon), batch size, and the number of episodes.
Step 5: Interaction with the Environment
The agent interacts with the grid world environment, moving from one grid cell to another
and taking actions based on its current policy.
Step 6: Observe State and Take Action
At each time step t, the agent observes the current state s_t (its current position in the Grid
world) and selects an action a_t using an epsilon-greedy exploration strategy.
Step 7: Receive Reward and Next State
After taking action a_t in state s_t, the agent receives a reward r_t from the environment
based on the following rules:
r_t = -1 for each step taken (penalty for time)
r_t = +10 if the agent reaches the goal state (G)
r_t = -10 if the agent hits an obstacle (X)
The agent transitions to the next state s_{t+1} based on the action a_t and the environment
dynamics.
Step 8: Store Experience
Store the experience tuple (s_t, a_t, r_t, s_{t+1}, done) in the replay memory buffer.
Step 9: Sample Mini-Batch from Replay Memory
Randomly sample a mini-batch of experiences (state, action, reward, next_state, done)
from the replay memory buffer.
Step 10: Calculate TD Targets
For each experience in the mini-batch, calculate the Temporal Difference (TD) target using
the target network and the Bellman equation: TD_target = r_t + gamma * max_a
Q_target(s_{t+1}, a)
Step 11: Update Deep Q-Network
Update the Deep Q-Network using the mini-batch of experiences and the TD targets.
Perform gradient descent on the Mean Squared Error (MSE) loss between the predicted.
Q-values and the TD targets to adjust the network's weights.
In this example, the DQN algorithm will learn to navigate the grid world, finding the shortest
path to the goal state while avoiding obstacles efficiently.
The replay memory and target network help stabilize the learning process, enabling the agent
to learn from past experiences and achieve better convergence in the RL task.
Here is an example implementation of DDPG in Python using the TensorFlow library:
import tensorflow as tf
import numpy as np

class DDPGAgent:
def __init__(self, env, n_episodes=1000, gamma=0.99, tau=0.001, buffer_size=100000,
batch_size=64):
self.env = env
self.n_episodes = n_episodes
self.gamma = gamma
self.tau = tau
self.buffer_size = buffer_size
self.batch_size = batch_size
self.memory = []
self.actor = self.build_actor()
self.critic = self.build_critic()
self.target_actor = self.build_actor()
self.target_critic = self.build_critic()
self.target_actor.set_weights(self.actor.get_weights())
self.target_critic.set_weights(self.critic.get_weights())
self.actor_optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
self.critic_optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

def build_actor(self):
inputs = tf.keras.layers.Input(shape=self.env.observation_space.shape)
x = tf.keras.layers.Dense(256, activation='relu')(inputs)
x = tf.keras.layers.Dense(256, activation='relu')(x)
outputs = tf.keras.layers.Dense(self.env.action_space.shape[0], activation='tanh')(x)
outputs = tf.keras.layers.Lambda(lambda x: x * self.env.action_space.high)(outputs)
return tf.keras.Model(inputs=inputs, outputs=outputs)

def build_critic(self):
 state_inputs = tf.keras.layers.Input(shape=self.env.observation_space.shape)
state_x = tf.keras.layers.Dense(16, activation='relu')(state_inputs)
action_inputs = tf.keras.layers.Input(shape=self.env.action_space.shape)
action_x = tf.keras.layers.Dense(16, activation='relu')(action_inputs)
x = tf.keras.layers.Concatenate()([state_x, action_x])
x = tf.keras.layers.Dense(256, activation='relu')(x)
x = tf.keras.layers.Dense(256, activation='relu')(x)
outputs = tf.keras.layers.Dense(1, activation='linear')(x)
return tf.keras.Model(inputs=[state_inputs, action_inputs], outputs=outputs)

def remember(self, state, action, reward, next_state, done):

self.memory.append((state, action, reward, next_state, done))
if len(self.memory) > self.buffer_size:
self.memory.pop(0)

def act(self, state):

return self.actor.predict(np.array([state]))[0]

def train(self):
for i in range(self.n_episodes):
state = self.env.reset()
done = False
while not done:
action = self.act(state)
next_state, reward, done, _ = self.env.step(action)
self.remember(state, action, reward, next_state, done)
self.update()
state = next_state

def update(self):
if len(self.memory) < self.batch_size:
return
minibatch = np.array(self.memory)[np.random.choice(len(self.memory),
self.batch_size, replace=False)]
states = np.array([m[0] for m in minibatch])
actions = np.array([m[1] for m in minibatch])
rewards = np.array([m[2] for m in minibatch])
next_states = np.array([m[3] for m in minibatch])
dones = np.array([m[4] for m in minibatch])
target_actions = self.target_actor.predict(next_states)
target_q_values = self.target_critic.predict([next_states, target_actions])
y = rewards + self.gamma * target_q_values * (1 - dones)
with tf.GradientTape() as tape:
q_values = self.critic([states, actions])
critic_loss = tf.keras.losses.MSE(y, q_values)
critic_grads = tape.gradient(critic_loss, self.critic.trainable_variables)
 self.critic_optimizer.apply_gradients(zip(critic_grads, self.critic.trainable_variables))
with tf.GradientTape
43 Develop a Policy Gradient algorithm to train a robotic arm to reach a target in a simulated
environment.
To develop a Policy Gradient algorithm to train a robotic arm to reach a target in a simulated
environment, you can follow these steps:
Introduction:
Policy Gradient algorithms are a family of model-free reinforcement learning (RL) methods
that directly optimize the policy of an agent to find the best actions to take in different states.
Unlike Q-learning, which approximates the value function and then derives the policy,
Policy Gradient methods focus on directly learning the policy function and updating it to
maximize the expected cumulative reward.
Steps:
Step 1: Initialize Policy Network:
Initialize a parameterized policy network, such as a neural network, with random weights.
This network takes the state as input and outputs a probability distribution over actions.
Step 2:Interaction with the Environment:
The agent interacts with the environment and takes actions based on its current policy.
Step 3. Observe State and Sample Action:
At each time step t, the agent observes the current state s_t and samples an action a_t from
the policy network's output probability distribution.
Step 4: Receive Reward and Next State:
After taking action a_t in state s_t, the agent receives a reward r_t from the environment
and transitions to the next state s_{t+1}.
Step 5: Calculate Policy Gradient:
Calculate the gradient of the policy with respect to its parameters, indicating how much the
policy should change to improve the expected cumulative reward.
Step 6: Update Policy Parameters:
Use the policy gradient to update the policy network's parameters in the direction that
improves the expected cumulative reward.
This can be done through gradient ascent: θ = θ + α * ∇θ J(θ)
where θ represents the policy network's parameters, J(θ) is the objective function to
maximize (e.g., expected cumulative reward), α is the learning rate, and ∇θ J(θ) is the policy
gradient.
Step 7: Repeat:
Repeat Steps 2 to 6 for multiple time steps or episodes, allowing the agent to learn and update
the policy based on its interactions with the environment.
Step 8: Convergence and Evaluation:
Monitor the performance of the Policy Gradient algorithm and evaluate the learned policy
on test scenarios to assess the agent's performance in the environment.

44 Analyze the impact of using different function approximation architectures in Fitted

Qlearning.
⮚ Fitted Qlearning is a technique used to represent value functions or policies using
parameterized functions, rather than explicit tabular representations.
 ⮚ In RL problems with large or continuous state spaces, using tabular representations
becomes infeasible due to the exponential growth in memory requirements.
⮚ Function Approximation addresses this issue by using a parameterized function to
estimate the value function or policy, allowing the RL agent to generalize its
knowledge from limited experiences to unseen states more efficiently.
⮚ The function approximation is typically implemented using machine learning
models, such as neural networks, decision trees, or linear regression models.
⮚ These models take the state or state-action pairs as input and output the estimated
value function or policy.
⮚ There has been a lot of research on the impact of using different function
approximation architectures in Fitted Q-learning.
⮚ One study by Santos et al1 analysed the interplay between the data distribution and
Q-learning-based algorithms with function approximation.
⮚ They provided a unified theoretical and empirical analysis as to how different
properties of the data distribution influence the performance of Q-learning-based
algorithms.
⮚ They found that high entropy data distributions are well-suited for learning in an
offline manner, and a certain degree of data diversity (data coverage) and data quality
(closeness to optimal policy) are jointly desirable for offline learning.
⮚ Another study by Wang et al2 compared the performance of different function
approximation architectures for Fitted Q-learning on a continuous control task.
⮚ They found that a deep neural network with rectified linear units (ReLU) and batch
normalization outperformed other architectures such as linear regression, radial basis
functions, and multilayer perceptrons.
⮚ These studies suggest that the choice of function approximation architecture can have
a significant impact on the performance of Fitted Q-learning.
⮚ However, the optimal architecture may depend on the specific problem and data
distribution, and further research is needed to determine the best approach for a given
task.

45 Assess the effectiveness of using Eligibility Traces for updating Q-values in a dynamic
environment.
⮚ Eligibility traces are a technique used in reinforcement In Reinforcement Learning
(RL), Eligibility Traces are a mechanism used to update the value function or policy
more efficiently, especially in Temporal Difference (TD) methods.
⮚ They help in handling the credit assignment problem by giving credit to past states
and actions that contributed to the observed rewards and encouraging learning from
both recent and distant experiences.
⮚ The main idea behind Eligibility Traces is to maintain a trace of the states and actions
visited during the agent's interaction with the environment.
⮚ These traces act as a record of "eligibility" for each state-action pair, indicating how
much they contributed to the observed rewards.
⮚ There are different types of Eligibility Traces, such as Accumulating Traces,
Replacing Traces, and Dutch Traces, each with its specific characteristics.
⮚ In Accumulating Traces, a trace is accumulated over time whenever a state-action
pair is visited.
⮚ The trace value increases with each visit, decaying at a specific rate over time.
⮚ When a TD update is performed, the accumulated trace is used to update the value
function or policy.
⮚ The update for a state-action value (Q-value) using Accumulating Traces can be
expressed as:

⮚ Eligibility traces are a technique used in reinforcement learning to update Q-values

in a dynamic environment.
⮚ They are used to keep track of the history of state-action pairs that have been visited
and to update the Q-values of these pairs based on their frequency of occurrence.
⮚ The effectiveness of using eligibility traces for updating Q-values in a dynamic
environment depends on several factors, such as the learning rate, the discount factor,
and the trace decay parameter.
⮚ The eligibility traces can be used to speed up learning in dynamic environments by
propagating knowledge back over time-steps in a single update.
⮚ This can be useful in situations where the environment is constantly changing and
the agent needs to adapt quickly to new conditions.
 However, eligibility traces can also introduce additional complexity into the learning process
and may require more computational resources than other methods. Eligibility traces can be
difficult to implement and may not always lead to better performance than other methods.

In conclusion, the effectiveness of using eligibility traces for updating Q-values in a dynamic
environment depends on several factors and may not always lead to better performance than
other methods. However, they can be useful in certain situations and can help speed up
learning in dynamic environments.
46 Evaluate the performance of Deep Q-Network (DQN) compared to Fitted Q-learning in a
grid world scenario with a large state space.
⮚ In a grid world scenario with a large state space, Deep Q-Network (DQN) and Fitted
Q-learning are two popular reinforcement learning algorithms that can be used to
learn an optimal policy.
⮚ DQN is a variant of Q-learning that uses a deep neural network to represent the Q-
function, while Fitted Q-learning uses a function approximator to estimate the Q-
function.
⮚ DQN has been shown to outperform Fitted Q-learning in several benchmark tasks,
including Atari games.
⮚ DQN is able to learn a good approximation of the Q-function even in high-
dimensional state spaces, which makes it well-suited for grid world scenarios with a
large state space.
⮚ However, DQN can be computationally expensive and may require a large amount
of memory to store the neural network weights.
⮚ Fitted Q-learning, on the other hand, is computationally less expensive and requires
less memory, but may not perform as well as DQN in high-dimensional state spaces.
⮚ In conclusion, both DQN and Fitted Q-learning are viable options for learning an
optimal policy in a grid world scenario with a large state space.
⮚ DQN may be a better choice if computational resources are not a constraint and high
performance is desired, while Fitted Q-learning may be a better choice if
computational resources are limited and a simpler algorithm is preferred.

47 Devise a novel function approximation method for handling continuous state spaces in RL.
Function approximation methods are used in reinforcement learning to estimate the value
function of a state.
In continuous state spaces, function approximation methods are often employed instead of
finely discretizing the state space to avoid the explosion of computational complexities.
One such method is Gaussian-based Non-linear Function Approximation (GBNLFA).
1.In GBNLFA, each discrete action is represented by a Gaussian distribution with two
standard parameters (mu and sigma)
2. Another method is Continuous-time Value Function Approximation in Reproducing
Kernel Hilbert Space (RKHS)
3. This method uses function approximators such as Gaussian networks with a fixed number
of basic functions 3.
 However, devising a novel function approximation method for handling continuous state
spaces in RL is an active area of research.
One such method is Deep Deterministic Policy Gradient (DDPG)
1. DDPG is an actor-critic algorithm that uses deep neural networks to represent the
policy and the Q-function .
2. It has been shown to be effective in handling continuous state spaces in RL
3. Another method is the use of autoencoders to learn a compressed representation of
the state space.
4. This compressed representation can then be used as input to a function
approximator.
⮚ In conclusion, there are several function approximation methods that can be used to
handle continuous state spaces in RL.
⮚ Gaussian-based Non-linear Function Approximation and Continuous-time Value
Function Approximation in Reproducing Kernel Hilbert Space are two such
methods.
⮚ However, devising a novel function approximation method for handling continuous
state spaces in RL is an active area of research, and there are several promising
methods that are being developed.

48 a. Compare the advantages and disadvantages of Eligibility Traces and Function

Approximation in RL.
 b. How does Fitted Q-learning leverage the concept of experience replay?
⮚ Fitted Q-learning is a reinforcement learning algorithm that uses function
approximation to estimate the Q-function. Experience replay is a technique
used in reinforcement learning to improve the efficiency of learning by reusing
past experiences.
⮚ In Fitted Q-learning, experience replay is used to store past experiences in a
buffer and to randomly sample a batch of experiences from the buffer to update
the Q-function.
⮚ This helps to reduce the correlation between consecutive updates and to
improve the stability of the learning process.
⮚ During experience replay, the agent’s experiences are stored in a buffer and are
reused to update the Q-function.
⮚ The buffer is a fixed-size queue that stores the most recent experiences of the
agent. During learning, a batch of experiences is randomly sampled from the
buffer and is used to update the Q-function.
⮚ This helps to break the temporal correlations between consecutive updates and
to improve the stability of the learning process.
⮚ In conclusion, Fitted Q-learning leverages the concept of experience replay to
improve the efficiency and stability of the learning process.
⮚ By reusing past experiences, Fitted Q-learning can reduce the correlation
between consecutive updates and can improve the stability of the learning
process.

49 a. What are the main advantages and limitations of Fitted Q-learning compared to DQN?
⮚ Fitted Q-learning and Deep Q-Network (DQN) are two popular reinforcement
learning algorithms that can be used to learn an optimal policy.
⮚ Fitted Q-learning is a function approximation method that estimates the Q-
function using a function approximator, while DQN is a variant of Q-learning
that uses a deep neural network to represent the Q-function.
⮚ DQN has been shown to outperform Fitted Q-learning in several benchmark
tasks, including Atari games.
⮚ DQN is able to learn a good approximation of the Q-function even in high-
dimensional state spaces, which makes it well-suited for grid world scenarios
with a large state space.
⮚ Fitted Q-learning, on the other hand, is computationally less expensive and
requires less memory than DQN, but may not perform as well as DQN in high-
dimensional state spaces.
⮚ Fitted Q-learning is also more interpretable than DQN, as it uses a linear
function approximator that can be easily visualized and understood.
 ⮚ In conclusion, both Fitted Q-learning and DQN are viable options for learning
an optimal policy.
⮚ DQN may be a better choice if computational resources are not a constraint and
high performance is desired, while Fitted Q-learning may be a better choice if
computational resources are limited and a simpler algorithm is preferred.
⮚ Fitted Q-learning is also more interpretable than DQN, which can be useful in
certain situations.

b. In which scenarios would you prefer to use Fitted Q-learning over DQN and vice
versa?
⮚ Fitted Q-learning and Deep Q-Network (DQN) are two popular reinforcement
learning algorithms that can be used to learn an optimal policy.
⮚ Here are some scenarios and preferences to use Fitted Q-learning over DQN
and vice versa:

Scenarios where Fitted Q-learning is preferred over DQN:

⮚ When the state space is small and discrete, Fitted Q-learning can be a better
choice than DQN as it is computationally less expensive and requires less
memory.
⮚ When the goal is to learn an interpretable model, Fitted Q-learning can be a
better choice than DQN as it uses a linear function approximator that can be
easily visualized and understood.

Scenarios where DQN is preferred over Fitted Q-learning:

⮚ When the state space is large and continuous, DQN can be a better choice than
Fitted Q-learning as it can learn a good approximation of the Q-function even
in high-dimensional state spaces.
⮚ When the goal is to achieve high performance, DQN can be a better choice than
Fitted Q-learning as it has been shown to outperform Fitted Q-learning in
several benchmark tasks, including Atari games.

In conclusion, the choice between Fitted Q-learning and DQN depends on several
factors such as the size and nature of the state space, the computational resources
available, and the desired level of performance and interpretability.
50 How do Policy Gradient algorithms and Least Squares Methods handle the exploration-
exploitation trade-off differently?

Policy Gradient algorithms and Least Squares Methods are two popular reinforcement
learning algorithms that handle the exploration-exploitation trade-off differently.

Policy Gradient algorithms use a stochastic policy to explore the state-action space and
update the policy parameters based on the gradient of the expected reward. The policy is
updated in the direction of the gradient of the expected reward, which encourages the policy
to take actions that lead to higher rewards. This approach can be effective in high-
dimensional or continuous action spaces, where it is difficult to enumerate all possible
actions.
Least Squares Methods, on the other hand, use a value function to estimate the expected
reward of each state-action pair. The value function is updated using a least squares
regression algorithm, which minimizes the difference between the predicted value and the
actual value. This approach can be effective in low-dimensional state-action spaces, where
it is possible to enumerate all possible actions.
In conclusion, Policy Gradient algorithms and Least Squares Methods handle the
exploration-exploitation trade-off differently. Policy Gradient algorithms use a stochastic
policy to explore the state-action space, while Least Squares Methods use a value function
to estimate the expected reward of each state-action pair.