ML MODULE – 5

1. What is Reinforcement Learning?

Reinforcement Learning (RL) is a type of machine learning where an autonomous agent learns to make
decisions by taking actions in an environment to achieve its goals. The process involves the agent
sensing its environment through sensors and performing actions that alter the state of the environment.
The core objective of RL is to learn a control policy, or strategy, that dictates the best actions to take in
various states to maximize a cumulative reward over time.

2. Explain the Q function and Q Learning Algorithm.

Q Function:
The Q function, Q(s,a) is a fundamental concept in reinforcement learning. It represents the expected
future rewards an agent can achieve, starting from state s and taking action a, and then following the
optimal policy thereafter. Formally, the Q function can be de ined as:
Q(s,a)=r(s,a)+γmaxa′Q(s′,a′)
Where:
 r(s,a) is the immediate reward received after taking action a in state s.
 γ is the discount factor, which ranges between 0 and 1, and determines the importance of future
rewards.
 s′ is the next state resulting from action a.
 maxa′Q(s′,a′) represents the maximum expected reward achievable from the next state s′ by
following the optimal policy.

Q Learning Algorithm:
The Q Learning algorithm is a model-free reinforcement learning technique used to learn the Q function.
It enables an agent to learn the optimal policy for an arbitrary environment by iteratively improving its
estimates of the Q values.

 Q learning algorithm assuming deterministic rewards and actions. The discount factor γ may be any
constant such that 0 ≤ γ < 1
 𝑄̂^ to refer to the learner's estimate, or hypothesis, of the actual Q function
The Q Learning algorithm is guaranteed to converge to the optimal Q function under certain conditions,
such as the agent visiting every possible state-action pair in initely often and using a decreasing
learning rate
3. Describe K-nearest Neighbor learning Algorithm for continuous valued target function.

4. Discuss the major drawbacks of K-nearest Neighbor learning Algorithm and how it can be corrected.
Drawbacks:
1. High Computational Cost: KNN requires the computation of the distance between the query point
and all points in the training dataset. This becomes computationally expensive, especially with large
datasets and higher dimensionality.
2. Storage Requirement: Since KNN stores all the training data, it requires signi icant storage space,
especially with large datasets.
3. Sensitivity to Irrelevant Features and Data Scaling: KNN treats all features equally when computing
distances, which can be problematic if some features are irrelevant or have different scales. This can
lead to poor performance if the irrelevant features overshadow the relevant ones.
4. Curse of Dimensionality: As the number of dimensions increases, the volume of the space increases
exponentially, causing the density of points to decrease. This makes it harder to ind nearest
neighbors that are actually close, as most points become equidistant in high-dimensional spaces.

Corrections:
1. Dimensionality Reduction: Techniques like Principal Component Analysis (PCA) or feature selection
can be used to reduce the number of dimensions, thereby mitigating the curse of dimensionality and
reducing computational costs.
2. Distance Weighting: Implementing distance-weighted KNN, where closer neighbors have a larger
in luence on the decision, can improve accuracy, especially when dealing with varying densities of
data points.
3. Data Normalization: Normalizing the data ensures that each feature contributes equally to the
distance computation, preventing features with larger scales from dominating the distance metric.
4. Ef icient Data Structures: Using data structures like KD-trees or Ball trees can help in ef iciently
organizing the training data and reducing the time complexity of nearest neighbor searches

5. Explain Q learning algorithm assuming deterministic rewards and actions.

Q-learning is a model-free reinforcement learning algorithm used to ind the optimal policy for an agent
in a given environment. The algorithm aims to learn the Q-function, which estimates the value of taking
a particular action in a given state, considering both the immediate reward and the expected future
rewards.

Q-Function and Deterministic Rewards and Actions: In environments with deterministic rewards and
actions, the outcomes of actions are predictable and consistent. This means that for a given state-action
 pair (s,a), the resulting state and the reward are always the same. The Q-function Q(s,a) is de ined as the
expected cumulative reward for taking action a in state s and following the optimal policy thereafter.
The Q-learning update rule, in the deterministic case, can be expressed as:
Q(s,a)=r(s,a)+γmaxa′Q(s′,a′)
where:
 r(s,a) is the immediate reward received after taking action a in state s,
 γ is the discount factor (with 0≤γ<1), which accounts for the present value of future rewards,
 s′ is the state resulting from taking action a in state s,
 maxa′Q(s′,a′) is the maximum Q-value for the subsequent state s′ over all possible actions a′.
The algorithm iteratively updates the Q-values based on the experiences of the agent, gradually
converging to the true Q-values under the assumptions of deterministic rewards and actions. This
allows the agent to learn the optimal policy, which is to choose the action with the highest Q-value in
each state.

Convergence:
In a deterministic setting, Q-learning will converge to the true Q-values as long as all state-action pairs
are visited in initely often and the learning rate is suf iciently small. This guarantees that the agent will
learn the optimal policy for maximizing the cumulative reward .

6. De ine the following terms with respect to K-nearest Neighbor Learning:

a. Regression : Regression means approximating a real-valued target function
b. Residual : Residual is the error 𝑓̂(x) - f (x) in approximating the target function.
c. Kernel Function :  Kernel function is the function of distance that is used to determine the
weight of each training example. In other words, the kernel function is the function K such that
wi = K(d(xi, xq))

7. Explain the K-nearest neighbor algorithm for approximating a discrete-valued function f:Hn→Vf: H^n
\to Vf:Hn→V with pseudo code.
The K-Nearest Neighbor (K-NN) algorithm is a simple, non-parametric method used for classi ication
and regression. In the case of a discrete-valued function, K-NN classi ies an instance based on the
majority label of its k-nearest neighbors.
Pseudo Code for K-NN:
Input:
- Training set D with n instances and corresponding labels
- Query instance xq
- Number of neighbors k

Output:
- Predicted label for xq

Algorithm:
1. Calculate the distance between xq and all instances in D.
2. Sort the distances in ascending order.
3. Select the k instances in D that are closest to xq.
4. Determine the most frequent label among the k nearest neighbors.
5. Assign this label to xq.

Return the label as the output.

 Explanation:
 Distance Calculation: The distance between the query instance xq and each training instance is
typically calculated using a distance metric like Euclidean distance.
 Sorting and Selection: The training instances are sorted based on their distance to xq, and the closest
k instances are selected.
 Majority Voting: The most common label among the selected neighbors is assigned to the query
instance.
K-NN works well with discrete-valued target functions where the goal is to classify instances into one of
several discrete categories. The choice of k can signi icantly affect the performance of the algorithm,
with smaller values of k making the classi ier sensitive to noise in the training data .

8. Explain Locally Weighted Linear Regression.

Locally Weighted Linear Regression (LWLR) is a type of regression analysis that focuses on itting a
linear model to a localized subset of the data. The idea is to give more weight to the points that are
closer to the query point when performing the regression, hence the term "locally weighted."

Given a new query instance xq, the general approach in locally weighted regression is to construct an
approximation 𝑓̂that its the training examples in the neighborhood surrounding xq. This approximation
is then used to calculate the value 𝑓̂(xq), which is output as the estimated target value for the query
instance.
 Key Points:
1. Locality: The method focuses on a local region of the data around the query point.
2. Weighting: The contribution of each data point is weighted based on its proximity to the query
point.
3. Regression: The technique its a linear model to the weighted data, which can be more accurate than
a global model when the data exhibits local variability.

LWLR is particularly useful for cases where the underlying relationship between the variables is
complex and varies signi icantly across different regions of the input space.

9. Explain CADET System using Case-based reasoning.

The CADET system is an example of Case-Based Reasoning (CBR), which is a learning paradigm based
on lazy learning methods. In CBR, new query instances are classi ied by analyzing similar instances,
while largely ignoring instances that differ signi icantly from the query. Unlike other learning paradigms
that often rely on numerical data, CBR utilizes a rich symbolic representation of instances.

CADET System Overview:

 Purpose: The CADET system is designed to assist in the conceptual design of simple mechanical
devices, such as water faucets.
 Library: It utilizes a library containing approximately 75 previous designs and design fragments to
suggest conceptual designs that meet new design speci ications.
 Representation: Each instance in the CADET system is represented by describing both its structure
and its qualitative function. For example, an instance might represent a component like a water pipe
with details about its structure and function.
 Functionality: New design problems are presented by specifying the desired function, and the
CADET system retrieves the corresponding structure based on the function's description. The
function is often represented in terms of qualitative relationships, such as the correlation between
water low levels and temperatures at the inputs and outputs of a faucet.
In summary, the CADET system leverages CBR to provide solutions by drawing parallels between new
problems and previously encountered cases, thereby reusing and adapting existing knowledge to solve
new design challenges.

10. Explain the two key dif iculties that arise while estimating the Accuracy of Hypothesis.
Two major challenges arise when estimating the accuracy of a hypothesis in machine learning:
1. Bias in the Estimate:
o The estimated accuracy of a hypothesis can be biased if the data used to evaluate it is not
representative of the broader population of interest. This bias can occur due to sampling
errors or if the training and testing data are not independent and identically distributed. For
instance, using the same data for both training and testing can lead to overestimated
accuracy, as the hypothesis may over it the speci ic examples it has seen.
2. Variance in the Estimate:
 oThe accuracy estimate can vary signi icantly depending on the particular sample of data
used for testing. This variance arises because different samples may produce different
accuracy estimates, especially if the sample size is small. High variance can make it
challenging to get a reliable estimate of the hypothesis's true performance.
These issues underline the importance of using proper evaluation techniques, such as cross-
validation, and ensuring a diverse and representative sample of data when estimating the accuracy
of a hypothesis.

11. De ine the following terms:

a. Sample error
b. True error
c. Random Variable
d. Expected value
e. Variance
f. Standard Deviation
a. Sample Error:
 The error rate of a hypothesis on the training set or a speci ic sample of data. It measures how well the
hypothesis performs on the data it was trained or tested on, but does not necessarily re lect its
performance on unseen data.
b. True Error:
 The error rate of a hypothesis over the entire distribution of data, including both seen and unseen
examples. It represents the true performance of the hypothesis in the real world.
c. Random Variable:
 A variable whose values are outcomes of a random phenomenon. In the context of machine learning,
random variables can represent elements such as the value of a feature or the outcome of a
classi ication.

d. Expected Value:
 The average value of a random variable over many trials or occurrences. It provides a measure of the
central tendency of the variable's possible values.
e. Variance:
 A measure of the dispersion of a set of values around their mean. In machine learning, variance can
refer to the variability in the model's predictions.
f. Standard Deviation:
 The square root of the variance, providing a measure of the amount of variation or dispersion of a set of
values. It is commonly used to quantify the amount of variation in a dataset or the error of a hypothesis

12. Explain Binomial Distribution with an example.

Binomial Distribution
The Binomial distribution is a discrete probability distribution that describes the number of successes
in a ixed number of independent Bernoulli trials. Each trial has two possible outcomes: success or
failure. The distribution is characterized by two parameters:
 n: the number of trials
 p: the probability of success on a single trial
 The probability of observing exactly k successes in n independent trials is given by the Binomial
probability formula:
𝑃(𝑋 = 𝑘) = 𝑝 (1 − 𝑝)
where:
!
 is the binomial coef icient, also written as "n choose k," and is calculated as !( )!
 p is the probability of success on a single trial
 1−p is the probability of failure on a single trial
 k is the number of successes
 n is the total number of trials
Properties of Binomial Distribution
1. Discrete Distribution: The random variable X can only take integer values from 0 to n.
2. Mean and Variance:
o Mean (μ) = np
o Variance (σ2) = np(1−p)
o
Example
Suppose you lip a fair coin 10 times, and you want to ind the probability of getting exactly 6 heads.
Here, each coin lip is a Bernoulli trial with two possible outcomes (heads or tails), and the probability
of success (getting heads) is p=0.5. The number of trials n=10 and the number of successes k=6.
The probability of getting exactly 6 heads is given by:
10
𝑃(𝑋 = 6) = (0.5) (1 − 0.5)
6
Calculating the binomial coef icient and probabilities, we get:
10!
𝑃(𝑋 = 6) = (0.5) (0.5)
6! (10 − 6)!
𝑃(𝑋 = 6) = 210 × 0.015625 × 0.0625 = 0.205

So, the probability of getting exactly 6 heads in 10 coin lips is 0.205.

The Binomial distribution is widely used in situations where there are a ixed number of independent
trials, each with the same probability of success, such as quality control testing, clinical trials, and
survey sampling.

13. Explain Normal or Gaussian distribution with an example.

Normal or Gaussian Distribution
The Normal distribution, also known as the Gaussian distribution, is a continuous probability
distribution characterized by a symmetric bell-shaped curve. It is de ined by two parameters: the mean
(μ) and the standard deviation (σ). The mean determines the center of the distribution, and the
standard deviation determines the width or spread of the distribution.

Probability Density Function (PDF)

The probability density function (PDF) of the Normal distribution is given by the formula:
( )
𝑓(𝑥) = exp −

 x: The variable
 μ: The mean of the distribution
 σ: The standard deviation of the distribution
 exp: The exponential function
 π: Pi, approximately 3.14159
The curve of the Normal distribution is symmetric about the mean, with the highest point at the mean.
As you move away from the mean, the probability decreases exponentially.
Properties of the Normal Distribution
1. Symmetry: The distribution is symmetric about the mean.
2. Bell-shaped Curve: The shape of the distribution is a bell curve.
 3. Mean, Median, and Mode: In a perfectly normal distribution, the mean, median, and mode are all
equal.
4. 68-95-99.7 Rule: Approximately 68% of the data lies within one standard deviation of the mean,
95% within two standard deviations, and 99.7% within three standard deviations.

Example
Suppose the heights of a group of people are normally distributed with a mean height of 170 cm and a
standard deviation of 10 cm. We can represent this distribution as N(170,102).
The probability density function for this distribution would be:
1 (𝑥 − 170)
𝑓(𝑥) = exp −
√2π ⋅ 10 2 ⋅ 10

This function can be used to calculate the probability of a person's height falling within a certain range.
For instance, to ind the probability that a randomly selected person from this group is between 160 cm
and 180 cm tall, you would integrate the PDF over this interval.

The Normal distribution is widely used in statistics, inance, natural and social sciences because many
variables are naturally distributed in this pattern. It's often used to model measurement errors, physical
characteristics, and many other phenomena.

14. What are instance-based learning? Explain key features and disadvantages of these methods.
Instance-Based Learning
Instance-based learning is a type of supervised learning algorithm that relies on storing and using
speci ic instances of the training data to make predictions. Unlike other learning methods that abstract a
model from the training data (like neural networks or decision trees), instance-based learning
algorithms make predictions based on the similarity between new data points and the stored instances.

Key Features of Instance-Based Learning

1. Memory-Based: These algorithms store all or most of the training data, making them memory-
intensive. Each new query is compared with stored instances to make a prediction.
2. Lazy Learning: Instance-based methods are often called "lazy learners" because they do not
explicitly build a model during training. Instead, they wait until a prediction is needed and then
perform computations using the stored data.
3. Similarity Measure: The effectiveness of instance-based learning heavily depends on the choice of
similarity or distance measure. Common measures include Euclidean distance, Manhattan distance,
and Minkowski distance.
4. Local Approximation: Predictions are made based on a local approximation of the target function.
This can be done using techniques like the k-nearest neighbors (k-NN) algorithm, where the output
is computed based on the closest training examples.
5. Adaptability: Since these methods rely on speci ic instances, they can easily adapt to new data by
simply adding new instances to the dataset. There's no need to retrain a model from scratch.

Disadvantages of Instance-Based Learning

1. Computational Complexity: Making predictions can be computationally expensive, especially with
large datasets, as the algorithm might need to compare a new instance with many stored instances.
2. Storage Requirements: Since instance-based learning stores the training data, it requires a
signi icant amount of memory, especially for large datasets.
3. Over itting: These methods can be prone to over itting, as they can be in luenced by noise or
irrelevant features present in the training data.
4. Lack of Interpretability: Unlike model-based approaches that provide a clear structure (like decision
trees or regression equations), instance-based methods do not provide a global model that can be
easily interpreted.
5. Sensitivity to Irrelevant Features: Instance-based methods can be sensitive to irrelevant or
redundant features, which can affect the distance measures used for inding similar instances.
 Examples of Instance-Based Learning Algorithms
1. k-Nearest Neighbors (k-NN): This is the most well-known instance-based algorithm. It classi ies a
new instance based on the majority class among its k-nearest neighbors in the training set.
2. Locally Weighted Regression: This method performs regression analysis locally around the input
point, weighting nearby points more heavily than distant ones.

These methods are particularly useful in scenarios where the relationship between input and output is
too complex to be captured by a simple model, or when the model needs to be frequently updated with
new data.

15. Explain radial basis function.

A Radial Basis Function (RBF) is a real-valued function whose value depends only on the distance from
a central point. In the context of machine learning and neural networks, RBFs are commonly used as
activation functions in hidden layers, particularly in Radial Basis Function Networks (RBFNs).

Radial Basis Function Network (RBFN)

An RBFN is a type of arti icial neural network that uses radial basis functions as activation functions. It
consists of three layers:
1. Input Layer: This layer receives the input features.
2. Hidden Layer: Each neuron in this layer uses an RBF as its activation function. The output of these
neurons depends on the distance between the input and a center (or prototype) vector, and it
usually involves a parameter known as the spread or width.
3. Output Layer: This layer typically involves a linear combination of the outputs from the hidden layer.

Key Concepts of RBF

1. Center (Prototype) Vector: Each RBF neuron has a center vector. The distance between the input
vector and this center vector determines the neuron's output.
2. Distance Measure: The output of an RBF neuron is typically a function of the Euclidean distance
between the input vector and the center vector.
3. Spread (Width): This parameter controls the "spread" or width of the RBF. A smaller spread means
the function is narrower, affecting fewer inputs, while a larger spread means it is broader, affecting
more inputs.

Common Types of Radial Basis Functions

1. Gaussian Function: The most commonly used RBF, de ined as:
𝜙(𝑥) = 𝑒𝑥𝑝 − |𝑥 − 𝑐| /(2𝜎 )
where x is the input vector, ccc is the center vector, and σ is the spread.

2. Multiquadric Function: De ined as:

𝜙(𝑥) = |𝑥 − 𝑐| + 𝜎

3. Inverse Multiquadric Function: De ined as:

𝜙(𝑥) = 1/ |𝑥 − 𝑐| + 𝜎

Applications
RBFNs are widely used in various applications, including:
 Function Approximation: RBFNs can approximate continuous functions and are particularly useful
for interpolating scattered data.
 Pattern Recognition: They can classify patterns by transforming the input space into a higher-
dimensional space where the patterns become more easily separable.
  Time Series Prediction: RBFNs can model complex temporal patterns and make predictions based
on past data.

16. What is Reinforcement Learning and explain Reinforcement learning problem with a neat diagram.
Reinforcement Learning
Reinforcement learning (RL) is a type of machine learning where an autonomous agent learns to
perform actions in an environment to maximize some notion of cumulative reward. The agent interacts
with the environment, observes its state, takes actions, and receives rewards as feedback. The primary
goal of the agent is to learn a policy that maps states to actions in a way that maximizes the total reward
over time.
Key Concepts:
 Agent: The learner or decision-maker.
 Environment: Everything the agent interacts with.
 State: A representation of the current situation.
 Action: A decision made by the agent.
 Reward: The feedback from the environment.
The RL problem can be formally de ined by a Markov Decision Process (MDP), where the agent seeks to
learn a policy that maximizes the expected sum of rewards, often considering a discount factor to
account for the value of future rewards.

Reinforcement Learning Problem

The agent exists in an environment characterized by a set of possible states S. At any given time t, the
agent can take an action a from a set of possible actions A. This action in luences the environment,
resulting in a new state st+1 and a reward r. The agent's goal is to learn a policy π that maximizes the
total expected reward over time.

The RL problem has several key characteristics:

1. Delayed Reward: The agent receives feedback not immediately, but after several actions.
2. Exploration vs. Exploitation: The agent must balance exploring new actions and states to ind
better long-term rewards with exploiting known actions that provide immediate rewards.
3. Partially Observable States: In many practical scenarios, the agent may not fully observe the
environment's state, requiring it to infer or remember past observations.
4. Life-long Learning: The agent may need to learn multiple tasks over time, utilizing past
experiences to improve future learning ef iciency.
Diagram:

17. Write Reinforcement learning problem characteristics.

1. Delayed reward: The task of the agent is to learn a target function 𝜋 that maps from the current state s
to the optimal action a = 𝜋 (s). In reinforcement learning, training information is not available in (s, 𝜋
(s)). Instead, the trainer provides only a sequence of immediate reward values as the agent executes its
sequence of actions. The agent, therefore, faces the problem of temporal credit assignment: determining
which of the actions in its sequence are to be credited with producing the eventual rewards.
2. Exploration: In reinforcement learning, the agent in luences the distribution of training examples by
the action sequence it chooses. This raises the question of which experimentation strategy produces
most effective learning. The learner faces a trade-off in choosing whether to favor exploration of
unknown states and actions, or exploitation of states and actions that it has already learned will yield
high reward.
3. Partially observable states: The agent's sensors can perceive the entire state of the environment at
each time step, in many practical situations sensors provide only partial information. In such cases, the
agent needs to consider its previous observations together with its current sensor data when choosing
actions, and the best policy may be one that chooses actions speci ically to improve the observability of
the environment.
4. Life-long learning: Robot requires to learn several related tasks within the same environment, using
the same sensors. For example, a mobile robot may need to learn how to dock on its battery charger,
how to navigate through narrow corridors, and how to pick up output from laser printers. This setting
raises the possibility of using previously obtained experience or knowledge to reduce sample
complexity when learning new tasks.