CS8082 – Machine Learning Techniques
Unit 4 – Instance-Based Learning
Locally weighted regression
Unit Objectives
• Understand the need for machine learning for various problem solving
• Study the various supervised, semi-supervised and unsupervised learning
algorithms in machine learning
2
Unit Outcomes
At the end of the course, the student should be able to:
• CO 1: Differentiate between supervised, unsupervised, semi-supervised machine learning
approaches
• CO 2: Apply specific supervised or unsupervised machine learning algorithm for a particular
problem
• CO 3: Analyze and suggest the appropriate machine learning approach for the various types of
problem
• CO 4: Design and make modifications to existing machine learning algorithms to suit an
individual application
• CO 5: Provide useful case studies on the advanced machine learning algorithms
3
Prerequisite
Understanding on k-NN algorithm
Familiar to distance based instance similarity identification
General idea on gradient descent
What is instance-based learning?
In contrast to learning methods that construct a general, explicit
description of the target function when training examples are
provided, instance-based learning methods simply store the
training examples.
Instance-based learning includes nearest neighbor and locally
weighted regression methods that assume instances can be
represented as points in a Euclidean space.
Instance-based methods are sometimes referred to as "lazy"
learning methods because they delay processing until a new
instance must be classified.
Instance-based learning
Learning in these algorithms consists of simply storing
the presented training data.
When a new query instance is encountered, a set of similar related
instances is retrieved from memory and used to classify the new query
instance.
Many techniques present only local approximation and
not for the entire instance space.
2. Locally weighted regression
Generalization of approximating as the target function f (x) at the single
query point x = xq
Locally weighted regression uses nearby or distance-weighted training
examples to form this local approximation to f.
Locally weighted regression
local because the function is approximated based only on data near the query
point,
weighted because the contribution of each training example is weighted by its
distance from the query point, and
Regression because this is the term used widely in the statistical learning
community for the problem of approximating real-valued functions.
Understanding Locally weighted regression
Locally weighted linear regression
Consider a linear function
While concentrating on global approximation to target function we
derive methods to choose weights that minimize the squared error
summed over the set D of training examples
Locally weighted linear regression
How shall we modify this procedure to derive a local approximation rather than a
global one?
The simple way is to redefine the error criterion E to emphasize fitting the local
training examples.
Three possible criteria are given below.
Locally weighted linear regression
Rederiving the gradient descent rule:
The contribution of instance x to the weight update is now multiplied
by the distance penalty and that the error is summed over only the k
nearest training examples.
Summary
The target function value for a new query is estimated from the known values
of the k nearest training examples.
Locally weighted regression methods are a generalization of k-NN which an
explicit local approximation to the target function is constructed for each query
instance.
The local approximation to the target function may be based on a variety of
functional forms such as constant, linear, or quadratic functions or on spatially
localized kernel functions.
References
TEXT BOOKS:
1. Tom M. Mitchell, ―Machine Learning‖, McGraw-Hill Education (India) Private Limited, 2013.
REFERENCES:
2. Ethem Alpaydin, ―Introduction to Machine Learning (Adaptive Computation and Machine
Learning), The MIT Press 2004.
3. Stephen Marsland, ―Machine Learning: An Algorithmic Perspective ‖, CRC Press, 2009.
