AIML Module-3

Deepak B L
Assistant Professor
Dept. of CSE,SJCIT
 TOPICS

1. Basics of Learning theory

2. Similarity Based Learning
3. Regression Analysis
 Introduction to learning
The process of acquiring knowledge and expertise through study,
experience, or being taught is called as learning.

Learning types/methods
1.Learn by memorization
2.Learn by examples
3.Learn by being taught
4.Learning by critical thinking
5.Self learning
6.Learning to solve problems
7.Learning by generalizing explanations
• According to Tom Mitchell, “A computer program is said to be
learning from experience (E), with respect to some task (T). Thus,
the performance measure (P) is the performance at task T, which
is measured by P, and it improves with experience E.”

• Example: In Spam E-Mail detection,

✔ Task, T: To classify mails into Spam or Not Spam.
✔ Performance measure, P: Total percent of mails being correctly
classified as being “Spam” or “Not Spam”.
✔ Experience, E: Set of Mails with label “Spam”
 Design of a learning system
A system that is built around a learning algorithm is called a learning
system.
steps
1. Choosing a training experience
2. Choosing a target function
3. Representation of a target function
4. Function approximation
Training experience

✔ It is important to note that the data or experience that we fed to

the algorithm must have a significant impact on the Success or
Failure of the Model. So Training data or experience should be
chosen wisely.
✔ The training experience will be able to provide direct or indirect
feedback regarding choices. For example: While Playing chess the
training data will provide feedback to itself like instead of this
move if this is chosen the chances of success increases.
Choosing target function

✔ According to the knowledge fed to the algorithm the machine

learning will choose NextMove function which will describe what
type of legal moves should be taken.
✔ For example : While playing chess with the opponent, when
opponent will play then the machine learning algorithm will
decide what be the number of possible legal moves taken in
order to get success.
Representation for Target function

✔ When the machine algorithm will know all the possible legal
moves the next step is to choose the optimized move
✔ For Example : while playing chess machine have 4 possible
moves, so the machine will choose that optimized move which
will provide success to it.
Function Approximation Algorithm

✔ An optimized move cannot be chosen just with the training data.

The training data had to go through with set of example and
through these examples the training data will approximates
which steps are chosen and after that machine will provide
feedback on it.
✔ For Example : When a training data of Playing chess is fed to
algorithm so at that time it is not machine algorithm will fail or
get success and again from that failure or success it will measure
while next move what step should be chosen and what is its
success rate.
Introduction to concept learning
A Concept is a subset of objects or events defined over a larger set
[Example: The concept of a bird is the subset of all objects (i.e., the set of
all things or all animals) that belong to the category of bird.]

Alternatively, a concept is a boolean-valued function defined over

this larger set [Example: a function defined over all animals whose
value is true for birds and false for every other animal].
 What is Concept-Learning?

• Given a set of examples labeled as members or non-members

of a concept, concept-learning consists of automatically
inferring the general definition of this concept.

• In other words, concept-learning consists of approximating a

Boolean-valued function from training examples of its input
and output.
Concept learning requires three things
1. Input: training dataset which is a set of training
instances, each labelled with the name of a concept. Use
this past experience to train and build the model.
2. Output: target function f. it is a mapping function f(x)
from input x output y. it is to determine specific features
to identify an object.
3. Test: new instances to test the learned model.
 A Concept Learning Task

• Consider the example task of learning the target

concept "days on which my friend Aldo enjoys his
favorite water sport.“
• Table describes a set of example days, each
represented by a set of attributes.
• The attribute Enjoy Sport indicates whether or not
Aldo enjoys his favorite water sport on this day.
• The task is to learn to predict the value of Enjoy Sport
for an arbitrary day, based
Table 2.1 Positive & negative training examples for the target concept EnjoySport.
 Representation of Hypothesis
• Hypothesis ‘h’ approximates a target function ‘f’ t represent the
relationship between the independent attributes and the dependent
attribute of the training instances.
• It predicts the model that best maps the inputs to outputs.
• Each hypothesis is represented in conjunction of attribute
conditions.
• Example: (sky=sunny) ^ ( wind=strong)
• Conjunction of constraints on each attribute where:
✔ “?” means “any value is acceptable”( sky=?)
✔ single values denotes a specific single value from
acceptable values of the attributes (wind=strong)
✔ “∅” means “no value is acceptable” (rain= ∅)
• Example of a hypothesis: <?, Cold, High, ?, ?, ?> (If the air
temperature is cold and the humidity high then it is a good day
for water sports)
• Goal: To infer the “best” concept-description from the set of all
possible hypotheses (“best” means “which best generalizes to
all (known or unknown) elements of the instance space”
concept-learning is an ill-defined task)
• The most general hypothesis-that every day is a good day for
water sports, positive example-is represented by
<?, ?, ?, ?, ?, ?>
• The most specific possible hypothesis-that no day is a positive
example-is represented by <0,0,0,0,0,0>
 Generalization and Specification

• Searching the hypothesis Space

✔ Specialization – General to Specific learning
✔ Generalization – Specific to General learning
 Find-S, a Maximally Specific Hypothesis
Learning Algorithm
• Problem1: Implement and demonstrate the FIND-S algorithm for
finding the most specific hypothesis based on a given set of training
data samples. Read the training data from a .CSV file.
Algorithm:
1. Initialize h to the most specific hypothesis in H
2. For each positive training instance x
For each attribute constraint ai in h
If the constraint ai in h is satisfied by x then do nothing
else replace ai in h by the next more general
constraint that is satisfied by x
3. Output hypothesis h
Illustration:
Step1: Find S
Step2: Find S
Step2: Find S
Iteration 3 and Step 4: Find S
 Properties of Find-S
✔ Hypothesis space described by conjunctions of attributes
✔ Find-S will output the most specific hypothesis within H that is consistent
with the positive training examples
✔ The output hypothesis will also be consistent with the negative examples,
provided the target concept is contained in H.

v
limitations of Find-S
✔ to find a hypothesis that is consistent with positive instances, ignoring all
negative instances. As long as the training dataset is consistent, the
hypothesis found by this algorithm is consistent.
✔ The algorithm finds only one unique hypothesis, where in there may be many
other hypothesis that are consistent with the training dataset.
✔ Inconsistent data instances can mislead this algorithm in determining the
consistent hypothesis.
 Complaints about Find-S

• Can’t tell if the learner has converged to the target

concept, in the sense that it is unable to determine
whether it has found the only hypothesis
consistent with the training examples.
• Can’t tell when training data is inconsistent, as
it ignores negative training examples.
• Why prefer the most specific hypothesis?
• What if there are multiple maximally specific
hypothesis?
 Version Spaces
The version space contains the subset of hypothesis from
the hypothesis space that is consistent with all training
instances in the training dataset.
List-Then-Elimination Algorithm
Eliminating any hypothesis that are inconsistent
Algorithm:
Input: Version Space- a list of all hypothesis.
Output: Set of consistent hypothesis.
✔ Initialize the version space with a list of hypothesis.
✔ For each training instance,
•remove from version space any hypothesis that is inconsistent.
 Similarity-based Learning
•Similarity-based learning is a supervised learning techniques that
predicts the class label of test instance by determining the
similarity of the test instance with the training instance.

•similarity-based learning belongs to the family of instance based

learning which is used to solve both classification and regression
problems.

•instance based learning makes prediction by computing distance

or similarities between test instance and specific set of training
instance.
similarity based classifiers uses similarity measures to locate the
nearest neighbors and classify the instance .
Similarity learning is also called as instance -based
learning/just-in time learning since it does not build an abstract
model of the training instances and performs lazy learning when
classifying a new instances.
 One more way to classify Machine Learning systems is based on how
they generalize.
1. Instance-based Learning
•the system learns the examples by heart, then generalizes to new
cases using a similarity measure. For example, Spam Filter flags the
email identical to known spam emails also can be programmed to
flag the emails that are very similar to known spam emails.
2. Model-based Learning
•In Model-based Learning, a model is built using the training
examples, then use that model to make predictions. How do we know
the model performs best? We need to define a utility function a.k.a
fitness function that measures how good the model is or define a cost
function that measures how bad the model is.
Instance based learning Model-based learning

Lazy learners Eager learners

Processing of training instances is done Processing of training instances is done
only during testing phase only during training phase

No model is built with the training Generalizes a model with the training
instances before receives a test instance instances before it receives a test
instance
Predicts the class of the test instance Predicts the class of the test instance
directly from the training data from the model built

Slow in testing phase Fast in testing phase

Learns by making many local Learns by creating global approximation
approximations
Some of the instance-based learning algorithms are :
✔ K Nearest Neighbor (KNN)
✔ Self-Organizing Map (SOM)
✔ Learning Vector Quantization (LVQ)
✔ Variants of Nearest Neighbor learning
✔ Locally Weighted Regression
✔ Radial Basis Function (RBF) networks
 Nearest-Neighbor Learning
✔It is a non-parametric method that
makes predictions based on the
similarity of data points in a given
dataset.
✔ The K-NN algorithm works by
finding the K nearest neighbors to a
given data point based on a distance
metric, such as Euclidean distance.
Algorithm: k-NN
Input: training dataset T, distance metric d, Test instance t, no. of
nearest neighbors k
Output: Predicted class or category
Prediction: for test instance t,
✔ For each instance I in T, compute the distance between the test
instance t and every other instance I in the training dataset using a
distance metric

✔ Sort the distances in an ascending order and select the first k

nearest training data instances to the test instance.
✔ Predict the class of the test instance by majority voting or mean of
the k selected nearest instance.
Advantages
•The KNN algorithm is very easy to implement
•Nearly optimal in the large sample limit
•Uses local information which can yield highly adaptive
behavior
•Lends itself very easily to parallel implementation
Disadvantages
•Large storage requirements
•Computationally intensive recall
•Highly susceptible to the curse of dimensionality
 Weighted K-Nearest-Neighbor algorithm
✔ It is a non-parametric method that
makes predictions based on the
similarity of data points in a given
dataset.
✔ The K-NN algorithm works by
finding the K nearest neighbors to a
given data point based on a distance
metric, such as Euclidean distance.
 Regression
• Regression in machine learning is a technique used to capture the
relationships between independent and dependent variables, with the main
purpose of predicting an outcome.
• Regression has a wide range of real-life applications. It is essential for any
machine learning problem that involves continuous numbers – this
includes, but is not limited to, a host of examples, including:
✔ Financial forecasting (like house price estimates, or stock prices)
✔ Sales and promotions forecasting
✔ Testing automobiles
✔ Weather analysis and prediction
✔ Time series forecasting
 Types of Regression
Regression Methods

Linear regression Non-linear regression

Logical regression
methods methods

Single linear Multiple linear Polynomial

regression regression regression