Module-I

Introduction to ML,

15CS73 - Machine Learning Harivinod N

Module 1- Outline
▪Chapter 1: Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Chapter 2: Concept Learning

•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
•Inductive Bias
Slides by :Harivinod N
•Summary
Module 1- Outline
▪Chapter 1: Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Chapter 2: Concept Learning

•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
•Inductive Bias 3

•Summary

Well-Posed Learning Problems

▪Learning is broadly defined as any computer program

that improves its performance at some task through
experience.
▪Definition: Machine Learning
A computer program is said to learn from
experience E with respect to some class of tasks
T and performance measure P, if its performance
at tasks in T, as measured by P, improves with
experience E.

Slides by :Harivinod N 4
Ex1: A checkers learning problem

• Task T - Playing checkers

• Performance Measure P -
Percentage of games won
against opponent
• Training Experience E -
Playing practice games
against itself

Slides by:Harivinod N 5

Checkers Board

Slides by :Harivinod N 6
Ex2: A handwriting recognition
learning problem
▪Task T: recognizing and classifying handwritten
words within images
▪Performance measure P: percent of words
correctly classified
▪Training experience E:
a database of
handwritten words with
given classifications

Slides by:Harivinod N 7

Ex3: A robot driving learning problem

▪T: driving on public 4-lane highways using vision
sensors
▪P: average distance traveled before an error (as
judged by human overseer)
▪E: a sequence of images and steering
commands recorded by observing a human
driver

Slides by :Harivinod N 8
Module 1- Outline
▪Chapter 1: Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Chapter 2: Concept Learning

•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
•Inductive Bias Slides by:Harivinod N 9

•Summary

Designing a learning system

Given Problem Description, the steps to design a

learning system are as follows

1. Choosing the Training

Experience

2. Choosing the Target Function

3. Choosing a Representation for the Target Function

4. Choosing a Function Approximation Algorithm

5. The Final Design

Slides by :Harivinod N 10
Problem Description:

A Checker Learning Problem

Let us consider designing a program to learn to play
checkers, with the goal of entering it in the world
checkers tournament.
We adopt the obvious performance measure: the
percent of games it wins in this world tournament.
▪Task T: Playing Checkers
▪Performance Measure P: Percent of games won
against opponents
▪Training Experience E: To be selected => Games
Played against itself
Slides by:Harivinod N 11

Slides by :Harivinod N 12
Designing a learning system

Given Problem Description, the steps to design a

learning system are as follows

1. Choosing the Training

Experience

2. Choosing the Target Function

3. Choosing a Representation for the Target Function

4. Choosing a Function Approximation Algorithm

5. The Final Design

Slides by:Harivinod N 13

Choosing a training experience

The type of training experience available can have a

significant impact on success or failure of the learner.
1. Will the training experience provide direct or
indirect feedback?
a. Direct Feedback: system learns from examples
of individual checkers board states and the
correct move for each
b. Indirect Feedback: Move sequences and final
outcomes of various games played
•Credit assignment problem: Value of early states
must be inferred from the outcome
Slides by :Harivinod N 14
Choosing a training experience

2. Degree to which the learner controls the

sequence of training examples
a. Teacher selects informative boards and gives
correct move
b. Learner proposes board states that it finds
particularly confusing. Teacher provides
correct moves
c. Learner controls board states and (indirect)
training classifications
3. How well it represents the distribution of examples
over which the final system performance P must be
measured. 15
Slides by:Harivinod N

Partial Design of Checkers Learning

Program

▪A checkers learning problem:

•Task T: playing checkers
•Performance measure P: percent of games won
in the world tournament
•Training experience E: games played against itself

▪Remaining choices
•The exact type of knowledge to be learned
•A representation for this target knowledge
•A learning mechanism

Slides by :Harivinod N 16
Designing a learning system

Given Problem Description, the steps to design a

learning system are as follows

1. Choosing the Training

Experience

2. Choosing the Target Function

3. Choosing a Representation for the Target Function

4. Choosing a Function Approximation Algorithm

5. The Final Design

Slides by:Harivinod N 17

Choosing the Target Function (1)

▪Assume that you can determine legal moves

▪Program needs to learn the best move from among
legal moves
•Defines large search space known a priori
•target function: ChooseMove : B → M
▪ ChooseMove is difficult to learn given indirect
training

▪So we go for Alternative target function

Slides by :Harivinod N 18
Choosing the Target Function (2)

Slides by:Harivinod N 19

Choosing the Target Function (3)

▪ ChooseMove is difficult to learn given indirect training

▪So we go for Alternative target function

▪An evaluation function that assigns a numerical score

to any given board state
▪V : B → ℜ ( where ℜ is the set of real numbers, B is the set of
board states)

Slides by :Harivinod N 20
 Choosing the Target Function (4)

15CS73 - Machine Learning Harivinod N 21

Choosing the Target Function (5)

▪V(b) for an arbitrary board state b in B

• if b is a final board state that is won, then V(b) = 100
…. is lost, then V(b) = -100, … is drawn, then V(b) = 0
• if b is not a final state, then V(b) = V(b’),
where b' is the best final board state that can be achieved
• starting from b and playing optimally until the end of the game
▪ V(b) gives a recursive definition for board state b
▪ Not usable because not efficient to compute except is first
three trivial cases
▪ V is a nonoperational definition

Slides by :Harivinod N 22
Choosing the Target Function (6)

▪ Goal of learning is to discover an operational description

of V
▪ Learning the target function is often called
function approximation
▪ Approximated function is Referred to as Vˆ

Slides by:Harivinod N 23

Designing a learning system

Given Problem Description, the steps to design a

learning system are as follows

1. Choosing the Training

Experience

2. Choosing the Target Function

3. Choosing a Representation for the Target Function

4. Choosing a Function Approximation Algorithm

5. The Final Design

Slides by :Harivinod N 24
Choosing a Representation for the
Target function
▪Choice of representations involve trade offs
• Pick a very expressive representation to allow close approximation
to the ideal target function V
• More expressive, more training data required to choose among
alternative hypotheses
▪Use linear combination of the following board features:
• x1: the number of black pieces on the board
• x2: the number of red pieces on the board
• x3: the number of black kings on the board
• x4: the number of red kings on the board
• x5: the number of black pieces threatened by red (i.e. which
can be captured on red's next turn)
• x6: the number of red pieces threatened by black
ˆ
V (b) = w + w x + w x + w x + w x + w x + w x
0 1 1 2 2 3 3 4 4 5 5 66 25
Slides by:Harivinod N

Partial Design of Checkers Learning

Program

▪A checkers learning problem:

•Task T: playing checkers
•Performance measure P: percent of games won
in the world tournament
•Training experience E: games played against itself

•Target Function: V: Board → ℜ

•Target function representation
ˆ
V (b) = w + w x + w x + w x + w x + w x + w x
0 1 1 2 2 3 3 4 4 5 5 66

Slides by :Harivinod N 26
Designing a learning system

Given Problem Description, the steps to design a

learning system are as follows

1. Choosing the Training

Experience

2. Choosing the Target Function

3. Choosing a Representation for the Target Function

4. Choosing a Function Approximation Algorithm

5. The Final Design

Slides by:Harivinod N 27

Slides by :Harivinod N 28
Choosing a Function Approximation
Algorithm
▪To learn we require a set of training examples
describing the board b and the training value Vtrain(b)

▪Ordered b,Vtrain(b)
Pair
x = 3, x = 0, x = 1, x = 0, x = 0, x = 0 , +100
1 2 3 4 5 6

Slides by:Harivinod N 29

Choosing a Function Approximation

Algorithm
1. Estimating Training Values
▪The only training information available to our
learner is whether the game was eventually won
or lost.
▪Despite the ambiguity inherent in estimating training
values for intermediate board states, one simple
approach has been found to be surprisingly
successful.
▪This approach is to assign the training value of Vtrain(b)
^
for any intermediate board state b to be V
(Successor(b))

Slides by :Harivinod N 30
Rule for estimating training values:
Choosing a Function Approximation
Algorithm
2. Adjusting the weights
One common approach is to define the best hypothesis,
or set of weights, as that which minimizes the square
error E between the training values and the values
predicted by the hypothesis V.

Slides by:Harivinod N 31

Choosing a Function Approximation

Algorithm
2. Adjusting the weights
We require an algorithm that will incrementally refine
the weights as new training examples become
available and that will be robust to errors in these
estimated training values.
One such algorithm is called the least mean squares
(LMS) training rule.

LMS Weight update rule

Slides by :Harivinod N 32
Designing a learning system

Given Problem Description, the steps to design a

learning system are as follows

1. Choosing the Training

Experience

2. Choosing the Target Function

3. Choosing a Representation for the Target Function

4. Choosing a Function Approximation Algorithm

5. The Final Design

Slides by:Harivinod N 33

The final design

▪The final design of our checkers learning system can be
naturally described by four distinct program modules that
represent the central components in many learning
systems.
1. The Performance System is the module that must
solve the given performance task,
•in this case playing checkers, by using the learned
target function(s).
•It takes an instance of a new problem (new game) as
input and produces a trace of its solution (game
history) as output.
2. The Critic takes as
•input - the history orbytrace
Slides ofN the game and
:Harivinod 34

•produces as output - a set of training examples of the

The final design
3. The Generalizer takes as input the training
examples and produces an output hypothesis that
is its estimate of the target function.
It generalizes from the specific training examples,
hypothesizing a general function that covers these
examples and other cases beyond the training
examples.

4. The Experiment Generator takes as input the

current hypothesis (currently learned function) and
outputs a new problem (i.e., initial board state) for
the Performance System to explore.
Its role is to pick new practice problems that
Slides by:Harivinod N 35
will maximize the learning rate of the
overall system

Summary of final design

Slides by :Harivinod N 36
 Summary of choices
in designing the
checkers learning
program

Slides by:Harivinod N 37

Module 1- Outline
▪Chapter 1: Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Chapter 2: Concept Learning

•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
•Inductive Bias Slides by :Harivinod N 38

•Summary
Perspective in ML
▪One useful perspective on machine learning is that
it involves searching a very large space of
possible hypotheses to determine
one that best fits the observed data
and any prior knowledge held by the learner.
▪For example, consider the space of hypotheses that
could in principle be output by the above checkers
learner.
▪ This hypothesis space
consists of all evaluation
functions
that can be represented by
some choice of values for the weights w0 through w6.
Slides by:Harivinod N 39

Perspective in ML
▪The learner's task is thus to search through this vast
space to locate the hypothesis that is most consistent
with the available training examples.
▪The LMS algorithm for fitting weights achieves this
goal by iteratively tuning the weights, adding a
correction to each weight each time the
hypothesized evaluation function predicts a value
that differs from the training value.
▪This algorithm works well when the hypothesis
representation considered by the learner defines a
continuously parameterized space of potential
hypotheses.
Slides by :Harivinod N 40
Issues in ML
Our checkers example raises a number of generic
questions about ML. It is concerned with answering
questions like;
▪What algorithms exist for learning general target
functions from specific training examples? In what
settings will particular algorithms converge to the
desired function, given sufficient training data? Which
algorithms perform best for which types of problems
and representations?
▪How much training data is sufficient? What general
bounds can be found to relate the confidence in learned
hypotheses to the amount of training experience and
the character of theSlides
learner's hypothesis space?
by:Harivinod N 41

Issues in ML
▪When and how can prior knowledge held by the learner
guide the process of generalizing from examples?
Can prior knowledge be helpful even when it is
only approximately correct?
▪What is the best strategy for choosing a useful next
training experience, and how does the choice of this
strategy alter the complexity of the learning problem?
▪What is the best way to reduce the learning task to one
or more function approximation problems?
Put another way, what specific functions should the
system attempt to learn? Can this process itself be
automated?
▪How can the learner automatically alter its
Slides by :Harivinod N 42
representation to improve its ability to represent and
learn the target function?
 Module 1- Outline
▪Chapter 1: Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Chapter 2: Concept Learning

•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
•Inductive Bias Slides by:Harivinod N 43

•Summary

Summary
▪ML have proven to be of great value in a variety of
applications.
1. Data mining problems containing large databases
a. To analyze outcomes of medical treatments from
patient databases
b. To learn general rules for credit worthiness from
financial databases
2. Poorly understood domains where humans might not
have the knowledge needed to develop effective
algorithms
e.g., human face recognition from images
3. Domains where the program must dynamically
adapt to changing conditions
44
a. controlling manufacturing processes
Slides by :Harivinod N under changing supply
stocks
Summary
▪Machine learning draws on ideas from a diverse set
of disciplines, including AI, probability and statistics,
computational complexity, information theory,
psychology and neurobiology, control theory, and
philosophy.
▪A well-defined learning problem requires a
well-specified task, performance metric, and source of
training experience.
▪Designing a machine learning approach involves a
number of design choices, including
•choosing the type of training experience,
•the target function to be learned,
•a representationSlides
for by:Harivinod
this target N
function, and 45
•an algorithm for learning the target function from
training examples.

Summary
▪Learning involves search:
searching through a space of possible hypotheses
to find the hypothesis that best fits the available
training examples and other prior constraints or
knowledge.

Slides by :Harivinod N 46
Module 1- Outline
▪Chapter 1: Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Chapter 2: Concept Learning

•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
•Inductive Bias Slides by:Harivinod N 47

•Summary

Concept Learning
▪Inducing general functions from specific training
examples is a main issue of machine learning.
▪A task of acquiring a potential hypothesis (solution) that
best fits the training examples
▪It is the process of acquiring the definition of a
general category from given sample positive and
negative training examples of the category.
▪Concept Learning can seen as a problem of searching
through a predefined space of potential hypotheses for
the hypothesis that best fits the training examples.
▪Concept learning: Inferring a boolean-valued function
from training examples of its input and output.
Slides by :Harivinod N 48
Module 1- Outline
▪Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Concept Learning
•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
•Inductive Bias Slides by:Harivinod N 49

•Summary

Hypothesis

50
Slides by :Harivinod N
Classification

Slides by:Harivinod N 51

Concept Learning Task

▪Problem is to learning the target concept
"Days on which my friend Sachin enjoys his
favorite water sport”.
▪Given

Slides by :Harivinod N 52
 Concept Learning Task
▪What hypothesis representation shall we provide to
the learner in this case?
▪For each attribute, the hypothesis will either
• indicate by a “ ? ” that any value is acceptable for this attribute,
• specify a single required value (e.g., Warm) for the attribute, or
• indicate by a "Φ" that no value is acceptable.

▪If some instance x satisfies all the constraints of

hypothesis h, then h classifies x as a positive example
(h(x) = 1).
▪hypothesis that Sachin enjoys his favorite sport only
on cold days with high humidity : (?, Cold, High, ?, ?,
?)
▪Most general hypothesis - (?,
15CS73 - Machine Learning
?, ?, ?, ?, ?)
Harivinod N 53

▪Most specific possible hypothesis- (Φ, Φ, Φ, Φ, Φ, Φ)

“EnjoySport” Concept Learning Task

Slides by :Harivinod N 54
 Inductive learning hypothesis
▪Our assumption is that the Fundamental
assumption of inductive learning.
“Best hypothesis regarding unseen instances is
the hypothesis that best fits the observed
training data.”

▪The inductive learning hypothesis.

Any hypothesis found to approximate the target
function well over a sufficiently large set of training
examples will also approximate the target function
well over other unobserved examples.

15CS73 - Machine Learning Harivinod N 55

Module 1- Outline
▪Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Concept Learning
•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
•Inductive Bias
Slides by :Harivinod N 56

•Summary
Concept learning as search
▪Concept learning can be viewed as the task of searching
through a large space of hypotheses
▪The goal is to find the hypothesis that best fits training
examples.
▪General-to-Specific Ordering of Hypotheses

h2 is more general than hl.

Harivinod N 57

Concept learning as search

Slides by :Harivinod N 58
 Module 1- Outline
▪Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Concept Learning
•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
15CS73 •Inductive Bias
- Machine Learning Harivinod N 59

•Summary

Find-S: Finding A Maximally Specific

Hypothesis
▪How can we use the more-general-than partial
ordering to organize the search for a hypothesis
consistent with the observed training examples?
▪One way is to begin with the most specific possible
hypothesis in H, then generalize this hypothesis each
time it fails to cover an observed positive training
example.
▪FIND-S algorithm is used for this purpose.

Slides by :Harivinod N 60
 Find-S Algorithm
To Find Maximally Specific Hypothesis

1. Initialize h to the most specific hypothesis in H

2. For each positive training instance x
For each attribute constrain ai in h
•If the constraint ai is satisfied by x then do
nothing
•Else replace ai in h by the next more
general constraint that is satisfied by x
3. Output Hypothesis h

15CS73 - Machine Learning Harivinod N 61

Illustration

Slides by :Harivinod N 62
 Key Property
▪The key property of the Find-S algorithm is that for
hypothesis spaces described by conjunctions of
attribute constraints
▪Find-S is guaranteed to output the most specific
hypothesis within H that is consistent with the positive
training examples.

15CS73 - Machine Learning Harivinod N 63

Find S - Drawback
Questions still left unanswered
▪Has the learner converged to the correct target concept?
▪Why prefer the most specific hypothesis?
If multiple hypotheses consistent with the training
examples, FIND-S will find the most specific. It is
unclear whether we should prefer this hypothesis
▪Are the training examples consistent?
Training examples may contain at least some errors or
noise. Such inconsistent sets of training examples can
severely mislead FIND-S, since it ignores negative
examples.
▪What if there are several maximally specific
consistent hypotheses?
There can be several maximally
Slides by :Harivinod N specific 64

hypotheses consistent with the data. Find S

 Module 1- Outline
▪Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Concept Learning
•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
15CS73 •Inductive Bias
- Machine Learning Harivinod N 65

•Summary

Definition: Consistent

▪Notice the key difference between this definition of

consistent and our earlier definition of satisfies.
▪An example x is said to satisfy hypothesis h when h(x)
= 1, regardless of whether x is a positive or negative
example of the target concept.
▪However, whether such an example is consistent
with h depends on the target concept, and in
particular, whether h(x) = c(x).

Slides by :Harivinod N 66
 Definition: Version Space

▪This subset of all hypotheses is called the version

space with respect to the hypothesis space H and the
training examples D, because it contains all plausible
versions of the target concept.

15CS73 - Machine Learning Harivinod N 67

The List-Then-Eliminate algorithm

▪One obvious way to represent the version space is

simply to list all of its members.
▪This leads to a simple learning algorithm, which we
might call the List-Then-Eliminate algorithm.

Slides by :Harivinod N 68
 Few definition

15CS73 - Machine Learning Harivinod N 69

Module 1- Outline
▪Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Concept Learning
•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
•Inductive Bias Slides by :Harivinod N 70

•Summary
 Candidate Elimination Algorithm

▪The CEA computes the version space containing

all hypotheses from H that are consistent with an
observed sequence of training examples.
▪It begins by
•initializing the version space to the set of all
hypotheses in H; that is, G-most general hypothesis
in H

•and initializing the S boundary set to contain the

most specific (least general) hypothesis

15CS73 - Machine Learning Harivinod N 71

Candidate Elimination Algorithm using Version Spaces

1. Initialize G to the set of maximally general hypotheses in H

2. Initialize S to the set of maximally specific hypotheses in H
3. For each training example d, do
a. If d is a positive example
▪ Remove from G any hypothesis inconsistent with d,
▪ For each hypothesis s in S that is not consistent with d,
• Remove s from S
• Add to S all minimal generalizations h of s such that h is consistent with d, and
some member of G is more general than h
• Remove from S, hypothesis that is more general than another hypothesis in S
b. If d is a negative example
▪ Remove from S any hypothesis inconsistent with d
▪ For each hypothesis g in G that is not consistent with d
• Remove g from G
• Add to G all minimal specializations h of g such that h is consistent with d, and
some member of S is more specific than h
• Remove from G any hypothesis that is less general than another in G

Slides by :Harivinod N 72
 Illustration

15CS73 - Machine Learning Harivinod N 73

Illustration

Slides by :Harivinod N 74
Illustration

Slides by:Harivinod N 75

Illustration

Slides by :Harivinod N 76
 Illustration

15CS73 - Machine Learning Harivinod N 77

Illustration

Slides by :Harivinod N 78
Illustration

The final version space for the EnjoySport concept

learning problem

Slides by:Harivinod N 79

Illustration
▪After processing these four examples, the boundary
sets S4 and G4 delimit the version space of all
hypotheses consistent with the set of incrementally
observed training examples.
▪The entire version space, including those
hypotheses bounded by S4 and G4.
▪This learned version space is independent of the
sequence in which the training examples are presented
▪As further training data is encountered, the S and G
boundaries will move monotonically closer to each
other

Slides by :Harivinod N 80
 Module 1- Outline
▪Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Concept Learning
•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
15CS73 •Inductive
- Machine LearningBias Harivinod N 81

•Summary

Inductive Bias
▪The CEA will converge toward the true target
concept provided it is given accurate training
examples
▪The fundamental questions for inductive inference in
general.
•What if the target concept is not contained in
the hypothesis space?
•How does the size of this hypothesis space
influence the ability of the algorithm to generalize
to unobserved instances?
•How does the size of the hypothesis space
influence the number of training examples that
must be observed?
Slides by 82
▪Here we examine them in the context of the CEA.
:Harivinod
N
 Inductive Bias
1. A Biased Hypothesis Space
▪Suppose we wish to assure that the hypothesis space
contains the unknown target concept.
▪The obvious solution is to enrich the hypothesis
space to include every possible hypothesis.

▪Consider EnjoySport example in which we restricted

the hypothesis space to include only conjunctions of
attribute values.

15CS73 - Machine Learning Harivinod N 83

Inductive Bias
1. A Biased Hypothesis Space
(Continued)

▪Most specific hypothesis consistent with the first

two examples

▪It incorrectly covers the third (negative) training

example
▪The problem is that we have biased the learner to
consider only conjunctive hypotheses.
▪In this case we require a more expressive hypothesis
Slides by 84
:Harivinod
space. N
 Inductive Bias
2. An unbiased learner
▪The obvious solution to be a unbiased learner–
design hypothesis space H to represent every
teachable concept;
▪It should capable of representing every possible subset
of the instances X. In general, the set of all subsets of a
set X is called the power-set of X.
▪In general, number of distinct subsets is 2|X|.
▪Thus, there are 296, or approximately distinct target
concepts that could be defined over this instance
space and that our learner might be called upon to
learn.
15CS73 - Machine Learning Harivinod N 85
▪Our conjunctive hypothesis space is able to
represent only 973 of these-a very biased
hypothesis space indeed!

Inductive Bias
2. An unbiased learner (Continued)
▪Let us reformulate the Enjoysport learning task
▪Let H’ represent every subset of instances; that is,
let H' correspond to the power set of X.
▪One way to define such an H' is to allow arbitrary
disjunctions, conjunctions, and negations of our
earlier hypotheses.
▪For instance, the target concept "Sky = Sunny or Sky =
Cloudy" could then be described as

Slides by 86
:Harivinod
N
 Inductive Bias
2. An unbiased learner (Continued)
▪Now new problem: we are completely unable to
generalize beyond the observed examples!
▪To see why, suppose we present three positive
examples (xl, x2, x3) and two negative examples (x4,
x5) to the learner.
▪At this point, the S and G boundary of the version
space will be

▪Here in order to converge to a single, final target

concept, we will have to present every single instance
in X as a training example!Harivinod N
15CS73 - Machine Learning 87

Inductive Bias
3. The Futility of Bias-Free Learning
▪The fundamental property of inductive inference:
A learner that makes no a priori assumptions
regarding the identity of the target concept has no
rational basis for classifying any unseen instances.
▪CEA generalizes observed training examples because
it was biased by the implicit assumption that the
target concept could be represented by a conjunction
of attribute values.
• If this assumption is correct (and the training examples
are error-free), its classification of new sample will also
be correct.
• If this assumption is incorrect,
Slides however,
by it is certain that the88
CEA will mis-classify at least:Harivinod
some instances from X.
N
 Inductive bias
▪Consider a concept learning algorithm L for the
set of instances X.
▪Let c be an arbitrary concept defined over X, and let Dc =
{ x, c(x) } be an arbitrary set of training examples of c.
▪Let L(xi, Dc) denote the classification assigned to the
instance xi by L after training on the data Dc.
▪The inductive bias of L is any minimal set of assertions
B such that for any target concept c and corresponding
training examples Dc

15CS73 - Machine Learning Harivinod N 89

Inductive bias of CEA

”The target concept c is contained in the given
hypothesis space H”.

Slides by 90
:Harivinod
N
 Algorithms listed from weakest to
strongest bias
▪Rote-Learner: Learning corresponds simply to storing
each observed training example in memory.
▪CEA: New instances are classified only in the case
where all members of the current version space
agree on the classification. Otherwise, the system
refuses to classify the new instance.
▪FIND-S: This algorithm, described earlier, finds the most
specific hypothesis consistent with the training
examples. It then uses this hypothesis to classify all
subsequent instances.

15CS73 - Machine Learning Harivinod N 91

Module 1- Outline
▪Introduction
•Well posed learning problems
•Designing a Learning system
•Perspective and Issues in Machine
Learning
•Summary

▪Concept Learning
•Concept learning task
•Concept learning as search
•Find-S algorithm
•Version space
•Candidate Elimination algorithm
•Inductive Bias Slides by 92
:Harivinod
•Summary N
 Summary …(1)
▪Concept learning can be cast as a problem of
searching through a large predefined space of
potential hypotheses.
▪The general-to-specific partial ordering of hypotheses,
provides a useful structure for organizing the search
through the hypothesis space.
▪The Find-S algorithm
utilizes general-to-specific ordering,
performing a specific-to-general
search through the hypothesis
space
along one branch of the partial
ordering, to find the most specific
15CS73hypothesis
- Machine Learning consistent with 93
the N
Harivinod

training examples.

Summary …(2)
▪ The Candidate Elimination Algorithm
utilizes this general-to-specific
ordering to compute the version
space
(the set of all hypotheses consistent with the training
data)
by incrementally computing the sets of maximally
specific (S) and maximally general (G) hypotheses.

▪Because the S and G sets delimit the entire set of

hypotheses consistent with the data, they provide the
learner with a description of its uncertainty regarding the
exact identity of the target concept.
Slides by 94
:Harivinod
N
 Summary …(3)
▪Version spaces and the CEA provide a useful
conceptual framework for studying concept
learning.
▪However, CEA is not robust to noisy data or to
situations in which the unknown target concept is not
expressible in the provided hypothesis space.
▪Inductive learning algorithms
are able to classify unseen examples
only because of their implicit inductive bias
for selecting one consistent hypothesis over another.

15CS73 - Machine Learning Harivinod N 95

Slides by 96
:Harivinod
N