Concept Learning

• Learning from examples

• General-to-specific ordering over hypotheses
• Version Spaces and candidate elimination
algorithm
• Picking new examples
• The need for inductive bias

CS 5751 Machine Chapter 2 Concept Learning 1

Learning
 Some Examples for SmileyFaces

Eyes Nose Head Fcolor Hair? Smile?

CS 5751 Machine Chapter 2 Concept Learning 2

Learning
 Features from Computer View

Eyes Nose Head Fcolor Hair? Smile?

Round Triangle Round Purple Yes Yes

Square Square Square Green Yes No

Square Triangle Round Yellow Yes Yes

Round Triangle Round Green No No

Square Square Round Yellow Yes Yes

CS 5751 Machine Chapter 2 Concept Learning 3

Learning
 Representing Hypotheses
Many possible representations for hypotheses h
Idea: h as conjunctions of constraints on features
Each constraint can be:
– a specific value (e.g., Nose = Square)
– don’t care (e.g., Eyes = ?)
– no value allowed (e.g., Water=Ø)
For example,
Eyes Nose Head Fcolor Hair?
<Round, ?, Round, ?, No>
?

CS 5751 Machine Chapter 2 Concept Learning 4

Learning
 Prototypical Concept Learning Task
Given:
– Instances X: Faces, each described by the attributes
Eyes, Nose, Head, Fcolor, and Hair?
– Target function c: Smile? : X -> { no, yes }
– Hypotheses H: Conjunctions of literals such as
<?,Square,Square,Yellow,?>
– Training examples D: Positive and negative examples
of the target function
 x1 , c( x1 ) ,  x2 , c( x2 ) ,...,  xm , c( xm ) 
Determine: a hypothesis h in H such that h(x)=c(x)
for all x in D.

CS 5751 Machine Chapter 2 Concept Learning 5

Learning
 Inductive Learning Hypothesis
Any hypothesis found to approximate the target
function well over a sufficiently large set of
training examples will also approximate the target
function well over other unobserved examples.

• What are the implications?

• Is this reasonable?
• What (if any) are our alternatives?
• What about concept drift (what if our views/tastes
change over time)?

CS 5751 Machine Chapter 2 Concept Learning 6

Learning
 Instances, Hypotheses, and More-General-Than

Instances X Hypotheses H

General

h3

h1 h2
Specific

x 1 =<Round,Square,Square,Purple,Yes> h 1 =<Round,?,Square,?,?>
x 2 =<Round,Square,Round,Green,Yes> h 2 =<Round,?,?,?,Yes>
h 3 =<Round,?,?,?,?>

CS 5751 Machine Chapter 2 Concept Learning 7

Learning
 Find-S Algorithm
1. Initialize h to the most specific hypothesis in H
2. For each positive training instance x
For each attribute constraint ai in h
IF the constraint ai in h is satisfied by x THEN
do nothing
ELSE
replace ai in h by next more general constraint satisfied by x

3. Output hypothesis h

CS 5751 Machine Chapter 2 Concept Learning 8

Learning
 Hypothesis Space Search by Find-S
Instances X Hypotheses H
h5
x
General
2
x x h 3,4

1 3

x
5

x h 1,2
4 Specific
h0
h 0 =<  >
x 1 =<Round,Triangle,Round,Purple,Yes> + h 1 =<Round,Triangle,Round,Purple,Yes>
x 2 =<Square,Square,Square,Green,Yes> - h 2 =<Round,Triangle,Round,Purple,Yes>
x 3 =<Square,Triangle,Round,Yellow,Yes> + h 3 =<?,Triangle,Round,?,Yes>
x 4 =<Round,Triangle,Round,Green,No> - h 4 =<?,Triangle,Round,?,Yes>
x 5 =<Square,Square,Round,Yellow,Yes> + h 5 =<?,?,Round,?,Yes>

CS 5751 Machine Chapter 2 Concept Learning 9

Learning
 Complaints about Find-S
• Cannot tell whether it has learned concept
• Cannot tell when training data inconsistent
• Picks a maximally specific h (why?)
• Depending on H, there might be several!

• How do we fix this?

CS 5751 Machine Chapter 2 Concept Learning 10

Learning
 The List-Then-Eliminate Algorithm
1. Set VersionSpace equal to a list containing every
hypothesis in H
2. For each training example, <x,c(x)>
remove from VersionSpace any hypothesis h for which
h(x) != c(x)
3. Output the list of hypotheses in VersionSpace

• But is listing all hypotheses reasonable?

• How many different hypotheses in our simple
problem?
– How many not involving “?” terms?
CS 5751 Machine Chapter 2 Concept Learning 11
Learning
 Version Spaces
A hypothesis h is consistent with a set of training
examples D of target concept c if and only if
h(x)=c(x) for each training example in D.
Consistent(h, D)  (  x, c( x)   D) h( x)  c( x)
The version space, VSH,D, with respect to hypothesis
space H and training examples D, is the subset of
hypotheses from H consistent with all training
examples in D.
VS H , D  {h  H | Consistent(h, D)}

CS 5751 Machine Chapter 2 Concept Learning 12

Learning
 Example Version Space

G: { <?,?,Round,?,?> <?,Triangle,?,?,?> }

<?,?,Round,?,Yes> <?,Triangle,Round,?,?> <?,Triangle,?,?,Yes>

S: { <?,Triangle,Round,?,Yes> }

CS 5751 Machine Chapter 2 Concept Learning 13

Learning
 Representing Version Spaces
The General boundary, G, of version space VSH,D is
the set of its maximally general members.

The Specific boundary, S, of version space VSH,D is

the set of its maximally specific members.

Every member of the version space lies between

these boundaries
VS H , D  {h  H | (s  S )(g  G )( g  h  s )}
where x  y means x is more general or equal to y
CS 5751 Machine Chapter 2 Concept Learning 14
Learning
 Candidate Elimination Algorithm
G = maximally general hypotheses in H
S = maximally specific hypotheses in H

For each training example d, do

If d is a positive example
Remove from G any hypothesis that does not include d
For each hypothesis s in S that does not include d
Remove s from S
Add to S all minimal generalizations h of s such that
1. h includes d, and
2. Some member of G is more general than h
Remove from S any hypothesis that is more general
than another hypothesis in S

CS 5751 Machine Chapter 2 Concept Learning 15

Learning
 Candidate Elimination Algorithm (cont)
For each training example d, do (cont)
If d is a negative example
Remove from S any hypothesis that does include d
For each hypothesis g in G that does include d
Remove g from G
Add to G all minimal generalizations h of g such that
1. h does not include d, and
2. Some member of S is more specific than h
Remove from G any hypothesis that is less general
than another hypothesis in G

If G or S ever becomes empty, data not consistent (with H)

CS 5751 Machine Chapter 2 Concept Learning 16

Learning
 Example Trace
G1 G0: { <?,?,?,?,?> }
X2=<S,S,S,G,Y> -

G2: { <R,?,?,?,?>, <?,T,?,?,?>, <?,?,R,?,?>, <?,?,?,P,?> } G3

G5 G4: { <?,T,?,?,Y>, <?,?,R,?,Y> } X4=<R,T,R,G,N> -

S5: { <?,?,R,?,Y> } X5=<S,S,R,Y,Y> +

S4 S3: { <?,T,R,?,Y> } X3=<S,T,R,Y,Y> +

S2 S1: { <R,T,R,P,Y> } X1=<R,T,R,P,Y> +

S0: { <Ø,Ø,Ø,Ø,Ø> }
CS 5751 Machine Chapter 2 Concept Learning 17
Learning
 What Training Example Next?

G: { <?,?,Round,?,?> <?,Triangle,?,?,?> }

<?,?,Round,?,Yes> <?,Triangle,Round,?,?> <?,Triangle,?,?,Yes>

S: { <?,Triangle,Round,?,Yes> }

CS 5751 Machine Chapter 2 Concept Learning 18

Learning
 How Should These Be Classified?
G: { <?,?,Round,?,?> <?,Triangle,?,?,?> }

<?,?,Round,?,Yes> <?,Triangle,Round,?,?> <?,Triangle,?,?,Yes>

S: { <?,Triangle,Round,?,Yes> }

? ? ?
CS 5751 Machine Chapter 2 Concept Learning 19
Learning
 What Justifies this Inductive Leap?
+ < Round, Triangle, Round, Purple, Yes >
+ < Square, Triangle, Round, Yellow, Yes >

S: < ?, Triangle, Round, ?, Yes >

Why believe we can classify the unseen?

< Square, Triangle, Round, Purple, Yes > ?

CS 5751 Machine Chapter 2 Concept Learning 20

Learning
 An UN-Biased Learner
Idea: Choose H that expresses every teachable
concept (i.e., H is the power set of X)
Consider H’ = disjunctions, conjunctions, negations
over previous H.
For example:
 ?, Triangle , Round , ?, Yes    Square, Square, ?, Purple, ? 

What are S, G, in this case?

CS 5751 Machine Chapter 2 Concept Learning 21

Learning
 Inductive Bias
Consider
– concept learning algorithm L
– instances X, target concept c
– training examples Dc={<x,c(x)>}
– let L(xi,Dc) denote the classification assigned to the
instance xi by L after training on data Dc.
Definition:
The inductive bias of L is any minimal set of assertions B
such that for any target concept c and corresponding
training examples Dc

(xi  X )[( B  Dc  xi ) L( xi , Dc )]
where A B means A logically entails B
CS 5751 Machine Chapter 2 Concept Learning 22
Learning
 Inductive Systems and Equivalent Deductive Systems

Inductive System
Candidate Classification of
Training examples new instance, or
Elimination
Algorithm "don't know"
New instance
Using Hypothesis
Space H

Equivalent Deductive System

Training examples Classification of
new instance, or
New instance "don't know"
Theorem Prover
Assertion "H
contains hypothesis"

CS 5751 Machine Chapter 2 Concept Learning 23

Learning
Three Learners with Different Biases
1. Rote learner: store examples, classify new
instance iff it matches previously observed
example (don’t know otherwise).

2. Version space candidate elimination algorithm.

3. Find-S

CS 5751 Machine Chapter 2 Concept Learning 24

Learning
 Summary Points
1. Concept learning as search through H
2. General-to-specific ordering over H
3. Version space candidate elimination algorithm
4. S and G boundaries characterize learner’s
uncertainty
5. Learner can generate useful queries
6. Inductive leaps possible only if learner is biased
7. Inductive learners can be modeled by equivalent
deductive systems

CS 5751 Machine Chapter 2 Concept Learning 25

Learning