UNIT1

CONCEPTLEARNING

• Learning involves acquiring general concepts from specific training examples.

Example:People continually learn general concepts or categories such as "bird," "car,"
"situations inwhichIshouldstudymoreinordertopasstheexam,"etc.
• Each such concept can be viewed as describing some subset of objects or events
definedoveralargerset
• Alternatively,eachconceptcanbethoughtofasaBoolean-valuedfunctiondefinedoverthislarger
set. (Example: A function defined over all animals, whose value is true for birds
andfalseforotheranimals).

Definition:Conceptlearning-InferringaBoolean-
valuedfunctionfromtrainingexamplesofitsinputandoutput

ACONCEPTLEARNINGTASK

Consider the example task of learning the target concept "Days on which Aldo
enjoyshis favorite water sport”

Example Sky AirTemp Humidity Wind Water Forecast EnjoySport

1 Sunny Warm Normal Strong Warm Same Yes

2 Sunny Warm High Strong Warm Same Yes

3 Rainy Cold High Strong Warm Change No

4 Sunny Warm High Strong Cool Change Yes

Table:PositiveandnegativetrainingexamplesforthetargetconceptEnjoySport.

The task is to learn to predict the value of EnjoySport for an arbitrary day, based on
thevaluesofitsotherattributes?

Whathypothesisrepresentationisprovidedtothelearner?

• Let’sconsiderasimplerepresentationinwhicheachhypothesisconsistsofaconjunctionofco
nstraints ontheinstanceattributes.
• Leteachhypothesisbeavectorofsixconstraints,specifyingthevaluesofthesixattributesSky,
AirTemp,Humidity,Wind, Water,andForecast.
Foreachattribute,thehypothesis willeither
• Indicatebya"?'that anyvalue isacceptable forthis attribute,
• Specifyasinglerequired value(e.g., Warm)forthe attribute,or
• Indicatebya "Φ"that novalueis acceptable

If some instance x satisfies all the constraints of hypothesis h, then h classifies x as a

positiveexample(h(x)= 1).

The hypothesis that PERSON enjoys his favorite sport only on cold days with high
humidityisrepresentedbytheexpression
(?,Cold,High,?, ?,?)

Themostgeneralhypothesis-thateverydayisapositiveexample-isrepresentedby
(?,?, ?,?,?,?)

The mostspecificpossiblehypothesis-that no dayisapositiveexample-isrepresentedby

(Φ,Φ,Φ,Φ,Φ,Φ)

Notation

• Thesetofitemsoverwhichtheconceptisdefinediscalledthesetofinstances,whichisdenotedbyX.

Example:Xisthesetofallpossibledays,eachrepresentedbytheattributes:Sky,AirTemp,Humidity,
Wind,Water,andForecast

• The concept or function to be learned is called the target concept, which is denoted by
c.ccan be anyBooleanvalued functiondefinedover theinstances X

c:X→{O,1}

Example:Thetargetconceptcorresponds tothevalueof theattributeEnjoySport

(i.e.,c(x)=1 ifEnjoySport=Yes, andc(x)=0ifEnjoySport= No).

• Instancesforwhichc(x)=1arecalledpositiveexamples,ormembersofthetargetconcept.
• Instancesforwhichc(x)=0arecallednegativeexamples,ornon-membersofthetargetconcept.
• Theorderedpair(x,c(x))todescribethetrainingexampleconsistingoftheinstancexanditstarget
conceptvaluec(x).
• Dtodenotethesetofavailabletrainingexamples
• The symbol H to denote the set of all possible hypotheses that the learner may
considerregarding the identity of the target concept. Each hypothesis h in H represents a
Boolean-valuedfunctiondefinedoverX
h:X→{O,1}

Thegoalofthelearneristofind ahypothesis hsuchthat h(x)=c(x)forallxinX.

• Given:
• InstancesX:Possibledays,eachdescribedbytheattributes
• Sky(withpossiblevaluesSunny, Cloudy,and Rainy),
• AirTemp(withvalues WarmandCold),
• Humidity(withvalues NormalandHigh),
• Wind(withvalues StrongandWeak),
• Water(withvaluesWarmand Cool),
• Forecast(withvalues SameandChange).

• Hypotheses H: Each hypothesis is described by a conjunction of constraints on

theattributes Sky, AirTemp, Humidity, Wind, Water, and Forecast. The constraints may
be"?"(anyvalueisacceptable),“Φ”(novalueis acceptable),oraspecificvalue.

• Targetconceptc:EnjoySport:X→{0,l}
• Trainingexamples D: Positiveandnegativeexamplesof thetargetfunction

• Determine:
• AhypothesishinHsuchthath(x)=c(x)forallxinX.

Table:TheEnjoySportconceptlearningtask.

Theinductivelearninghypothesis

Any hypothesis found to approximate the target function well over a sufficiently large set
oftrainingexampleswillalsoapproximatethetargetfunctionwelloverotherunobservedexamples.
CONCEPTLEARNINGASSEARCH

• Conceptlearningcanbeviewedasthetaskofsearchingthroughalargespaceofhypothesesimp
licitlydefinedbythehypothesisrepresentation.
• Thegoalofthissearchistofindthehypothesisthatbestfitsthetrainingexamples.

Example:
Consider the instances X and hypotheses H in the EnjoySport learning task. The attribute
Skyhas three possible values, and AirTemp, Humidity, Wind, Water, Forecast each have
twopossiblevalues,theinstancespaceXcontainsexactly
3.2.2.2.2.2=96distinctinstances
5.4.4.4.4.4 =5120syntacticallydistincthypotheseswithinH.

Every hypothesis containing one or more "Φ" symbols represents the empty set of
instances;thatis,itclassifieseveryinstanceasnegative.
1+(4.3.3.3.3.3)=973. Semanticallydistincthypotheses

General-to-SpecificOrderingofHypotheses

Considerthetwohypotheses
h1=(Sunny,?,?,Strong,?,?)
h2= (Sunny,?,?, ?,?,?)

• Considerthesetsofinstancesthatareclassifiedpositivebyhlandbyh2.
• h2imposesfewerconstraintsontheinstance,itclassifiesmoreinstancesaspositive.So,anyinst
anceclassifiedpositivebyhlwillalsobeclassifiedpositivebyh2.Therefore,h2is
moregeneralthanhl.

Given hypotheses hjand hk, hjis more-general-than or- equal do hkif and only if any
instancethatsatisfieshkalsosatisfies hi

Definition:LethjandhkbeBoolean-valuedfunctionsdefinedoverX.Thenhjismoregeneral-than-
or-equal-tohk(writtenhj≥hk)ifandonlyif

(xX)[(hk(x)=1)→(hj(x)=1)]
 • Inthefigure,theboxontheleftrepresentsthesetXofallinstances,theboxontherightthesetHof
allhypotheses.
• EachhypothesiscorrespondstosomesubsetofX-
thesubsetofinstancesthatitclassifiespositive.
• Thearrows connectinghypothesesrepresentthemore-general-thanrelation, with
thearrowpointing towardthe lessgeneralhypothesis.
• Notethesubsetofinstancescharacterizedbyh2subsumesthesubsetcharacterizedbyhl,hence
h2ismore- general–than h1

FIND-S:FINDINGAMAXIMALLYSPECIFICHYPOTHESIS

FIND-SAlgorithm

1. Initialize h to the most specific hypothesis in H

2. For each positive training instance x
For each attribute constraint aiin h
If the constraintaiis satisfied by x
Then do nothing
Else replace aiin h by the next more general constraint that is satisfied by x
3. Output hypothesis h
To illustrate this algorithm, assume the learner is given the sequence of training
examplesfromthe EnjoySport task

Example Sky AirTemp Humidity Wind Water Forecast EnjoySport

1 Sunny Warm Normal Strong Warm Same Yes
2 Sunny Warm High Strong Warm Same Yes
3 Rainy Cold High Strong Warm Change No
4 Sunny Warm High Strong Cool Change Yes

• ThefirststepofFIND-Sistoinitializeh tothemostspecifichypothesisinH
h-(Ø,Ø,Ø,Ø,Ø,Ø)

• Considerthefirsttrainingexample
x1=<SunnyWarmNormalStrong WarmSame>,+

Observing the first training example, it is clear that hypothesis h is too specific.
Noneofthe"Ø"constraintsinharesatisfiedbythisexample,soeachisreplacedbythenextmore
generalconstraintthatfitstheexample
h1=<SunnyWarmNormalStrong WarmSame>

• Considerthesecond trainingexample
x2=<Sunny, Warm,High,Strong, Warm,Same>,+

The second training example forces the algorithm to further generalize h, this
timesubstituting a "?" in place of any attribute value in h that is not satisfied by the
newexample
h2=<SunnyWarm?Strong WarmSame>

• Considerthethirdtrainingexample
x3=<Rainy,Cold,High,Strong,Warm, Change>,-

Upon encountering the third training the algorithm makes no change to h. The FIND-
Salgorithmsimplyignoreseverynegativeexample.
h3=<SunnyWarm? Strong WarmSame>

• Considerthefourthtrainingexample
x4=<SunnyWarmHighStrong Cool Change>,+

Thefourthexample leadstoafurthergeneralizationofh
h4=<SunnyWarm?Strong ??>
Thekey propertyoftheFIND-Salgorithm
• FIND-Sis guaranteedtooutputthe mostspecifichypothesiswithinH
thatisconsistentwiththepositivetrainingexamples
• FIND-
Salgorithm’sfinalhypothesiswillalsobeconsistentwiththenegativeexamplesprovidedthec
orrecttargetconceptiscontainedinH,andprovidedthetrainingexamplesarecorrect.

UnansweredbyFIND-S

1. Hasthelearnerconvergedtothecorrecttargetconcept?
2. Whypreferthemostspecifichypothesis?
3. Arethetrainingexamplesconsistent?
4. Whatifthereareseveral maximallyspecificconsistenthypotheses?
VERSIONSPACESANDTHECANDIDATE-ELIMINATIONALGORITHM

ThekeyideaintheCANDIDATE-
ELIMINATIONalgorithmistooutputadescriptionofthesetofallhypotheses
consistentwiththetrainingexamples

Representation

Definition:consistent- Ahypothesishisconsistentwithasetof trainingexamples Dif and

onlyifh(x)=c(x)foreachexample(x,c(x)) inD.

Consistent (h, D)  (x, c(x)D) h(x) =

c(x))Notedifferencebetweendefinitionsofconsistentandsatisfies
• An example x is said to satisfy hypothesis h when h(x) = 1, regardless of whether x
isapositive or negativeexample ofthetargetconcept.
• Anexample xissaidtoconsistentwithhypothesis hiff h(x)=c(x)

Definition:versionspace-Theversionspace,denotedVS withrespecttohypothesisspace
H,D
HandtrainingexamplesD,isthesubsetofhypothesesfromHconsistentwiththetrainingexamplesinD
VS {hH|Consistent(h,D)}
H, D

TheLIST-THEN-ELIMINATIONalgorithm

The LIST-THEN-ELIMINATE algorithm first initializes the version space to contain

allhypotheses in H and then eliminates any hypothesis found inconsistent with any
trainingexample.

1. VersionSpacecalistcontainingeveryhypothesisinH
2. Foreach training example,(x,c(x))
removefromVersionSpace anyhypothesishforwhichh(x)≠c(x)
3. OutputthelistofhypothesesinVersionSpace

TheLIST-THEN-ELIMINATEAlgorithm

• List-Then-Eliminateworksinprinciple,solongasversionspaceisfinite.
• However,sinceitrequiresexhaustiveenumerationofallhypothesesinpracticeitisnotfeasibl
e.
AMoreCompact RepresentationforVersionSpaces

The version spaceisrepresentedby itsmost generaland least generalmembers. Thesemembers

form general and specific boundary sets that delimit the version space within
thepartiallyorderedhypothesisspace.

Definition:ThegeneralboundaryG,withrespecttohypothesisspaceHandtrainingdataD,
isthesetofmaximallygeneral membersofH consistentwithD

G{gH|Consistent(g,D)(g'H)[(g'g)Consistent(g', D)]}
g

Definition: ThespecificboundaryS,withrespecttohypothesisspaceHandtrainingdataD,
isthesetofminimallygeneral(i.e.,maximallyspecific)members ofHconsistentwithD.

S{sH|Consistent(s,D)(s'H)[(ss')Consistent(s',D)]}
g

Theorem: Version Spacerepresentationtheorem

Theorem: Let X be an arbitrary set of instances and Let H be a set of Boolean-

valuedhypotheses defined over X. Let c: X →{O, 1} be an arbitrary target concept defined
over
X,andletDbeanarbitrarysetoftrainingexamples{(x,c(x))).ForallX,H,c,andDsuchthatSandGare
well defined,

VS ={hH|(sS)(gG)(ghs)}
H,D g g

ToProve:
1. Everyhsatisfyingtherighthand sideof theaboveexpressionisinVS
H, D
2. EverymemberofVS satisfiestheright-handsideof theexpression
H, D

Sketchof proof:
1. letg,h,sbearbitrarymembersof G,H,Srespectivelywithgghgs
• BythedefinitionofS, smustbesatisfiedbyallpositiveexamplesinD.Becausehgs,
h mustalsobesatisfiedbyallpositiveexamplesinD.
• BythedefinitionofG,gcannotbesatisfiedbyanynegativeexampleinD,andbecauseg g h h
cannot be satisfied by any negative example in D. Because h is satisfied by allpositive
examples in D and by no negative examples in D, h is consistent with D,
andthereforehisamemberofVSH,D.
2. It can be proven by assuming some h in VSH,D,that does not satisfy the right-hand
sideoftheexpression,thenshowingthatthis leadstoaninconsistency
CANDIDATE-ELIMINATIONLearningAlgorithm

TheCANDIDATE-
ELIMINTIONalgorithmcomputestheversionspacecontainingallhypothesesfromH that
areconsistentwithan observed sequenceof trainingexamples.

Initialize G to the set of maximally general hypotheses in

HInitialize S to the set of maximally specific hypotheses in
HForeachtrainingexampled,do
• Ifdisapositiveexample
• RemovefromG anyhypothesisinconsistentwithd
• ForeachhypothesissinSthatis notconsistentwithd
• RemovesfromS
• AddtoSallminimalgeneralizationshof ssuchthat
• hisconsistentwithd,andsomememberof G is moregeneralthanh
• RemovefromSanyhypothesisthatismoregeneralthananotherhypothesisinS

• Ifdisanegativeexample
• RemovefromSanyhypothesisinconsistentwithd
• ForeachhypothesisginGthatis notconsistentwithd
• RemovegfromG
• AddtoGall minimalspecializationshof gsuchthat
• hisconsistentwithd,andsomememberofSismorespecificthanh
• RemovefromGanyhypothesisthatislessgeneralthananotherhypothesisinG

CANDIDATE-ELIMINTIONalgorithmusingversionspaces

AnIllustrativeExample

Example Sky AirTemp Humidity Wind Water Forecast EnjoySport

1 Sunny Warm Normal Strong Warm Same Yes
2 Sunny Warm High Strong Warm Same Yes
3 Rainy Cold High Strong Warm Change No
4 Sunny Warm High Strong Cool Change Yes
CANDIDATE-
ELIMINTIONalgorithmbeginsbyinitializingtheversionspacetothesetofallhypothesesinH;

Initializing theGboundarysettocontainthemostgeneralhypothesisinH
G0?, ?,?, ?,?,?

InitializingtheSboundarysettocontainthemostspecific(leastgeneral)hypothesis
S0,,,,,

• When the first training example is presented, the CANDIDATE-ELIMINTION

algorithmchecks the S boundary and finds that it is overly specific and it fails to cover the
positiveexample.
• The boundary is therefore revised by moving it to the least more general hypothesis
thatcoversthisnew example
• No update of the G boundary is needed in response to this training example because
Gocorrectlycovers thisexample

• When the second training example is observed, it has a similar effect of generalizing
SfurthertoS2,leavingGagainunchangedi.e.,G2=G1=G0
 • Considerthethirdtrainingexample.ThisnegativeexamplerevealsthattheGboundaryof the
version space is overly general, that is, the hypothesis in G incorrectly
predictsthatthisnewexampleisapositiveexample.
• The hypothesis in the G boundary must therefore be specialized until it
correctlyclassifiesthisnewnegativeexample

Given that there are six attributes that could be specified to specialize G2, why are there
onlythreenewhypothesesinG3?
For example, the hypothesis h = (?, ?, Normal, ?, ?, ?) is a minimal specialization of
G2thatcorrectlylabelsthenewexampleasanegativeexample,butitisnotincludedinG3.The
reason this hypothesis is excluded is that it is inconsistent with the
previouslyencountered positive examples

• Considerthefourthtrainingexample.
 • This positive example further generalizes the S boundary of the version space. It
alsoresults in removing one member of the G boundary, because this member fails
tocoverthenew positiveexample

Afterprocessingthesefourexamples,theboundarysets S4and
G4delimittheversionspaceofallhypothesesconsistentwiththeset of
incrementallyobservedtrainingexamples.
INDUCTIVEBIAS

Thefundamentalquestionsforinductiveinference

1. Whatifthetargetconceptisnotcontainedinthehypothesisspace?
2. Canweavoidthisdifficultybyusingahypothesisspacethatincludeseverypossiblehypothe
sis?
3. How does the size of this hypothesis space influence the ability of the algorithm
togeneralizetounobservedinstances?
4. How does the size of the hypothesis space influence the number of training
examplesthat mustbeobserved?

These fundamental questions are examined in the context of the CANDIDATE-

ELIMINTIONalgorithm

ABiasedHypothesis Space

• Suppose the target concept is not contained in the hypothesis space H, then
obvioussolutionis toenrichthehypothesisspacetoincludeeverypossiblehypothesis.
• ConsidertheEnjoySportexampleinwhichthehypothesisspaceisrestrictedtoincludeonlyco
njunctionsofattributevalues.Becauseofthisrestriction,thehypothesisspaceisunabletorepr
esent evensimpledisjunctivetargetconceptssuchas
"Sky= SunnyorSky=Cloudy."
• The following three training examples of disjunctive hypothesis, the algorithm
wouldfindthattherearezerohypothesesinthe versionspace

SunnyWarmNormalStrongCoolChange Y
CloudyWarmNormalStrongCoolChange Y
RainyWarmNormalStrongCoolChange N

• IfCandidateEliminationalgorithmisapplied,thenitendupwithemptyVersionSpace.Afterfi
rsttwotrainingexample
S=?WarmNormalStrongCoolChange

• This new hypothesis is overly general and it incorrectly covers the third
negativetrainingexample!SoHdoesnotincludetheappropriatec.
• Inthiscase,amore expressivehypothesisspaceisrequired.
AnUnbiasedLearner

• ThesolutiontotheproblemofassuringthatthetargetconceptisinthehypothesisspaceHis to
provide a hypothesis space capable of representing every teachable concept that
isrepresenting everypossiblesubsetoftheinstancesX.
• ThesetofallsubsetsofasetXiscalledthepower setofX

• IntheEnjoySportlearningtaskthesizeoftheinstancespaceXofdaysdescribedbythesix
attributesis96instances.
• Thus,thereare296distincttargetconceptsthatcouldbedefinedoverthisinstancespaceandlearner
mightbecalledupontolearn.
• Theconjunctivehypothesisspaceisabletorepresentonly973ofthese-
abiasedhypothesisspaceindeed

• LetusreformulatetheEnjoySportlearningtaskinanunbiasedway
bydefininganewhypothesis spaceH'thatcanrepresenteverysubsetofinstances
• Thetargetconcept"Sky=SunnyorSky=Cloudy"couldthenbedescribedas

(Sunny,?,?,?,?,?)v (Cloudy,?, ?,?,?,?)

TheFutility ofBias-FreeLearning

Inductivelearningrequiressomeformofpriorassumptions,or inductivebias

Definition:
ConsideraconceptlearningalgorithmLforthesetofinstancesX.
• LetcbeanarbitraryconceptdefinedoverX
• LetD ={(x,c(x))}beanarbitrarysetoftrainingexamplesofc.
c
• LetL(x,D)denotetheclassificationassignedtotheinstancex byLafter trainingon
i c i
thedataD.
c
• TheinductivebiasofLisanyminimalsetofassertionsBsuchthatforanytargetconceptcand
correspondingtrainingexamples D
c

• (xiX)[(BDcxi)├L(xi,Dc)]
Thebelowfigureexplains
• Modellinginductivesystemsbyequivalentdeductivesystems.
• The input-output behavior of the CANDIDATE-ELIMINATION algorithm using
ahypothesis space H is identical to that of a deductive theorem prover utilizing
theassertion"Hcontainsthetargetconcept."Thisassertionisthereforecalledtheinductivebia
softheCANDIDATE-ELIMINATIONalgorithm.
• Characterizinginductivesystemsbytheirinductivebiasallowsmodellingthembytheirequiv
alentdeductivesystems.Thisprovidesaway
tocompareinductivesystemsaccordingtotheirpoliciesforgeneralizingbeyond
theobservedtrainingdata.