Machine Learning

Lecture 02
Concept Learning

Dr. Rao Muhammad Adeel Nawab

How to Work

Dr. Rao Muhammad Adeel Nawab 2

Power of Dua

Dr. Rao Muhammad Adeel Nawab 3

Dua – Take Help from Allah before starting any task

Dr. Rao Muhammad Adeel Nawab 4

Course Focus
Mainly get EXCELLENCE in two things
1. Become a great human being
2. Become a great Machine Learning Engineer

To become a great human being

Get sincere with yourself
When you get sincere with yourself your ‫ ﺧﻠﻭﺕ‬and ‫ ﺟﻠﻭﺕ‬is the
same
Dr. Rao Muhammad Adeel Nawab 5
Lecture Outline
What is Concept Learning?
General-to-Specific Ordering of Hypotheses
FIND -S: Finding a Maximally Specific Hypothesis
Version Spaces
The Candidate Elimination Algorithm
Inductive Bias

Reading:
Chapter 2 of Mitchell

Dr. Rao Muhammad Adeel Nawab 6

What is Concept Learning
Much learning is acquiring general concepts/categories
from specific examples
– house
– game
– driving situations in which I should brake sharply
– credit-worthy loan applicant

Dr. Rao Muhammad Adeel Nawab 7

What is Concept Learning (cont…)
Such concepts can be viewed as describing
– a subset over a larger set
∗ houses are a particular subset of human artifacts
– a Boolean-valued function over larger set
∗ a function over human artifacts which is true for houses
and false for other artifacts

Dr. Rao Muhammad Adeel Nawab 8

What is Concept Learning (cont…)
Definition
Concept learning: Inferring a Boolean-valued function
from training examples of its input and output. (Mitchell,
p.21)
Concept learning is also known as binary classification.

Dr. Rao Muhammad Adeel Nawab 9

Definition of Function in Mathematics
Function is a relation between
a set of inputs and a set of permissible outputs
with the property that each input is related to exactly one
output
Example - function that relates each real number x to its
square x2

Dr. Rao Muhammad Adeel Nawab 10

 Binary Classification - Learning
Problem

Dr. Rao Muhammad Adeel Nawab 11

Binary Classification Example: SICK (SK) or Not
Attributes:
Temperature (T) Blood Pressure (BP)

Low (L)
High (H)
High (H) Low (L)
Normal (N)
Normal (N)

Dr. Rao Muhammad Adeel Nawab 12

Instance Space (X)
X T BP SK
x1 L L -
x2 L N -
x3 L H -
x4 N L -
x5 N N -
x6 N H -
x7 H L -
x8 H N -
x9 H H -
Dr. Rao Muhammad Adeel Nawab 13
Concept as a Function
The solution to any problem is a function that converts its
inputs to corresponding outputs.
A concept itself is merely a function, which we don’t know
yet.
We do have some of the inputs and their corresponding
outputs.
From these input-output pair we would try to find the
generic function that generates these results.
Dr. Rao Muhammad Adeel Nawab 14
Concept Space (C)
One of the possible concepts for the X T BP SK
concept SICK might be enumerated in x1 L L 0
the following table: x2 L N 0
x3 L H 1
x4 N L 0
x5 N N 0
x6 N H 1
x7 H L 1
x8 H N 1
x9 H H 1
Dr. Rao Muhammad Adeel Nawab 15
Concept Space
But there are a lot of other possibilities besides this one
The question is:
How many total concepts can be generated out of this given
situation?
The answer is: 2|X|
Here 29, since |X| = 9
In short the true concept SK is a function defined over the
attributes T and BP, such that it gives a 0 or a 1 as output for
each of the 9 instances xi belonging to Instance Space X.
Dr. Rao Muhammad Adeel Nawab 16
Concept Space (C)
For any arbitrary concept C, if the X T BP SK
following table is the format for x1 L L C(x1)
representation of each output x2 L N C(x2)
corresponding to each instance. x3 L H C(x3)
x4 N L C(x4)
x5 N N C(x5)
x6 N H C(x6)
x7 H L C(x7)
x8 H N C(x8)
x9 H H C(x9)
Dr. Rao Muhammad Adeel Nawab 17
Concept Space (C)
Since we don’t know the true X T BP SK
concept yet, so there might be x11 L L 0 1
concepts which can produce 29 x22 L N 10
different outputs, such as: x33 L H 01
x44 N L 01
x55 N N 01
CCC42When
1 When C(x
9 CC When
23When
C(x )C(x
to C(x
C(x ) = 01
)) == 11
1)11and12C(x992) = 1 x66 N H 01
x77 H L 01
x88 H N 01
x99 H H 01
Dr. Rao Muhammad Adeel Nawab 18
So what is a Concept?
Concept is nothing more than a function whose
independent variables are the attributes.
In this case T and BP
Maybe the true concept is some complicated
arrangement of conjunctions and disjunctions like:
C = < T = H AND BP = H OR
T = N AND BP = H OR
T = H AND BP = N >

Dr. Rao Muhammad Adeel Nawab 19

Instance Space (X)
X T BP SK
x1 L L -
x2 L N -
x3 L H -
x4 N L -
x5 N N -
x6 N H -
x7 H L -
x8 H N -
x9 H H -
Dr. Rao Muhammad Adeel Nawab 20
Concept Space (C)
One of the possible concepts for the X T BP SK
concept SICK might be enumerated in x1 L L 0
the following table: x2 L N 0
x3 L H 1
x4 N L 0
x5 N N 0
x6 N H 1
x7 H L 1
x8 H N 1
x9 H H 1
Dr. Rao Muhammad Adeel Nawab 21
Concept Space
For any arbitrary concept C, the following table is another
format for representation of each output corresponding to
each instance.
Each c(xi) is the output of the concept c for that particular
instance.
C(x3) C(x6) C(x9)
C(x2) C(x5) C(x8)
C(x1) C(x4) C(x7)

Dr. Rao Muhammad Adeel Nawab 22

Concept Space
There two attributes and each can have 3 possible values,
there are 29 unique combinations or functions.

0 0 0 0 0 0 0 0 0 0 0 0 1 1 1
0 0 0 0 0 0 1 0 0 1 0 0 1 1 1
0 0 0 1 0 0 0 0 0 1 0 0 1 1 1

C1 C2 C3 C4 C29

Dr. Rao Muhammad Adeel Nawab 23

Training Data Set (D)

D T BP SK
x1 N L 1
x2 L N 0
x3 N N 0

Dr. Rao Muhammad Adeel Nawab 24

Hypothesis Space (H)
The learner has to apply some hypothesis, that introduces
a search bias to reduce the size of the concept space.
This reduced concept space becomes the hypothesis
space.
For example, the most common bias is one that uses the AND
relationship between the attributes
In other words the hypothesis space uses the conjunction
(AND) of the attributes T and BP
i.e. h = <T, BP>
Dr. Rao Muhammad Adeel Nawab 25
Representation of Hypothesis
Many possible representations.
Here, Hypothesis h is a conjunction (AND) of constraints
on attributes
Each constraint can be:
A specific value : e.g. BP = H
A don’t care value (any of possible values): e.g. BP = ?
No value allowed (null hypothesis Ø): e.g. BP = Ø

Dr. Rao Muhammad Adeel Nawab 26

Hypothesis Space
H denotes the hypothesis space
Here it is the conjunction of attributes T and BP
If written in English it would mean:
H = <t, bp>: IF “Temperature” = T AND “Blood Pressure” = BP
Then H=1
Otherwise H=0
In other words, the function gives a 1 output for all
conjunctions of T and BP, e.g., H and H, H and L, H and M,
etc.
Dr. Rao Muhammad Adeel Nawab 27
Hypothesis Space
h = <H, H>: <T, BP>

BP
H 0 0 1
N 0 0 0
L 0 0 0
L N H T

Dr. Rao Muhammad Adeel Nawab 28

Hypothesis Space
h = <L, L>: <T, BP> Notice that this is the C2 that
we discussed earlier in the
BP Concept Space section
H 0 0 0
N 0 0 0 0 0 0
0 0 0
L 1 0 0
1 0 0
L N H T

Dr. Rao Muhammad Adeel Nawab 29

Hypothesis Space
H = <T, BP>
Where T and BP can take on five values
H, N, L (High, Normal, Low)
Also ? and Ø
? means that for all values of the input H = 1 (don’t care)
Ø means that there will be no value for which H will be 1

Dr. Rao Muhammad Adeel Nawab 30

Hypothesis Space
h1 = <?, ?>: <T, BP> For any value of T and BP,
the person is always sick.
BP
H 1 1 1
N 1 1 1
L 1 1 1
L N H T

Dr. Rao Muhammad Adeel Nawab 31

Hypothesis Space
h2 = <?, H>: <T, BP> For any value of T AND
BP = High, the person is
BP sick, Irrespective of
H 1 1 1 temperature, if BP is
High, the person is sick.
N 0 0 0
L 0 0 0
L N H T

Dr. Rao Muhammad Adeel Nawab 32

Hypothesis Space
h3 = <Ø,Ø >: <T, BP> For no value of T AND BP,
the person is sick, The
BP person is never sick.
H 0 0 0
N 0 0 0
L 0 0 0
L N H T

Dr. Rao Muhammad Adeel Nawab 33

Hypothesis Space
Having said all this, how does this still reduce the
hypothesis space to 17?
Well it’s simple, now each attribute T and BP can take 5
values each: L, N, H, ? and Ø
So there are 5 x 5 = 25 total number of possible
hypotheses

Dr. Rao Muhammad Adeel Nawab 34

Hypothesis Space
Now this is a tremendous reduction from (29 ) 512 to 25
This number can be reduced further
There are redundancies within these 25 hypotheses
Caused by Ø

Dr. Rao Muhammad Adeel Nawab 35

Hypothesis Space
These redundancies are caused by Ø
Whenever there is Ø in any of the inputs and we are
considering conjunctions the output will always be 0
If there’s this ‘Ø’ in the T or the BP or both, we’ll have the
same hypothesis as the outcome is always, all zeros
For a ?: we will either get a full column of 1’s, or a full row
of 1’s in the concept matrix representation.
For both ?: all 1’s
Dr. Rao Muhammad Adeel Nawab 36
Hypothesis Space
h = <N,Ø >: <T, BP> h = < Ø,Ø >: <T, BP>

BP BP
H 0 0 0 H 0 0 0
N 0 0 0 N 0 0 0
L 0 0 0 L 0 0 0
L N H T L N H T

Dr. Rao Muhammad Adeel Nawab 37

 “Enjoy Sport” Learning Problem

Dr. Rao Muhammad Adeel Nawab 38

Example: Concept- Enjoy Sport
Concept Learning is the process of inferring a boolean-
valued function from training examples of its input and
output.
Concept:
Days on which Fred enjoys his favourite water sports
Task:
predict the value of Enjoy Sport for an arbitrary day based on
the values of the other attributes

Dr. Rao Muhammad Adeel Nawab 39

Example: Concept- Enjoy Sport
Attributes and Possible Values
Sr. # Attribute Possible Values
1 Sky Sunny, Cloudy, Rainy
2 Temp Warm, Cold
3 Humid Normal, High
4 Wind Strong, Weak
5 Water Warm, Cold
6 Forecast Same, Change
7 EnjoySport Yes, No

Dr. Rao Muhammad Adeel Nawab 40

Example: Concept- Enjoy Sport
Training Examples Attributes/ Features
Sky Temp Humid Wind Water Forecast EnjoySport
Sunny Warm Normal Strong Warm Same Yes
Sunny Warm High Strong Warm Same Yes
Rainy Cold High Strong Warm Change No
Sunny Warm High Strong Cool Change Yes

Class Attribute
Instance Input Attributes
Dr. Rao Muhammad Adeel Nawab 41
Representing Hypothesis
Many possible representations.
Here, Hypothesis h is a conjunction (AND) of constraints
on attributes
Each constraint can be:
A specific value : e.g. Water = Warm
A don’t care value (any of possible values): e.g. Water = ?
No value allowed (null hypothesis Ø): e.g. Water=Ø

Dr. Rao Muhammad Adeel Nawab 42

Representing Hypothesis
Example: hypothesis h
< Sky Temp Humid Wind Water Forecast>
< Sunny ? ? Strong ? Same > ? = Any value
Most general hypothesis h is acceptable
< Sky Temp Humid Wind Water Forecast >
< ? ? ? ? ? ? > ∅ = No value
Most specific hypothesis h is acceptable
< Sky Temp Humid Wind Water Forecast >
< ∅ ∅ ∅ ∅ ∅ ∅ >

Dr. Rao Muhammad Adeel Nawab 43

Prototypical Concept Learning Task
Given:
Instances X : Possible days, each decribed by the attributes
Sky, Temp, Humidity, Wind, Water, Forecast
Target function c: EnjoySport: X → {0,1}
Hypotheses H: conjunction of literals e.g.
< Sunny ? ? Strong ? Same >
Training examples D: positive and negative examples of the
target function:
< x1, c(x1) >, … , < xn, c(xn) >

Dr. Rao Muhammad Adeel Nawab 44

Prototypical Concept Learning Task
Determine:
A hypothesis h in H such that h(x) = c(x) for all x in D.

Inductive Learning Hypothesis:

Any hypothesis found to approximate the target function well
over the training examples, will also approximate the target
function well over the unobserved examples.

Dr. Rao Muhammad Adeel Nawab 45

Number of Instances, Concepts, Hypotheses
#distinct instances Sr. Attribu Possible Possible
3*2*2*2*2*2 = 96 # te Values Values in #
1 Sky Sunny, 3
#distinct concepts Cloudy, Rainy

296 2 Temp Warm, Cold 2

3 Humid Normal, High 2
4 Wind Strong, Weak 2
5 Water Warm, Cold 2
6 Forecast Same, Change 2

Dr. Rao Muhammad Adeel Nawab 46

Number of Instances, Concepts, Hypotheses
#syntactically distinct Sr. Attribute Possible Values Possible
hypotheses # Values in #
5*4*4*4*4*4=5120 1 Sky Sunny, Cloudy, Rainy, ?, ∅ 5
2 Temp Warm, Cold, ?, ∅ 4
3 Humid Normal, High, ?, ∅ 4
#semantically distinct 4 Wind Strong, Weak, ?, ∅ 4
hypotheses 5 Water Warm, Cold, ?, ∅ 4

1+4*3*3*3*3*3=973 6 Forecast Same, Change, ?, ∅ 4

< ∅, ∅, ∅, ∅, ∅, ∅ > same for all

Dr. Rao Muhammad Adeel Nawab 47
 General-to-Specific Ordering and
Find-S Algorithm

Dr. Rao Muhammad Adeel Nawab 48

Instance, Hypotheses and More-General-Than Ordering
Instances Hypotheses
Specific

x1 h3 h1
h2 ≥ h1 h2
x2
h2 ≥ h3 General

x1 = < Sunny, Warm, High, Strong, Cool, Same> h1 = < Sunny, ?, ?, Strong, ?, ? >
x2 = < Sunny, Warm, High, Light, Warm, Same> h2 = < Sunny, ?, ?, ?, ?, ? >
h3 = < Sunny, ?, ?, ?, Cool, ?>

Dr. Rao Muhammad Adeel Nawab 49

More-General-Than Ordering – Formal Definition
Let hj and hk be Boolean-valued functions defined over X.
hj is more-general-than-or-equal-to hk (hj ≥g hk) iff
(∀
∀x ∈ X) (hk(x) = 1 → hj(x) = 1)
hj is (strictly) more-general-than hk (hj >g hk) iff
hj ≥g hk ∧ hk !≥g hj.
hk is more-specific-than hj iff
hj is more-general-than hk.
≥g is a partial ordering over H,
i.e., it is reflexive, antisymmetric, transitive and some pairs may
not be ordered.

Dr. Rao Muhammad Adeel Nawab 50

General to Specific Order
Consider two hypotheses:
h1 = < Sunny, ?, ?, Strong, ?, ?> , h2 = < Sunny, ?, ?, ?, ?, ?>
Set of instances covered by h1 and h2:
h2 imposes fewer constraints than h1 and therefore classifies more
instances x as positive h(x)=1.
Definition:
Let hj and hk be boolean-valued functions defined over X. Then hj is
more general than or equal to hk (hj ≥ hk) iff
∀x ∈ X : [ (hk(x) = 1) → (hj(x) = 1) ]
The relation ≥ imposes a partial order over the hypothesis space H that is
utilized by many concept learning methods.

Dr. Rao Muhammad Adeel Nawab 51

Find-S Algorithm
Initialize h to the most specific hypothesis in H
For each positive training instance x
For each attribute constraint ai in h
If the constraint ai in h is satisfied by x
then do nothing
else replace ai in h by the next more general constraint that
is satisfied by x
Output hypothesis h

Dr. Rao Muhammad Adeel Nawab 52

Hypothesis Space Search by Find-S
Instances Hypotheses
x3 h0 Specific
h1
x1
h2,3
x2
x4 h4
General

x1=<Sunny,Warm,Normal,Strong,Warm,Same> + h0=< Ø, Ø, Ø, Ø, Ø, Ø,>

x2=<Sunny,Warm,High,Strong,Warm,Same> + h1=< Sunny,Warm,Normal,Strong,Warm,Same>

x3=<Rainy,Cold,High,Strong,Warm,Change> - h2,3=< Sunny,Warm,?, Strong,Warm,Same>

x4=<Sunny,Warm,High,Strong,Cool,Change> + h4=< Sunny,Warm,?, Strong,?,?>

Dr. Rao Muhammad Adeel Nawab 53

Properties of Find-S
Hypothesis space is described by conjunctions of
attributes.
Find-S will output the most specific hypothesis within H
that is consistent with the positve training examples.
The output hypothesis will also be consistent with the
negative examples, provided the target concept is
contained in H.

Dr. Rao Muhammad Adeel Nawab 54

Complaints about Find-S
Can’t tell whether it has learned a concept.
Can’t tell when training data inconsistent.
Picks a most specific h Why?
Depending on H, there might be several most specific
hypothesis!
Is it possible to describe all hypothesis consistent with the
training data?
Version spaces and the candidate-elimination algorithm.

Dr. Rao Muhammad Adeel Nawab 55

 Version Spaces

Dr. Rao Muhammad Adeel Nawab 56

Version Spaces
Idea: Compute the set of all hypothesis consistent with
the training examples.
A hypothesis h is consistent with a set of training examples D
of target concept iff h(x) = c(x) for each training example <x,
c(x)> in D.
Consistent (h, D) := ∀ < x, c(x) > ∈ D h (x) = c (x)
The version space VSH,D with respect to hypothesis space
H, and training set D, is the subset of hypotheses from H
consistent with all training examples:
VSH,D = { h ∈ H | Consistent (h, D) }
Dr. Rao Muhammad Adeel Nawab 57
Version Spaces
How can we represent a version space

List-Then Eliminate

Dr. Rao Muhammad Adeel Nawab 58

List-Then Eliminate Algorithm
VersionSpace ← a list containing every hypothesis in H
For each training example < x, c(x) > do:
Remove from VersionSpace any hypothesis that is
inconsistent with the training example h(x) ≠ c(x)
Output the list of hypotheses in VersionSpace

We need to find a more

compact representation.

Dr. Rao Muhammad Adeel Nawab 59

 Example Version Space: Enjoy Sport
S: { < Sunny, Warm, ?, Strong, ?, ? > }

< Sunny, ?, ?, Strong, ?, ? > < Sunny, Warm, ?, ?, ?, ? > < ?, Warm, ?, Strong, ?, ? >

G: { < Sunny, ?, ?, ?, ?, ? > , < ?, Warm, ?, ?, ? > }

x1 = < Sunny Warm Normal Strong Warm Same > +
x2 = < Sunny Warm High Strong Warm Same > +
x3 = < Rainy Cold High Strong Warm Change > -
x4 = < Sunny Warm High Strong Cool Change > +
Dr. Rao Muhammad Adeel Nawab 60
Representing Version Spaces
The general boundary G of version space VSH,D is the set of
maximally general members.
The specific boundary S of version space VSH,D is the set of
maximally specific members.
Every member of the version space lies between these
boundaries
VSH,D = {h ∈ H| (∃
∃ s ∈ S) (∃
∃ g ∈ G) (g ≥g h ≥g s)
where x ≥g y means x is more general or equal than y

Dr. Rao Muhammad Adeel Nawab 61

 Candidate Elimination Algorithm

Dr. Rao Muhammad Adeel Nawab 62

Candidate Elimination Algorithm
G ← maximally general hypotheses in H
S ← maximally specific hypotheses in H
For each training example d = < x, c(x) > do:
If d is a positive example
Remove from G any hypothesis that is inconsistent with d
For each hypothesis s in S that is not consistent with d
remove s from S.
Add to S all minimal generalizations h of s such that
h consistent with d
Some member of G is more general than h
Remove from S any hypothesis that is more general than another
hypothesis in S
Dr. Rao Muhammad Adeel Nawab 63
Candidate Elimination Algorithm
If d is a negative example
Remove from S any hypothesis that is inconsistent with d
For each hypothesis g in G that is not consistent with d
remove g from G.
Add to G all minimal specializations h of g such that
h consistent with d
Some member of S is more specific than h
Remove from G any hypothesis that is less general than another
hypothesis in G

Dr. Rao Muhammad Adeel Nawab 64

Example Candidate Elimination
S: { < ∅, ∅, ∅, ∅, ∅, ∅ > }

G: { < ?, ?, ?, ?, ?, ? > }

x1 = < Sunny Warm Normal Strong Warm Same > +

S: {< Sunny Warm Normal Strong Warm Same > }

G: { < ?, ?, ?, ?, ?, ? > }
x2 = < Sunny Warm High Strong Warm Same > +
S: { < Sunny Warm ? Strong Warm Same > }

G: { < ?, ?, ?, ?, ?, ? > }
Dr. Rao Muhammad Adeel Nawab 65
Example Candidate Elimination
S: { < Sunny Warm ? Strong Warm Same > }
G: { < ?, ?, ?, ?, ?, ? > }
x3 = < Rainy Cold High Strong Warm Change > -
S: {< Sunny Warm ? Strong Warm Same >}

G: { < Sunny,?,?,?,?,? >, < ?,Warm,?,?,? >, < ?,?,?,?,?,Same > }

x4 = < Sunny Warm High Strong Cool Change > +
S: { < Sunny Warm ? Strong ? ? > }

G: { < Sunny,?,?,?,?,? > , < ?,Warm,?,?,? > }

Dr. Rao Muhammad Adeel Nawab 66
Example Candidate Elimination
Instance space:
integer points in the x,y plane
hypothesis space :
Rectangles, that means hypotheses
are of the form a ≤ x ≤ b , c ≤ y ≤ d.

Dr. Rao Muhammad Adeel Nawab 67

Classification of New Data
S: { < Sunny, Warm, ?, Strong, ?, ?> }

< Sunny, ?, ?, Strong, ?, ? > < Sunny, Warm, ?, ?, ?, ?> < ?, Warm, ?, Strong, ?, ? >

G: { < Sunny,?,?,?,?,?>, <?,Warm,?,?,? > }

x5 = <Sunny Warm Normal Strong Cool Change> + 6/0
x6 = <Rainy Cold Normal Light Warm Same> - 0/6
x7 = <Sunny Warm Normal Light Warm Same> ? 3/3
x8 = <Sunny Cold Normal Strong Warm Same> ? 2/4

Dr. Rao Muhammad Adeel Nawab 68

Inductive Leap
+ <Sunny Warm Normal Strong Cool Change>
+ <Sunny Warm Normal Light Warm Same>
S : <Sunny Warm Normal ? ? ?>
How can we justify to classify the new example as
+ <Sunny Warm Normal Strong Warm Same>
Bias: We assume that the hypothesis space H
contains the target concept c. In other words that
c can be described by a conjunction of literals.
Dr. Rao Muhammad Adeel Nawab 69
Biased Hypothesis Space
Our hypothesis space is unable to represent a simple
disjunctive target concept : (Sky=Sunny) v (Sky=Cloudy)

x1 = <Sunny Warm Normal Strong Cool Change> +

x2 = <Cloudy Warm Normal Strong Cool Change> +
S : { <?, Warm, Normal, Strong, Cool, Change> }

x3 = <Rainy Warm Normal Light Warm Same> -

S:{ }
Dr. Rao Muhammad Adeel Nawab 70
Unbiased Learner
Idea:
Choose H that expresses every teachable concept, that means
H is the set of all possible subsets of X called the power set
P(X)
|X|=96, |P(X)|=296 ~ 1028 distinct concepts
H = disjunctions, conjunctions, negations
e.g. <Sunny Warm Normal ? ? ?> v <? ? ? ? ? Change>
H surely contains the target concept.

Dr. Rao Muhammad Adeel Nawab 71

Unbiased Learner
What are S and G in this case?
Assume positive examples (x1, x2, x3) and negative examples (x4,
x5)
S : { (x1 v x2 v x3) } G : { ¬ (x4 v x5) }
The only examples that are classified are the training examples
themselves. In other words in order to learn the target concept
one would have to present every single instance in X as a training
example.
Each unobserved instance will be classified positive by precisely
half the hypothesis in VS and negative by the other half.
Dr. Rao Muhammad Adeel Nawab 72
Futility of Bias-Free Learning
A learner that makes no prior assumptions regarding the
identity of the target concept has no rational basis for
classifying any unseen instances.

Dr. Rao Muhammad Adeel Nawab 73

Inductive Bias
Consider:
Concept learning algorithm L
Instances X, target concept c
Training examples Dc={<x,c(x)>}
Let L(xi,Dc ) denote the classification assigned to instance xi by L
after training on Dc.
Definition:
The inductive bias of L is any minimal set of assertions B such that
for any target concept c and corresponding training data Dc
(∀
∀xi ∈ X)[B ∧ Dc ∧ xi] |-- L(xi, Dc)
Where A |-- B means that A logically entails B.

Dr. Rao Muhammad Adeel Nawab 74

Inductive Systems and Equivalent Deductive Systems

Training examples Classification of new

Candidate Elimination instance or don’t know
New instance Algorithm using
hypothesis space H

Equivalent Deductive System

Training examples Classification of new
instance or don’t know
New instance Theorem prover
Assertion H contains
target concept
Dr. Rao Muhammad Adeel Nawab 75
Three Learners with Different Biases
Rote learner:
Store examples classify x if and only if it matches a previously
observed example.
No inductive bias
Find-S:
Bias: The hypothesis space contains the target concept and all
instances are negative instances unless the opposite is entailed by
its other knowledge.
Version space candidate elimination:
Bias: The hypothesis space contains the target concept.

Dr. Rao Muhammad Adeel Nawab 76

ADD

Algorithm Order Strategy N/P

FIND-S Specific-to-general Top-down Positive

LIST-THEN-ELIMINATE General-to-Specific Bottom-up Negative

CANDIDATE-ELIMINATION Bi-directional Bi-directional Both

Dr. Rao Muhammad Adeel Nawab 77

 How To Become a Great Human
Being

Dr. Rao Muhammad Adeel Nawab 78

Balanced Life is Ideal Life
Get Excellence in five things
1. Health
2. Spirituality
3. Work
4. Friend
5. Family
A Journey from BIGNNER to EXCELLENCE
You must have a combination of five things with different
variations. However, aggregate will be same.
Dr. Rao Muhammad Adeel Nawab 79
Excellence
1. Health
I can run (or brisk walk) 5 kilometers in one go
I take 7-9 hours sleep per night (TIP: Go to bed at 10pm)
I take 3 meals of balanced diet daily
2. Spirituality
ٔ ٔ
. ‫م‬ ‫ا‬ ‫د‬

Dr. Rao Muhammad Adeel Nawab 80

Excellence
1. Work
Become an authority in your field
For example - Dr. Abdul Qadeer Khan Sb is an authority in research
2. Friend
Have a DADDU YAR in life to drain out on daily basis
3. Family
1. Take Duas of Parents and elders by doing their ‫ ﺧﺩﻣﺕ‬and ‫ﺍﺩﺏ‬
2. Your wife/husband should be your best friend
3. Be humble and kind to kids, subordinates and poor people
Dr. Rao Muhammad Adeel Nawab 81
Dr. Rao Muhammad Adeel Nawab 82
Dr. Rao Muhammad Adeel Nawab 83
Dr. Rao Muhammad Adeel Nawab 84
Dr. Rao Muhammad Adeel Nawab 85
Dr. Rao Muhammad Adeel Nawab 86
Dr. Rao Muhammad Adeel Nawab 87
Dr. Rao Muhammad Adeel Nawab 88
Dr. Rao Muhammad Adeel Nawab 89
Dr. Rao Muhammad Adeel Nawab 90
Dr. Rao Muhammad Adeel Nawab 91
Lower Bound –
Upper Bound – No Limit
vs

Dr. Rao Muhammad Adeel Nawab 92