Course 395: Machine Learning

• Lecturers: Maja Pantic ([email protected])

Stavros Petridis ([email protected])

• Goal (Lectures): To present basic theoretical concepts and key algorithms that
form the core of machine learning

• Goal (CBC): To enable hands-on experience with implementing machine

learning algorithms (developed using Matlab or Python)

• Material: Machine Learning by Tom Mitchell (1997)

Neural Networks & Deep Learning by Michael Nielsen (2017)
Manual for completing the CBC

• More Info: https://www.ibug.doc.ic.ac.uk/courses

Maja Pantic Machine Learning (course 395)

Course 395: Machine Learning – Lectures

• Lecture 1-2: Concept Learning (M. Pantic)

• Lecture 3-4: Decision Trees & CBC Intro (M. Pantic & S. Petridis)

• Lecture 5-6: Evaluating Hypotheses (S. Petridis)

• Lecture 7-8: Artificial Neural Networks I (S. Petridis)

• Lecture 9-10: Artificial Neural Networks II (S. Petridis)

• Lecture 11-12: Artificial Neural Networks III (S. Petridis)

• Lecture 13-14: : Instance Based Learning & Genetic Algorithms (M. Pantic)

Maja Pantic Machine Learning (course 395)

Course 395: Machine Learning - CBC

• Lecture 1-2: Concept Learning

• Lecture 3-4: Decision Trees & CBC Intro

• Lecture 5-6: Evaluating Hypotheses

• Lecture 7-8: Artificial Neural Networks I

• Lecture 9-10: Artificial Neural Networks II

• Lecture 11-12: Artificial Neural Networks III

• Lecture 13-14: Instance Based Learning & Genetic Algorithms

Maja Pantic Machine Learning (course 395)

Course 395: Machine Learning

NOTE
CBC accounts for 33.3% of the final grade for the Machine Learning Exam.

2 1
final _ grade = exam _ grade + exam _ grade
3 3

Maja Pantic Machine Learning (course 395)

Course 395: Machine Learning – Lectures

!
• Lecture 1-2: Concept Learning

• Lecture 3-4: Decision Trees & CBC Intro

• Lecture 5-6: Evaluating Hypotheses

• Lecture 7-8: Artificial Neural Networks I

• Lecture 9-10: Artificial Neural Networks II

• Lecture 11-12: Artificial Neural Networks III

• Lecture 13-14: Instance Based Learning & Genetic Algorithms

Maja Pantic Machine Learning (course 395)

Concept Learning – Lecture Overview

• Why machine learning?

• Well-posed learning problems

• Designing a machine learning system

• Concept learning task

• Concept learning as Search

• Find-S algorithm

• Candidate-Elimination algorithm

Maja Pantic Machine Learning (course 395)

Machine Learning

• Learning ↔ Intelligence
(Def: Intelligence is the ability to learn and use concepts to solve problems.)

• Machine Learning ↔ Artificial Intelligence

– Def: AI is the science of making machines do things that require
intelligence if done by men (Minsky 1986)
– Def: Machine Learning is an area of AI concerned with development of
techniques which allow machines to learn

• Why Machine Learning? ↔ Why Artificial Intelligence?

Maja Pantic Machine Learning (course 395)

Machine Learning

Maja Pantic Machine Learning (course 395)

Machine Learning

Maja Pantic Machine Learning (course 395)

Machine Learning

Maja Pantic Machine Learning (course 395)

Machine Learning

• Learning ↔ Intelligence
(Def: Intelligence is the ability to learn and use concepts to solve problems.)

• Machine Learning ↔ Artificial Intelligence

– Def: AI is the science of making machines do things that require
intelligence if done by men (Minsky 1986)
– Def: Machine Learning is an area of AI concerned with development of
techniques which allow machines to learn

• Why Machine Learning? ↔ Why Artificial Intelligence?

≡ To build machines exhibiting intelligent behaviour (i.e., able to reason,
predict, and adapt) while helping humans work, study, and entertain
themselves

Maja Pantic Machine Learning (course 395)

Machine Learning

• Machine Learning ↔ Artificial Intelligence

• Machine Learning ← Biology (e.g., Neural Networks, Genetic Algorithms)

• Machine Learning ← Cognitive Sciences (e.g., Case-based Reasoning)

• Machine Learning ← Statistics (e.g., Support Vector Machines)

• Machine Learning ← Probability Theory (e.g., Bayesian Networks)

• Machine Learning ← Logic (e.g., Inductive Logic Programming)

• Machine Learning ← Information Theory (e.g., used by Decision Trees)

Maja Pantic Machine Learning (course 395)

Machine Learning

• Human Learning ↔ Machine Learning

– human-logic inspired problem solvers (e.g., rule-based reasoning)
– biologically inspired problem solvers (e.g., Neural Networks)
• supervised learning - generates a function that maps inputs to desired outputs
• unsupervised learning - models a set of inputs, labelled examples are not available
– learning by education (e.g., reinforcement learning, case-based reasoning)

• General Problem Solvers vs. Purposeful Problem Solvers

– emulating general-purpose human-like problem solving is impractical
– restricting the problem domain results in ‘rational’ problem solving
– example of General Problem Solver: Turing Test
– examples of Purposeful Problem Solvers: speech recognisers, face recognisers,
facial expression recognisers, data mining, games, etc.

• Application domains: security, medicine, education, finances, genetics, etc.

Maja Pantic Machine Learning (course 395)

Well-posed Learning Problems

• Def 1 (Mitchell 1997):

A computer program is said to learn from experience E with respect to some
class of tasks T and performance measure P, if its performance at tasks in T,
as measured by P, improves by experience E.

• Def 2 (Hadamard 1902):

A (machine learning) problem is well-posed if a solution to it exists, if that
solution is unique, and if that solution depends on the data / experience but it
is not sensitive to (reasonably small) changes in the data / experience.

Maja Pantic Machine Learning (course 395)

 Designing a Machine Learning System

• Target Function V represents the problem to be solved

Well-posed (e.g., choosing the best next move in chess, identifying people,
Problem?
classifying facial expressions into emotion categories)

• V: D → C where D is the input state space and C is the set of classes

Determine type of
V: D → [-1, 1] is a general target function of a binary classifier
training examples

Determine
• Ideal Target Function is usually not known; machine learning
algorithms learn an approximation of V, say V’
Target Function
• Representation of function V’ to be learned should
Choose Target F-on
– be as close an approximation of V as possible
Representation
– require (reasonably) small amount of training data to be learned
Choose Learning
Algorithm • V’(d) = w0 + w1x1 +…+ wnxn where ‹x1…xn› ≡ d ∈ D is an input state.
This reduces the problem to learning (the most optimal) weights w.

Maja Pantic Machine Learning (course 395)

 Designing a Machine Learning System
• V: D → C where D is the input state and C is the set of classes
V: D → [-1, 1] is a general target function of a binary classifier
Well-posed
• V’(d) = w0 + w1x1 +…+ wnxn where ‹x1…xn› ≡ d ∈ D is an input state.
Problem?
This reduces the problem to learning (the most optimal) weights w.
Determine type of • Training examples suitable for the given target function representation
training examples V’ are pairs ‹d, c› where c ∈ C is the desired output (classification) of
the input state d ∈ D.
Determine
Target Function • Learning algorithm learns the most optimal set of weights w (so-called
best hypothesis), i.e., the set of weights that best fit the training
examples ‹d, c›.
Choose Target F-on
Representation
• Learning algorithm is selected based on the availability of training
examples (supervised vs. unsupervised), knowledge of the final set of
Choose Learning classes C (offline vs. online, i.e., eager vs. lazy), availability of a tutor
Algorithm (reinforcement learning).

• The learned V’ is then used to solve new instances of the problem.

Maja Pantic Machine Learning (course 395)

Concept Learning

• Concept learning
– supervised, eager learning
– target problem: whether something belongs to the target concept or not
– target function: V: D → {true, false}

• Underlying idea: Humans acquire general concepts from specific examples

(e.g., concepts: beauty, good friend, well-fitting-shoes)
(note: each concept can be thought of as Boolean-valued function)

• Concept learning is inferring a Boolean-valued function from training data

→ concept learning is the prototype binary classification

Maja Pantic Machine Learning (course 395)

 Concept Learning Task – Notation

• Concept learning task:

– target concept: Girls who Simon likes
– target function: c: D → {0, 1}
– data d ∈ D: Girls, each described in terms of the following attributes
• a1 ≡ Hair (possible values: blond, brown, black)
• a2 ≡ Body (possible values: thin, average, plump)
instances • a3 ≡ likesSimon (possible values: yes, no)
• a4 ≡ Pose (possible values: arrogant, natural, goofy)
• a5 ≡ Smile (possible values: none, pleasant, toothy)
• a6 ≡ Smart (possible values: yes, no) error rate
– target f-on representation: h ≡ c’: ‹a1, a2, a3, a4, a5, a6› → {0, 1}
– training examples D: positive and negative examples of target function c

• Aim: Find a hypothesis h∈ H such that (∀d ∈ D) h(d) – c(d) < ε ≈ 0, where H is the
set of all possible hypotheses h ≡ ‹a1, a2, a3, a4, a5, a6›, where each ak, k = [1..6], may
be ‘?’ (≡ any value is acceptable), ‘0’ (≡ no value is acceptable), or a specific value.
h ≡ ‹?, ?, ?, ?, ?, ?› h ≡ ‹0, 0, 0, 0, 0, 0› h ≡ ‹?, ?, yes, ?, ?, ?›
Maja Pantic Machine Learning (course 395)
 Concept Learning as Search

• Concept learning task:

– target concept: Girls who Simon likes
– target function: c: D → {0, 1}
– data d ∈ D: Girls, each described in terms of the following attributes
• a1 ≡ Hair (possible values: blond, brown, black)
• a2 ≡ Body (possible values: thin, average, plump)
+‘?’
instances • a3 ≡ likesSimon (possible values: yes, no) +‘?’ |H| = 1 + 4· 4· 3· 4· 4· 3 = 2305
• a4 ≡ Pose (possible values: arrogant, natural, goofy)
• a5 ≡ Smile (possible values: none, pleasant, toothy) +‘?’ h ≡‹0,0,0,0,0,0›
• a6 ≡ Smart (possible values: yes, no) +‘?’ error rate
– target f-on representation: h ≡ c’: ‹a1, a2, a3, a4, a5, a6› → {0, 1}
– training examples D: positive and negative examples of target function c

• Aim: Find a hypothesis h∈ H such that (∀d ∈ D) h(d) – c(d) < ε ≈ 0, where H is the
set of all possible hypotheses h ≡ ‹a1, a2, a3, a4, a5, a6›, where each ak, k = [1..6], may
be ‘?’ (≡ any value is acceptable), ‘0’ (≡ no value is acceptable), or a specific value.
concept learning ≡ searching through H

Maja Pantic Machine Learning (course 395)

General-to-Specific Ordering

• Many concept learning algorithms utilize general-to-specific ordering of hypotheses

• General-to-Specific Ordering:
– h1 precedes (is more general than) h2 ⇔ (∀d ∈ D) (h1(d) = 1) ← (h2(d) = 1)
(e.g., h1 ≡ ‹?, ?, yes,?, ?, ?› and h2 ≡ ‹?, ?, yes,?, ?, yes› ⇒ h1 >g h2 )
– h1 and h2 are of equal generality ⇔ (∃d ∈ D) { [(h1(d) = 1) → (h2(d) = 1)] ∧
[(h2(d) = 1) → (h1(d) = 1)] ∧ h1 and h2 have equal number of ‘?’ }
(e.g., h1 ≡ ‹?, ?, yes,?, ?, ?› and h2 ≡ ‹?, ?, ?, ?, ?, yes› ⇒ h1 =g h2 )
– h2 succeeds (is more specific than) h1 ⇔ (∀d ∈ D) (h1(d) = 1) ← (h2(d) = 1)
(e.g., h1 ≡ ‹?, ?, yes,?, ?, ?› and h2 ≡ ‹?, ?, yes,?, ?, yes› ⇒ h2 ≥g h1 )

Maja Pantic Machine Learning (course 395)

 Find-S Algorithm – Example

1. Initialise h∈ H to the most specific hypothesis: h ← ‹a1,…,an›, (∀i) ai = 0.

2. FOR each positive training instance d ∈ D, do:
FOR each attribute ai, i = [1..n], in h, do:
IF ai is satisfied by d
THEN do nothing
ELSE replace ai in h so that the resulting h’ >g h, h ← h’.
3. Output hypothesis h.
c(d) hair body likesSimon pose smile smart
1 1 blond thin yes arrogant toothy no
2 0 brown thin no natural pleasant yes
3 1 blond plump yes goofy pleasant no
4 0 black thin no arrogant none no
5 0 blond plump no natural toothy yes

h ← ‹0,0,0,0,0,0› → h ≡ d1 → h ← ‹blond, ?, yes, ?, ?, no›

Maja Pantic Machine Learning (course 395)

Find-S Algorithm

• Find-S is guaranteed to output the most specific hypothesis h that best fits positive
training examples.
• The hypothesis h returned by Find-S will also fit negative examples as long as
training examples are correct.

• However,
– Find-S is sensitive to noise that is (almost always) present in training examples.
– there is no guarantee that h returned by Find-S is the only h that fits the data.
– several maximally specific hypotheses may exist that fits the data but, Find-S
will output only one.
– Why we should prefer most specific hypotheses over, e.g., most general
hypotheses?

Maja Pantic Machine Learning (course 395)

Find-S Algorithm – Example

1. Initialise h∈ H to the most specific hypothesis: h ← ‹a1,…,an›, (∀i) ai = 0.

2. FOR each positive training instance d ∈ D, do:
FOR each attribute ai, i = [1..n], in h, do:
IF ai is satisfied by d
THEN do nothing
ELSE replace ai in h so that the resulting h’ >g h, h ← h’.
3. Output hypothesis h.
c(d) hair body likesSimon pose smile smart
1 1 blond thin yes arrogant toothy no
2 0 brown thin no natural pleasant yes
3 1 blond plump yes goofy pleasant no
4 0 black thin no arrogant none no
5 0 blond plump no natural toothy yes

Find-S → h = ‹blond, ?, yes, ?, ?, no› BUT h2 = ‹blond,?, ?, ?, ?, no> fits D as well

Maja Pantic Machine Learning (course 395)

Find-S Algorithm – Example

1. Initialise h∈ H to the most specific hypothesis: h ← ‹a1,…,an›, (∀i) ai = 0.

2. FOR each positive training instance d ∈ D, do:
FOR each attribute ai, i = [1..n], in h, do:
IF ai is satisfied by d
THEN do nothing
ELSE replace ai in h so that the resulting h’ >g h, h ← h’.
3. Output hypothesis h.
c(d) hair body likesSimon pose smile smart
1 1 blond thin yes arrogant toothy no
2 0 brown thin no natural pleasant yes
3 1 blond plump yes goofy pleasant no
4 0 black thin no arrogant none no
5 0 blond plump no natural toothy yes

Find-S → h1 = ‹blond, ?, ?, ?, ?, no› YET h2 = ‹blond,?, yes, ?, ?, ?> fits D as well

Maja Pantic Machine Learning (course 395)

Candidate-Elimination Algorithm

• Find-S is guaranteed to output the most specific hypothesis h that best fits positive
training examples.
• The hypothesis h returned by Find-S will also fit negative examples as long as
training examples are correct.

• However,
1. Find-S is sensitive to noise that is (almost always) present in training examples.
2. there is no guarantee that h returned by Find-S is the only h that fits the data.
3. several maximally specific hypotheses may exist that fits the data but, Find-S
will output only one.
4. Why we should prefer most specific hypotheses over, e.g., most general
hypotheses?

To address the last three drawbacks of Find-S, Candidate-Elimination was proposed

Maja Pantic Machine Learning (course 395)

Candidate-Elimination (C-E) Algorithm

• Main idea: Output a set of hypothesis VS ⊆ H that fit (are consistent) with data D

• Candidate-Elimination (C-E) Algorithm is based upon:

– general-to-specific ordering of hypotheses
– Def: h is consistent (fits) data D ⇔ (∀‹d, c(d)›) h(d) = c(d)
– Def: version space VS ⊆ H is set of all h ∈ H that are consistent with D

• C-E algorithm defines VS in terms of two boundaries:

– general boundary G ⊆ VS is a set of all h ∈ VS that are the most general
– specific boundary S ⊆ VS is a set of all h ∈ VS that are the most specific

Maja Pantic Machine Learning (course 395)

Candidate-Elimination (C-E) Algorithm

1. Initialise G⊆ VS to the most general hypothesis: h ← ‹a1,…,an›, (∀i) ai = ?.

Initialise S⊆ VS to the most specific hypothesis: h ← ‹a1,…,an›, (∀i) ai = 0.
2. FOR each training instance d ∈ D, do:
IF d is a positive example
Remove from G all h that are not consistent with d.
FOR each hypothesis s ∈ S that is not consistent with d, do:
- replace s with all h that are consistent with d, h >g s, h ≥g g ∈ G,
- remove from S all s being more general than other s in S.
IF d is a negative example
Remove from S all h that are not consistent with d.
FOR each hypothesis g ∈ G that is not consistent with d, do:
- replace g with all h that are consistent with d, g >g h, h >g s ∈ S,
- remove from G all g being less general than other g in G.

3. Output hypothesis G and S.

Maja Pantic Machine Learning (course 395)

C-E Algorithm – Example
c(d) hair body likesSimon pose smile smart
1 1 blond thin yes arrogant toothy no
2 0 brown thin no natural pleasant yes
3 1 blond plump yes goofy pleasant no
4 0 black thin no arrogant none no
5 0 blond plump no natural toothy yes

G0 ← {‹?, ?, ?, ?, ?, ?›} , S0 ← {‹0, 0, 0, 0, 0, 0›}

Maja Pantic Machine Learning (course 395)

C-E Algorithm – Example
c(d) hair body likesSimon pose smile smart
1 1 blond thin yes arrogant toothy no
2 0 brown thin no natural pleasant yes
3 1 blond plump yes goofy pleasant no
4 0 black thin no arrogant none no
5 0 blond plump no natural toothy yes

d1 is positive → refine S

no g ∈ G0 is inconsistent with d1 → G1 ← G0 ≡ {‹?, ?, ?, ?, ?, ?›}

add to S all minimal generalizations of s∈ S0 such that s∈ S1 is consistent with d1

S1 ← {‹blond, thin, yes, arrogant, toothy, no›}

Maja Pantic Machine Learning (course 395)

C-E Algorithm – Example
c(d) hair body likesSimon pose smile smart
1 1 blond thin yes arrogant toothy no
2 0 brown thin no natural pleasant yes
3 1 blond plump yes goofy pleasant no
4 0 black thin no arrogant none no
5 0 blond plump no natural toothy yes

d2 is negative → refine G

no s ∈ S1 is inconsistent with d2 → S2 ← S1 ≡ {‹blond, thin, yes, arrogant, toothy, no›}

add to G all minimal specializations of g∈ G1 such that g∈ G2 is consistent with d2

G1 ≡ {‹?, ?, ?, ?, ?, ?›}
G2 ← {‹blond, ?, ?, ?, ?, ?› , ‹?, ?, yes, ?, ?, ?› , ‹?, ?, ?, arrogant, ?, ?› ,
‹?, ?, ?, ?, toothy, ?›, ‹?, ?, ?, ?, ?, no› }

Maja Pantic Machine Learning (course 395)

C-E Algorithm – Example
c(d) hair body likesSimon pose smile smart
1 1 blond thin yes arrogant toothy no
2 0 brown thin no natural pleasant yes
3 1 blond plump yes goofy pleasant no
4 0 black thin no arrogant none no
5 0 blond plump no natural toothy yes

d3 is positive → refine S
two g∈ G2 are inconsistent with d3, i.e., ‹?, ?, ?, arrogant, ?, ?› and ‹?, ?, ?, ?, toothy, ?› →
G3 ← {‹blond, ?, ?, ?, ?, ?› , ‹?, ?, yes, ?, ?, ?› , ‹?, ?, ?, ?, ?, no› }

add to S all minimal generalizations of s∈ S2 such that s∈ S3 is consistent with d3

S2 ≡ {‹blond, thin, yes, arrogant, toothy, no›}
S3 ← {‹blond, ?, yes, ?, ?, no›}

Maja Pantic Machine Learning (course 395)

C-E Algorithm – Example
c(d) hair body likesSimon pose smile smart
1 1 blond thin yes arrogant toothy no
2 0 brown thin no natural pleasant yes
3 1 blond plump yes goofy pleasant no
4 0 black thin no arrogant none no
5 0 blond plump no natural toothy yes

d4 is negative → refine G

no s ∈ S3 is inconsistent with d4 → S4 ← S3 ≡ {‹blond, ?, yes, ?, ?, no›}

add to G all minimal specializations of g∈ G3 such that g∈ G4 is consistent with d4

G3 ≡ {‹blond, ?, ?, ?, ?, ?› , ‹?, ?, yes, ?, ?, ?› , ‹?, ?, ?, ?, ?, no› }
G4 ← {‹blond, ?, ?, ?, ?, ?› , ‹?, ?, yes, ?, ?, ?› }

Maja Pantic Machine Learning (course 395)

C-E Algorithm – Example
c(d) hair body likesSimon pose smile smart
1 1 blond thin yes arrogant toothy no
2 0 brown thin no natural pleasant yes
3 1 blond plump yes goofy pleasant no
4 0 black thin no arrogant none no
5 0 blond plump no natural toothy yes

d5 is negative → refine G

no s ∈ S4 is inconsistent with d4 → S5 ← S4 ≡ {‹blond, ?, yes, ?, ?, no›}

add to G all minimal specializations of g∈ G4 such that g∈ G5 is consistent with d5

One g∈ G4 is inconsistent with d5, i.e., ‹blond, ?, ?, ?, ?, ?› →
G4 ≡ {‹blond, ?, ?, ?, ?, ?› , ‹?, ?, yes, ?, ?, ?›}
G5 ← {‹?, ?, yes, ?, ?, ?› }

Maja Pantic Machine Learning (course 395)

C-E Algorithm – Example
c(d) hair body likesSimon pose smile smart
1 1 blond thin yes arrogant toothy no
2 0 brown thin no natural pleasant yes
3 1 blond plump yes goofy pleasant no
4 0 black thin no arrogant none no
5 0 blond plump no natural toothy yes

Output of C-E:
version space of hypotheses VS ⊆ H bound with
specific boundary S ≡ {‹blond, ?, yes, ?, ?, no›} and
general boundary G ≡ {‹?, ?, yes, ?, ?, ?› }

Output of Find-S:
most specific hypothesis h ≡ ‹blond, ?, yes, ?, ?, no›

Maja Pantic Machine Learning (course 395)

C-E Algorithm – Example
c(d) hair body likesSimon pose smile smart
1 1 blond thin yes arrogant toothy no
2 0 brown thin no natural pleasant yes
3 1 blond plump yes goofy pleasant no
4 0 black thin no arrogant none no
5 0 blond plump no natural toothy yes

Output of C-E:
version space of hypotheses VS ⊆ H bound with
specific boundary S ≡ {‹blond, ?, yes, ?, ?, no›} and
general boundary G ≡ {‹?, ?, yes, ?, ?, ?› }

VS ≡ {‹?, ?, yes, ?, ?, ?› , ‹blond, ?, yes, ?, ?, ?› ,

‹?, ?, yes, ?, ?, no› , ‹blond, ?, yes, ?, ?, no›}

Maja Pantic Machine Learning (course 395)

Concept Learning – Lecture Overview

• Why machine learning?

• Well-posed learning problems

• Designing a machine learning system

• Concept learning task

• Concept learning as Search

• Find-S algorithm

• Candidate-Elimination algorithm

Maja Pantic Machine Learning (course 395)

Concept Learning – Practice

• Tom Mitchell’s book – chapter 1 and chapter 2

• Relevant exercises from chapter 1: 1.1, 1.2, 1.3, 1.5

• Relevant exercises from chapter 2: 2.1, 2.2, 2.3, 2.4, 2.5

Maja Pantic Machine Learning (course 395)

Course 395: Machine Learning – Lectures

• Lecture 1-2: Concept Learning

!
• Lecture 3-4: Decision Trees & CBC Intro

• Lecture 5-6: Evaluating Hypotheses

• Lecture 7-8: Artificial Neural Networks I

• Lecture 9-10: Artificial Neural Networks II

• Lecture 11-12: Artificial Neural Networks III

• Lecture 13-14: Instance Based Learning & Genetic Algorithms

Maja Pantic Machine Learning (course 395)