25-08-2023

Machine Learning

Learning from Observations

1
 25-08-2023

What is Learning?
• Herbert Simon: “Learning is any process
by which a system improves performance
from experience.”

• “A computer program is said to learn from

experience E with respect to some class of
tasks T and performance measure P, if its
performance at tasks in T, as measured by
P, improves with experience E.”
– Tom Mitchell
2
2

2
 25-08-2023

Learning
• Learning is essential for unknown environments,
– i.e., when designer lacks omniscience

• Learning is useful as a system construction

method,
– i.e., expose the agent to reality rather than trying to
write it down

• Learning modifies the agent's decision

mechanisms to improve performance

3
 25-08-2023

Machine Learning

• Machine learning: how to acquire a model

on the basis of data / experience
– Learning parameters (e.g. probabilities)
– Learning structure (e.g. BN graphs)
– Learning hidden concepts (e.g. clustering)

4
 25-08-2023

Machine Learning Areas

• Supervised Learning: Data and corresponding labels
are given

• Unsupervised Learning: Only data is given, no labels

provided

• Semi-supervised Learning: Some (if not all) labels are

present

• Reinforcement Learning: An agent interacting with the

world makes observations, takes actions, and is
rewarded or punished; it should learn to choose actions
in such a way as to obtain a lot of reward
5

5
 25-08-2023

Supervised Learning : Important Concepts

• Data: labeled instances <xi, y>, e.g. emails marked
spam/not spam
– Training Set
– Held-out Set
– Test Set
• Features: attribute-value pairs which characterize each x
• Experimentation cycle
– Learn parameters (e.g. model probabilities) on training set
– (Tune hyper-parameters on held-out set)
– Compute accuracy of test set
– Very important: never “peek” at the test set!
• Evaluation
– Accuracy: fraction of instances predicted correctly
• Overfitting and generalization
– Want a classifier which does well on test data
– Overfitting: fitting the training data very closely, but not
generalizing well 6

6
 25-08-2023

Example: Spam Filter

7
Slide from Mackassy

7
 25-08-2023

Example: Digit Recognition

8
Slide from Mackassy

8
 25-08-2023

Classification Examples
• In classification, we predict labels y (classes) for inputs x

• Examples:
– OCR (input: images, classes: characters)
– Medical diagnosis (input: symptoms, classes: diseases)
– Automatic essay grader (input: document, classes: grades)
– Fraud detection (input: account activity, classes: fraud / no fraud)
– Customer service email routing
– Recommended articles in a newspaper, recommended books
– DNA and protein sequence identification
– Categorization and identification of astronomical images
– Financial investments
– … many more

9
 25-08-2023

Inductive learning
• Simplest form: learn a function from examples
•
• f is the target function
• An example is a pair (x, f(x))
•

• Pure induction task:

– Given a collection of examples of f, return a
function h that approximates f.
– find a hypothesis h, such that h ≈ f, given a training
set of examples
–

• (This is a highly simplified model of real learning:

10

– Ignores prior knowledge

10
 25-08-2023

Inductive learning method

• Construct/adjust h to agree with f on training set
• (h is consistent if it agrees with f on all examples)
•
• E.g., curve fitting:
•

11

11
 25-08-2023

Inductive learning method

• Construct/adjust h to agree with f on training set
• (h is consistent if it agrees with f on all examples)
•
• E.g., curve fitting:
•

12

12
 25-08-2023

Inductive learning method

• Construct/adjust h to agree with f on training set
• (h is consistent if it agrees with f on all examples)
•
• E.g., curve fitting:
•

13

13
 25-08-2023

Inductive learning method

• Construct/adjust h to agree with f on training set
• (h is consistent if it agrees with f on all examples)
•
• E.g., curve fitting:
•

14

14
 25-08-2023

Inductive learning method

• Construct/adjust h to agree with f on training set

• (h is consistent if it agrees with f on all examples)
•
• E.g., curve fitting:

15

15
 25-08-2023

Inductive learning method

• Construct/adjust h to agree with f on training set

• (h is consistent if it agrees with f on all examples)
•
• E.g., curve fitting:

• Ockham’s razor: prefer the simplest hypothesis 16

consistent with data

16
 25-08-2023

Generalization
• Hypotheses must generalize to correctly
classify instances not in the training data.
• Simply memorizing training examples is a
consistent hypothesis that does not
generalize.
• Occam’s razor:
– Finding a simple hypothesis helps ensure
generalization.

17
17

17
 25-08-2023

Training Error vs Test Error

18

18
 25-08-2023

Supervised Learning
• Learning a discrete function: Classification
– Boolean classification:
• Each example is classified as true(positive) or
false(negative).
• Learning a continuous function: Regression

19

19
 25-08-2023

Classification—A Two-Step Process

• Model construction: describing a set of predetermined classes
– Each tuple/sample is assumed to belong to a predefined class,
as determined by the class label
– The set of tuples used for model construction is training set
– The model is represented as classification rules, decision
trees, or mathematical formulae
• Model usage: for classifying future or unknown objects
– Estimate accuracy of the model
• The known label of test sample is compared with the
classified result from the model
• Test set is independent of training set, otherwise over-
fitting will occur
– If the accuracy is acceptable, use the model to classify data
tuples whose class labels are not known

20
Data Mining: Concepts and Techniques 20

20
 25-08-2023

Illustrating Classification Task

21

21
 25-08-2023

Issues: Data Preparation

• Data cleaning
– Preprocess data in order to reduce noise and
handle missing values
• Relevance analysis (feature selection)
– Remove the irrelevant or redundant attributes
• Data transformation
– Generalize data to (higher concepts, discretization)
– Normalize attribute values

22
Data Mining: Concepts and Techniques

22
 25-08-2023

Classification Techniques
• Decision Tree based Methods
• Rule-based Methods
• Naïve Bayes and Bayesian Belief
Networks
• Neural Networks
• Support Vector Machines
• and more...

23

23
 25-08-2023

Learning decision trees

Example Problem: decide whether to wait for a table at a
restaurant, based on the following attributes:
1. Alternate: is there an alternative restaurant nearby?
2. Bar: is there a comfortable bar area to wait in?
3. Fri/Sat: is today Friday or Saturday?
4. Hungry: are we hungry?
5. Patrons: number of people in the restaurant (None, Some, Full)
6. Price: price range ($, $$, $$$)
7. Raining: is it raining outside?
8. Reservation: have we made a reservation?
9. Type: kind of restaurant (French, Italian, Thai, Burger)
10. WaitEstimate: estimated waiting time (0-10, 10-30, 30-60, >60)

24

24
 25-08-2023

Feature(Attribute)-based representations
• Examples described by feature(attribute) values
– (Boolean, discrete, continuous)
• E.g., situations where I will/won't wait for a table:

• Classification of examples is positive (T) or negative (F) 25

•

25
 25-08-2023

Decision trees
• One possible representation for hypotheses
• E.g., here is the “true” tree for deciding whether to wait:

26

26
 25-08-2023

Expressiveness
• Decision trees can express any function of the input attributes.
• E.g., for Boolean functions, truth table row → path to leaf:

• Trivially, there is a consistent decision tree for any training set with
one path to leaf for each example (unless f nondeterministic in x) but
it probably won't generalize to new examples

• Prefer to find more compact decision trees

27

27
 25-08-2023

Decision tree learning

• Aim: find a small tree consistent with the training examples
• Idea: (recursively) choose "most significant" attribute as root of
(sub)tree

28

28
 25-08-2023

Decision Tree Construction Algorithm

• Principle
– Basic algorithm (adopted by ID3, C4.5 and CART): a greedy
algorithm
– Tree is constructed in a top-down recursive divide-and-conquer
manner
• Iterations
– At start, all the training tuples are at the root
– Tuples are partitioned recursively based on selected attributes
– Test attributes are selected on the basis of a heuristic or
statistical measure (e.g, information gain)
• Stopping conditions
– All samples for a given node belong to the same class
– There are no remaining attributes for further partitioning –
majority voting is employed for classifying the leaf
– There are no samples left

29

29
 25-08-2023

Decision Tree Induction: Training Dataset

This
follows an
example
of
Quinlan’s
ID3
(Playing
Tennis)

30
August 25, 2023 Data Mining: Concepts and Techniques 30

30
 25-08-2023

Example

31

31
 25-08-2023

Example

32

32
 25-08-2023

Example

33

33
 25-08-2023

Example

34

34
 25-08-2023

Example

35

35
 25-08-2023

Example

36

36
 25-08-2023

Example

37

37
 25-08-2023

Tree Induction
• Greedy strategy.
– Split the records based on an attribute test
that optimizes certain criterion.

• Issues
– Determine how to split the records
• How to specify the attribute test condition?
• How to determine the best split?
– Determine when to stop splitting
38

38
 25-08-2023

Choosing an attribute
• Idea: a good attribute splits the examples into subsets
that are (ideally) "all positive" or "all negative"

• Patrons? is a better choice

39

39
 25-08-2023

How to determine the Best Split

• Greedy approach:
– Nodes with homogeneous class distribution
are preferred
• Need a measure of node impurity:

Non-homogeneous, Homogeneous,
High degree of impurity Low degree of impurity

40

40
 25-08-2023

Measures of Node Impurity

• Information Gain

• Gini Index

• Misclassification error

Choose attributes to split to achieve minimum impurity

41

41
 25-08-2023

Attribute Selection Measure:

Information Gain (ID3/C4.5)
 Select the attribute with the highest information gain
 Let pi be the probability that an arbitrary tuple in D belongs
to class Ci, estimated by |Ci, D|/|D|
 Expected information (entropy) needed to classify a tuple in
D:

 Information needed (after using A to split D into v partitions)

to classify D:

 Information gained by branching on attribute A

42
August 25, 2023 Data Mining: Concepts and Techniques 42

42
 25-08-2023

Information gain
For the training set, p = n = 6, I(6/12, 6/12) = 1 bit

Consider the attributes Patrons and Type (and others too):

Patrons has the highest IG of all attributes and so is chosen by the DTL
algorithm as the root

43

43
 25-08-2023

Example contd.
• Decision tree learned from the 12 examples:

• Substantially simpler than “true” tree---a more complex

hypothesis isn’t justified by small amount of data

44

44
 25-08-2023

Measure of Impurity: GINI

(CART, IBM IntelligentMiner)
• Gini Index for a given node t :

(NOTE: p( j | t) is the relative frequency of class j at node t).

– Maximum (1 - 1/nc) when records are equally distributed among
all classes, implying least interesting information
– Minimum (0.0) when all records belong to one class, implying
most interesting information

45

45
 25-08-2023

Splitting Based on GINI

• Used in CART, SLIQ, SPRINT.
• When a node p is split into k partitions (children),
the quality of split is computed as,

where, ni = number of records at child i,

n = number of records at node p.

46

46
 25-08-2023

Comparison of Attribute Selection Methods

• The three measures return good results but
– Information gain:
• biased towards multivalued attributes
– Gain ratio:
• tends to prefer unbalanced splits in which one
partition is much smaller than the others
– Gini index:
• biased to multivalued attributes
• has difficulty when # of classes is large
• tends to favor tests that result in equal-sized
partitions and purity in both partitions
47
August 25, 2023 Data Mining: Concepts and Techniques 47

47
 25-08-2023

Example Algorithm: C4.5

• Simple depth-first construction.
• Uses Information Gain
• Sorts Continuous Attributes at each node.
• Needs entire data to fit in memory.
• Unsuitable for Large Datasets.

• You can download the software from

Internet
48

48
 25-08-2023

Decision Tree Based Classification

• Advantages:
– Easy to construct/implement
– Extremely fast at classifying unknown records
– Models are easy to interpret for small-sized trees
– Accuracy is comparable to other classification
techniques for many simple data sets
– Tree models make no assumptions about the
distribution of the underlying data : nonparametric
– Have a built-in feature selection method that makes
them immune to the presence of useless variables

49

49
 25-08-2023

Decision Tree Based Classification

• Disadvantages
– Computationally expensive to train
– Some decision trees can be overly complex that
do not generalise the data well.
– Less expressivity: There may be concepts that are
hard to learn with limited decision trees

50

50
 25-08-2023

Overfitting and Tree Pruning

• Overfitting: An induced tree may overfit the training
data
– Too many branches, some may reflect anomalies due to noise or outliers
– Poor accuracy for unseen samples

• Two approaches to avoid overfitting

– Prepruning: Halt tree construction early—do not split a node if this would
result in the goodness measure falling below a threshold
• Difficult to choose an appropriate threshold
– Postpruning: Remove branches from a “fully grown” tree—get a sequence of
progressively pruned trees
• Use a set of data different from the training data to decide which is the
“best pruned tree”

51
August 25, 2023 Data Mining: Concepts and Techniques 51

51
 25-08-2023

Rule-Based Classifier
• Classify records by using a collection of
“if…then…” rules
• Rule: (Condition)  y
– where
• Condition is a conjunctions of attributes
• y is the class label
– LHS: rule antecedent or condition
– RHS: rule consequent
– Examples of classification rules:
• (Blood Type=Warm)  (Lay Eggs=Yes)  Birds
• (Taxable Income < 50K)  (Refund=Yes)  Evade=No52

52
 25-08-2023

Rule-based Classifier (Example)

R1: (Give Birth = no)  (Can Fly = yes)  Birds

R2: (Give Birth = no)  (Live in Water = yes)  Fishes
R3: (Give Birth = yes)  (Blood Type = warm)  Mammals
R4: (Give Birth = no)  (Can Fly = no)  Reptiles
R5: (Live in Water = sometimes)  Amphibians 53

53
 25-08-2023

Rule Extraction from a Decision Tree

 Rules are easier to understand than large trees
age?
 One rule is created for each path from the
<=30 31..40
root to a leaf >40

student? credit rating?

 Each attribute-value pair along a path forms a yes

conjunction: the leaf holds the class prediction no yes excellent fair

no yes yes

• Example: Rule extraction from our buys_computer decision-tree

IF age = young AND student = no THEN buys_computer = no
IF age = young AND student = yes THEN buys_computer = yes
IF age = mid-age THEN buys_computer = yes
IF age = old AND credit_rating = excellent THEN buys_computer = yes
IF age = young AND credit_rating = fair THEN buys_computer = no

54
August 25, 2023 Data Mining: Concepts and Techniques 54

54
 25-08-2023

Extra Slides

55

55
 25-08-2023

Learning agents

56

56
 25-08-2023

Classification(Sınıflandırma)

• IDEA: Build a model based on past data to

predict the class of the new data
• Given a collection of records (training set )
– Each record contains a set of attributes, one of the attributes is
the class.
• Find a model for class attribute as a function of
the values of other attributes.

• Goal: previously unseen records should be

assigned a class as accurately as possible.

57

57
 25-08-2023

Hypothesis spaces
How many distinct decision trees with n Boolean attributes?
= number of Boolean functions
n
= number of distinct truth tables with 2n rows = 22

• E.g., with 6 Boolean attributes, there are

18,446,744,073,709,551,616 trees

How many purely conjunctive hypotheses (e.g., Hungry  Rain)?

• Each attribute can be in (positive), in (negative), or out
 3n distinct conjunctive hypotheses
• More expressive hypothesis space
– increases chance that target function can be expressed
– increases number of hypotheses consistent with training set
 may get worse predictions

58

58
 25-08-2023

Using information theory

• To implement Choose-Attribute in the DTL
algorithm
• Information Content (Entropy):
I(P(v1), … , P(vn)) = Σi=1 -P(vi) log2 P(vi)
• For a training set containing p positive examples
and n negative examples:

59

59
 25-08-2023

Information gain
• A chosen attribute A divides the training set E into
subsets E1, … , Ev according to their values for A, where
A has v distinct values.

• Information Gain (IG) or reduction in entropy from the

attribute test:

• Choose the attribute with the largest IG

60

60
 25-08-2023

Performance measurement
• How do we know that h ≈ f ?
1. Use theorems of computational/statistical learning theory
2. Try h on a new test set of examples
(use same distribution over example space as training set)
Learning curve = % correct on test set as a function of training set size

61

61