UNIT-4 Classification and Prediction

Topic
**********************************************
• What is classification?
• What is prediction?
• Issues regarding classification and prediction
• Classification by decision tree induction
• Bayesian Classification
• Classification by backpropagation
• Classification based on concepts from association rule
mining
• Other Classification Methods
• Prediction
• Classification accuracy
1
 What is Classification & Prediction
• Classification:
– predicts categorical class labels
– classifies data (constructs a model) based on the training
set and the values (class labels) in a classifying attribute
and uses it in classifying new data
• Prediction:
– models continuous-valued functions
– predicts unknown or missing values
• Applications
– credit approval
– target marketing
– medical diagnosis
– treatment effectiveness analysis
2
Lecture-32 - What is classification? What is prediction?
 Classification—A Two-Step Process
• Learning step- describing a set of predetermined classes
– Each tuple/sample is assumed to belong to a predefined class, as
determined by the class label attribute
– The set of tuples used for model construction: training set
– The model is represented as classification rules, decision trees, or
mathematical formulae
• Classification- for classifying future or unknown objects
– Estimate accuracy of the model
• The known label of test sample is compared with the classified
result from the model
• Accuracy rate is the percentage of test set samples that are
correctly classified by the model
• Test set is independent of training set, otherwise over-fitting will
occur
3
Lecture-32 - What is classification? What is prediction?
 Classification Process : Model Construction

Classification
Algorithms
Training
Data

NAME RANK YEARS TENURED Classifier

Mike Assistant Prof 3 no (Model)
Mary Assistant Prof 7 yes
Bill Professor 2 yes
Jim Associate Prof 7 yes IF rank = ‘professor’
Dave Assistant Prof 6 no
OR years > 6
Anne Associate Prof 3 no
THEN tenured = ‘yes’
4
Lecture-32 - What is classification? What is prediction?
Classification Process : Use the Model in Prediction

Classifier

Testing
Data Unseen Data

(Jeff, Professor, 4)
NAME RANK YEARS TENURED
Tom Assistant Prof 2 no Tenured?
Merlisa Associate Prof 7 no
George Professor 5 yes
Joseph Assistant Prof 7 yes 5
Lecture-32 - What is classification? What is prediction?
 Supervised vs. Unsupervised Learning
• Supervised learning (classification)
– Supervision: The training data (observations,
measurements, etc.) are accompanied by labels indicating
the class of the observations
– New data is classified based on the training set
• Unsupervised learning (clustering)
– The class labels of training data is unknown
– Given a set of measurements, observations, etc. with the
aim of establishing the existence of classes or clusters in
the data

6
Lecture-32 - What is classification? What is prediction?
 Issues regarding classification and prediction -
Preparing the data for classification and prediction

• Data cleaning
– Preprocess data in order to reduce noise and handle
missing values
• Relevance analysis (feature selection)
– Remove the irrelevant or redundant attributes
• Data transformation
– Generalize and/or normalize data

7
Lecture-33 - Issues regarding classification and prediction
 Issues regarding classification and prediction
Comparing Classification Methods
• Accuracy
• Speed and scalability
– time to construct the model
– time to use the model
• Robustness
– handling noise and missing values
• Scalability
– efficiency in disk-resident databases
• Interpretability:
– understanding and insight provded by the model
• interpretability
– decision tree size
– compactness of classification rules

8
Lecture-33 - Issues regarding classification and prediction
 Classification by Decision Tree Induction
• Decision tree
– A flow-chart-like tree structure
– Internal node denotes a test on an attribute
– Branch represents an outcome of the test
– Leaf nodes represent class labels or class distribution
• Decision tree generation consists of two phases
– Tree construction
• At start, all the training examples are at the root
• Partition examples recursively based on selected attributes
– Tree pruning
• Identify and remove branches that reflect noise or outliers
• Use of decision tree: Classifying an unknown sample
– Test the attribute values of the sample against the decision tree

9
Lecture-34 - Classification by decision tree induction
 Training Dataset
age income student credit_rating
This <=30
<=30
high
high
no fair
no excellent
follows 31…40 high no fair
an >40 medium no fair
>40 low yes fair
example >40 low yes excellent
from 31…40 low yes excellent
<=30 medium no fair
Quinlan’s <=30 low yes fair
ID3 >40 medium yes fair
<=30 medium yes excellent
31…40 medium no excellent
31…40 high yes fair
>40 medium no excellent

10
Lecture-34 - Classification by decision tree induction
Output: A Decision Tree for “buys_computer”

age?

<=30 overcast
30..40 >40

student? yes credit rating?

no yes excellent fair

no yes no yes

11
Lecture-34 - Classification by decision tree induction
 Algorithm for Decision Tree Induction
• Basic algorithm (a greedy algorithm)
– Tree is constructed in a top-down recursive divide-and-conquer manner
– At start, all the training examples are at the root
– Attributes are categorical (if continuous-valued, they are discretized in
advance)
– Examples are partitioned recursively based on selected attributes
– Test attributes are selected on the basis of a heuristic or statistical
measure (e.g., information gain)
• Conditions for stopping partitioning
– All samples for a given node belong to the same class
– There are no remaining attributes for further partitioning – majority
voting is employed for classifying the leaf
– There are no samples left

12
Lecture-34 - Classification by decision tree induction
 Attribute Selection Measure
• Information gain (ID3/C4.5)
– All attributes are assumed to be categorical
– Can be modified for continuous-valued attributes
• Gini index (IBM IntelligentMiner)
– All attributes are assumed continuous-valued
– Assume there exist several possible split values for each
attribute
– May need other tools, such as clustering, to get the possible
split values
– Can be modified for categorical attributes

13
Lecture-34 - Classification by decision tree induction
 Information Gain (ID3/C4.5)

• Select the attribute with the highest information gain

• Assume there are two classes, P and N
– Let the set of examples S contain p elements of class P and n
elements of class N
– The amount of information, needed to decide if an arbitrary
example in S belongs to P or N is defined as

p p n n
I ( p, n)   log 2  log 2
pn pn pn pn

14
Lecture-34 - Classification by decision tree induction
Information Gain in Decision Tree Induction

• Assume that using attribute A a set S will be

partitioned into sets {S1, S2 , …, Sv}
– If Si contains pi examples of P and ni examples of N, the
entropy, or the expected information needed to classify
objects in all subtrees Si is
 p n
E ( A)   i i I ( pi , ni )
i 1 p  n

• The encoding information that would be gained by

branching on A Gain( A)  I ( p, n)  E ( A)
15
Lecture-34 - Classification by decision tree induction
 Attribute Selection by Information Gain
Computation
 Class P: buys_computer = E ( age) 
5
I ( 2,3) 
4
I ( 4,0)
14 14
“yes”
5
 I (3,2)  0.69
 Class N: buys_computer = 14
“no”
 I(p, n) = I(9, 5) =0.940 Hence
Gain(age)  I ( p, n)  E (age)
 Compute the entropy for
age:
Similarly
age pi ni I(pi, ni) Gain(income)  0.029
<=30 2 3 0.971 Gain( student )  0.151
30…40 4 0 0 Gain(credit _ rating )  0.048
>40 3 2 0.971 16
Lecture-34 - Classification by decision tree induction
 Gini Index (IBM IntelligentMiner)
• If a data set T contains examples from n classes, gini
index, gini(T) is defined as
n
gini(T )  1  p 2j
j 1
where pj is the relative frequency of class j in T.
• If a data set T is split into two subsets T1 and T2 with
sizes N1 and N2 respectively, the gini index of the split
data contains examples from n classes, the gini index
gini(T) is defined as
(T )  N 1 gini( )  N 2 gini( )
gini split T1 T2
N N
• The attribute provides the smallest ginisplit(T) is chosen to
split the node (need to enumerate all possible splitting
points for each attribute).
17
Lecture-34 - Classification by decision tree induction
 Extracting Classification Rules from Trees
• Represent the knowledge in the form of IF-THEN rules
• One rule is created for each path from the root to a leaf
• Each attribute-value pair along a path forms a conjunction
• The leaf node holds the class prediction
• Rules are easier for humans to understand
• Example
IF age = “<=30” AND student = “no” THEN buys_computer = “no”
IF age = “<=30” AND student = “yes” THEN buys_computer = “yes”
IF age = “31…40” THEN buys_computer = “yes”
IF age = “>40” AND credit_rating = “excellent” THEN buys_computer =
“yes”
IF age = “>40” AND credit_rating = “fair” THEN buys_computer = “no”

18
Lecture-34 - Classification by decision tree induction
 Avoid Overfitting in Classification
• The generated tree may overfit the training data
– Too many branches, some may reflect anomalies due to
noise or outliers
– Result is in poor accuracy for unseen samples
• Two approaches to avoid overfitting
– Prepruning: Halt tree construction early—do not split a
node if this would result in the goodness measure falling
below a threshold
• Difficult to choose an appropriate threshold
– Postpruning: Remove branches from a “fully grown” tree
—get a sequence of progressively pruned trees
• Use a set of data different from the training data to
decide which is the “best pruned tree”
19
Lecture-34 - Classification by decision tree induction
 Approaches to Determine the Final Tree Size
• Separate training and testing sets
• Use cross validation, 10-fold cross validation
• Use all the data for training
– apply a statistical test (chi-square) to estimate whether
expanding or pruning a node may improve the entire
distribution
• Use minimum description length (MDL) principle:
– halting growth of the tree when the encoding is minimized

20
Lecture-34 - Classification by decision tree induction
 Enhancements to basic decision tree
induction
• Allow for continuous-valued attributes
– Dynamically define new discrete-valued attributes that
partition the continuous attribute value into a discrete set
of intervals
• Handle missing attribute values
– Assign the most common value of the attribute
– Assign probability to each of the possible values
• Attribute construction
– Create new attributes based on existing ones that are
sparsely represented
– This reduces fragmentation, repetition, and replication
21
Lecture-34 - Classification by decision tree induction
 Classification in Large Databases
• Classification—a classical problem extensively studied by
statisticians and machine learning researchers
• Scalability: Classifying data sets with millions of examples and
hundreds of attributes with reasonable speed
• Why decision tree induction in data mining?
– relatively faster learning speed (than other classification
methods)
– convertible to simple and easy to understand classification
rules
– can use SQL queries for accessing databases
– comparable classification accuracy with other methods

22
Lecture-34 - Classification by decision tree induction
Scalable Decision Tree Induction Methods in Data
Mining Studies
• SLIQ (EDBT’96 — Mehta et al.)
– builds an index for each attribute and only class list and the
current attribute list reside in memory
• SPRINT (VLDB’96 — J. Shafer et al.)
– constructs an attribute list data structure
• PUBLIC (VLDB’98 — Rastogi & Shim)
– integrates tree splitting and tree pruning: stop growing the
tree earlier
• RainForest (VLDB’98 — Gehrke, Ramakrishnan & Ganti)
– separates the scalability aspects from the criteria that
determine the quality of the tree
– builds an AVC-list (attribute, value, class label) 23
Lecture-34 - Classification by decision tree induction
 Lecture-35

Bayesian Classification

24
 Bayesian Classification

• Statical classifiers
• Based on Baye’s theorem
• Naïve Bayesian classification
• Class conditional independence
• Bayesian belief netwoks

25
Lecture-35 - Bayesian Classification
 Bayesian Classification

• Probabilistic learning
– Calculate explicit probabilities for hypothesis, among the most practical
approaches to certain types of learning problems
• Incremental
– Each training example can incrementally increase/decrease the probability
that a hypothesis is correct. Prior knowledge can be combined with
observed data.
• Probabilistic prediction
– Predict multiple hypotheses, weighted by their probabilities
• Standard
– Even when Bayesian methods are computationally intractable, they can
provide a standard of optimal decision making against which other
methods can be measured

26
Lecture-35 - Bayesian Classification
 Baye’s Theorem

• Let X be a data tuple and H be hypothesis, such that

X belongs to a specific class C.
• Posterior probability of a hypothesis h on X, P(h|X)
follows the Baye’s theorem

P( X | H ) P( H )
P( H | X ) 
P( X )

27
Lecture-35 - Bayesian Classification
 Naïve Bayes Classifier

• Let D be a training data set of tuples and associated

class labels
• X = (x1, x2, x3,…xn) and m = C1, C2, C3,..Cm
• Bayes theorem:
P(Ci|X) = P(X|Ci)·P(Ci) / P(X)
• Naïve Baye’s predicts that X belongs to class Ci if and
only if
P(Ci/X) > P(Cj/X) for 1<=j<=m, i!=j
29
Lecture-35 - Bayesian Classification
 Naïve Bayes Classifier

• P(X) is constant for all classes

• P(C1)=P(C2)=…..=P(Cn)
• P(X|Ci)·P(Ci) is to be maximize
• P(Ci)=|Ci,d|/|D|

30
Lecture-35 - Bayesian Classification
 Naïve Bayesian Classification

• Naïve assumption: attribute independence

P(x1,…,xk|C) = P(x1|C)·…·P(xk|C)
• If attribute is categorical:
P(xi|C) is estimated as the relative freq of samples
having value xi as i-th attribute in class C
• If attribute is continuous:
P(xi|C) is estimated thru a Gaussian density function
• Computationally easy in both cases

31
Lecture-35 - Bayesian Classification
 Play-tennis example: estimating P(xi|C)
outlook
Outlook Temperature Humidity Windy Class P(sunny|p) = 2/9 P(sunny|n) = 3/5
sunny hot high false N
sunny hot high true N P(overcast|p) = 4/9 P(overcast|n) = 0
overcast hot high false P
rain mild high false P P(rain|p) = 3/9 P(rain|n) = 2/5
rain cool normal false P
rain cool normal true N temperature
overcast cool normal true P
sunny mild high false N P(hot|p) = 2/9 P(hot|n) = 2/5
sunny cool normal false P
rain
sunny
mild
mild
normal false
normal true
P
P
P(mild|p) = 4/9 P(mild|n) = 2/5
overcast
overcast
mild
hot
high true
normal false
P
P
P(cool|p) = 3/9 P(cool|n) = 1/5
rain mild high true N
humidity
P(high|p) = 3/9 P(high|n) = 4/5
P(p) = 9/14 P(normal|p) = 6/9 P(normal|n) = 2/5
windy
P(n) = 5/14
P(true|p) = 3/9 P(true|n) = 3/5
P(false|p) = 6/9 P(false|n) = 2/5
33
Lecture-35 - Bayesian Classification
 Play-tennis example: classifying X

• An unseen sample X = <rain, hot, high, false>

• P(X|p)·P(p) =
P(rain|p)·P(hot|p)·P(high|p)·P(false|p)·P(p) =
3/9·2/9·3/9·6/9·9/14 = 0.010582
• P(X|n)·P(n) =
P(rain|n)·P(hot|n)·P(high|n)·P(false|n)·P(n) =
2/5·2/5·4/5·2/5·5/14 = 0.018286

• Sample X is classified in class n (don’t play)

34
Lecture-35 - Bayesian Classification
 How effective are Bayesian classifiers?

• makes computation possible

• optimal classifiers when satisfied
• but is seldom satisfied in practice, as attributes
(variables) are often correlated.
• Attempts to overcome this limitation:
– Bayesian networks, that combine Bayesian reasoning with
causal relationships between attributes
– Decision trees, that reason on one attribute at the time,
considering most important attributes first

35
Lecture-35 - Bayesian Classification
 Bayesian Belief Networks (I)
Family
Smoker
History
(FH, S) (FH, ~S)(~FH, S) (~FH, ~S)

LC 0.8 0.5 0.7 0.1

LungCancer Emphysema ~LC 0.2 0.5 0.3 0.9

The conditional probability table

for the variable LungCancer
PositiveXRay Dyspnea

Bayesian Belief Networks

36
Lecture-35 - Bayesian Classification
 Bayesian Belief Networks

• Bayesian belief network allows a subset of the variables

conditionally independent
• A graphical model of causal relationships
• Several cases of learning Bayesian belief networks
– Given both network structure and all the variables: easy
– Given network structure but only some variables
– When the network structure is not known in advance

37
Lecture-35 - Bayesian Classification
 Lecture-36
Classification by Backpropagation

38
 Classification by Backpropagation
• Neural network learning algorithm
• Psychologists and Neurobiologists
• Neural network – set of connected input/output
units in which each connection has a weight
associated with it.

39
Lecture-36 - Classification by Backpropagation
 Neural Networks

• Advantages
– prediction accuracy is generally high
– robust, works when training examples contain errors
– output may be discrete, real-valued, or a vector of
several discrete or real-valued attributes
– fast evaluation of the learned target function
• limitations
– long training time
– difficult to understand the learned function (weights)
– not easy to incorporate domain knowledge

40
Lecture-36 - Classification by Backpropagation
 A Neuron
- k
x0 w0
x1 w1
 f
output y
xn wn

Input weight weighted Activation

vector x vector w sum function
• The n-dimensional input vector x is mapped into variable y by
means of the scalar product and a nonlinear function mapping

41
Lecture-36 - Classification by Backpropagation
 Network Training
• The ultimate objective of training
– obtain a set of weights that makes almost all the tuples
in the training data classified correctly
• Steps
– Initialize weights with random values
– Feed the input tuples into the network one by one
– For each unit
• Compute the net input to the unit as a linear combination of all
the inputs to the unit
• Compute the output value using the activation function
• Compute the error
• Update the weights and the bias

42
Lecture-36 - Classification by Backpropagation
 Multi-Layer Perceptron

Output vector
Err j  O j (1  O j ) Errk w jk
Output nodes k

 j   j  (l) Err j
wij  wij  (l ) Err j Oi
Hidden nodes Err j  O j (1  O j )(T j  O j )
wij 1
Oj  I j
1 e
Input nodes
I j   wij Oi   j
i

Input vector: xi
43
Lecture-36 - Classification by Backpropagation
 Lecture-37
Classification based on concepts
from association rule mining

45
 Association-Based Classification

• Several methods for association-based

classification
– ARCS: Quantitative association mining and
clustering of association rules (Lent et al’97)
• It beats C4.5 in (mainly) scalability and also accuracy
– Associative classification: (Liu et al’98)
• It mines high support and high confidence rules in the
form of “cond_set => y”, where y is a class label

46
Lecture-37 - Classification based on concepts from association rule mining
 Association-Based Classification

– CAEP (Classification by aggregating emerging

patterns) (Dong et al’99)
• Emerging patterns (EPs): the itemsets whose support
increases significantly from one class to another
• Mine Eps based on minimum support and growth rate

47
Lecture-37 - Classification based on concepts from association rule mining
 Lecture-38
Other Classification Methods

48
 Other Classification Methods

• k-nearest neighbor classifier

• case-based reasoning
• Genetic algorithm
• Rough set approach
• Fuzzy set approaches

49
Lecture-38 - Other Classification Methods
 Instance-Based Methods
• Instance-based learning:
– Store training examples and delay the processing (“lazy
evaluation”) until a new instance must be classified
• Approaches
– k-nearest neighbor approach
• Instances represented as points in a Euclidean space.
– Locally weighted regression
• Constructs local approximation
– Case-based reasoning
• Uses symbolic representations and knowledge-based
inference

50
Lecture-38 - Other Classification Methods
 The k-Nearest Neighbor Algorithm
• All instances correspond to points in the n-D space.
• The nearest neighbor are defined in terms of
Euclidean distance.
• The target function could be discrete- or real-
valued.
• For discrete-valued, the k-NN returns the most
common value among the k training examples
nearest to xq.
• Vonoroi diagram:
_ _
the decision surface induced by
1-NN
+ for a typical set of training examples.

51
Lecture-38 - Other Classification Methods
 Discussion on the k-NN Algorithm
• The k-NN algorithm for continuous-valued target functions
– Calculate the mean values of the k nearest neighbors
• Distance-weighted nearest neighbor algorithm
– Weight the contribution of each of the k neighbors according
to their distance to the query point xq
• giving greater weight to closer neighbors
– Similarly, for real-valued target functions
• Robust to noisy data by averaging k-nearest neighborsw 1
d ( xq , xi )2
• Curse of dimensionality: distance between neighbors could be
dominated by irrelevant attributes.
– To overcome it, axes stretch or elimination of the least
relevant attributes.

52
Lecture-38 - Other Classification Methods
 Case-Based Reasoning
• Also uses: lazy evaluation + analyze similar instances
• Difference: Instances are not “points in a Euclidean space”
• Example: Water faucet problem in CADET (Sycara et al’92)
• Methodology
– Instances represented by rich symbolic descriptions ( function
graphs)
– Multiple retrieved cases may be combined
– Tight coupling between case retrieval, knowledge-based
reasoning, and problem solving
• Research issues
– Indexing based on syntactic similarity measure, and when
failure, backtracking, and adapting to additional cases

53
Lecture-38 - Other Classification Methods
 Lazy vs. Eager Learning
• Instance-based learning: lazy evaluation
• Decision-tree and Bayesian classification: eager evaluation
• Key differences
– Lazy method may consider query instance xq when deciding how to
generalize beyond the training data D
– Eager method cannot since they have already chosen global
approximation when seeing the query
• Efficiency: Lazy - less time training but more time predicting
• Accuracy
– Lazy method effectively uses a richer hypothesis space since it uses many
local linear functions to form its implicit global approximation to the
target function
– Eager: must commit to a single hypothesis that covers the entire instance
space

54
Lecture-38 - Other Classification Methods
 Genetic Algorithms
• GA: based on an analogy to biological evolution
• Each rule is represented by a string of bits
• An initial population is created consisting of randomly
generated rules
– e.g., IF A1 and Not A2 then C2 can be encoded as 100
• Based on the notion of survival of the fittest, a new population
is formed to consists of the fittest rules and their offsprings
• The fitness of a rule is represented by its classification accuracy
on a set of training examples
• Offsprings are generated by crossover and mutation

55
Lecture-38 - Other Classification Methods
 Rough Set Approach

• Rough sets are used to approximately or “roughly”

define equivalent classes
• A rough set for a given class C is approximated by
two sets: a lower approximation (certain to be in C)
and an upper approximation (cannot be described
as not belonging to C)
• Finding the minimal subsets (reducts) of attributes
(for feature reduction) is NP-hard but a
discernibility matrix is used to reduce the
computation intensity

56
Lecture-38 - Other Classification Methods
 Fuzzy Set Approaches
• Fuzzy logic uses truth values between 0.0 and 1.0 to
represent the degree of membership (such as using fuzzy
membership graph)
• Attribute values are converted to fuzzy values
– e.g., income is mapped into the discrete categories {low,
medium, high} with fuzzy values calculated
• For a given new sample, more than one fuzzy value may apply
• Each applicable rule contributes a vote for membership in the
categories
• Typically, the truth values for each predicted category are
summed

57
Lecture-38 - Other Classification Methods
Lecture-39
Prediction

58
 What Is Prediction?
• Prediction is similar to classification
– First, construct a model
– Second, use model to predict unknown value
• Major method for prediction is regression
– Linear and multiple regression
– Non-linear regression
• Prediction is different from classification
– Classification refers to predict categorical class label
– Prediction models continuous-valued functions

59
Lecture-39 - Prediction
 Predictive Modeling in Databases
• Predictive modeling: Predict data values or construct
generalized linear models based on the database data.
• One can only predict value ranges or category distributions
• Method outline:
– Minimal generalization
– Attribute relevance analysis
– Generalized linear model construction
– Prediction
• Determine the major factors which influence the prediction
– Data relevance analysis: uncertainty measurement, entropy
analysis, expert judgement, etc.
• Multi-level prediction: drill-down and roll-up analysis

60
Lecture-39 - Prediction
 Regress Analysis and Log-Linear Models in
Prediction
• Linear regression: Y =  +  X
– Two parameters ,  and  specify the line and are to be
estimated by using the data at hand.
– using the least squares criterion to the known values of Y1,
Y2, …, X1, X2, ….
• Multiple regression: Y = b0 + b1 X1 + b2 X2.
– Many nonlinear functions can be transformed into the
above.
• Log-linear models:
– The multi-way table of joint probabilities is approximated
by a product of lower-order tables.
– Probability: p(a, b, c, d) = ab acad bcd
61
Lecture-39 - Prediction
 Locally Weighted Regression
• Construct an explicit approximation to f over a local region
surrounding query instance xq.
• Locally weighted linear regression:
– The target function f is approximated near xq using the
linear function: f ( x)  w0  w1a1( x) wnan ( x)
– minimize the squared error: distance-decreasing weight K

E ( xq )  1  ( f ( x)  f ( x)) 2 K (d ( xq , x))
2 xk _nearest _neighbors_of _ x
– the gradient descent training rule: q

w j    K (d ( xq , x))(( f ( x)  f ( x))a j ( x)
• k _ nearest
In mostx cases, the _target
neighbors _ of _ xqis approximated by a
function
constant, linear, or quadratic function.

62
Lecture-39 - Prediction
 Lecture-40
Classification accuracy

63
 Classification Accuracy: Estimating Error Rates
• Partition: Training-and-testing
– use two independent data sets- training set , test set
– used for data set with large number of samples
• Cross-validation
– divide the data set into k subsamples
– use k-1 subsamples as training data and one sub-sample as
test data --- k-fold cross-validation
– for data set with moderate size
• Bootstrapping (leave-one-out)
– for small size data
64
Lecture-40 - Classification accuracy
 Boosting and Bagging
• Boosting increases classification accuracy
– Applicable to decision trees or Bayesian
classifier
• Learn a series of classifiers, where each classifier
in the series pays more attention to the examples
misclassified by its predecessor
• Boosting requires only linear time and constant
space

65
Lecture-40 - Classification accuracy
 Boosting Technique — Algorithm

• Assign every example an equal weight 1/N

• For t = 1, 2, …, T Do
– Obtain a hypothesis (classifier) h(t) under w(t)
– Calculate the error of h(t) and re-weight the
examples based on the error
– Normalize w(t+1) to sum to 1
• Output a weighted sum of all the hypothesis, with
each hypothesis weighted according to its accuracy
on the training set

66
Lecture-40 - Classification accuracy