Data Analysis in Management

Programme: Modern Business Management

2022/2023

Roman Huptas, Department of Statistics,

Cracow University of Economics
 Lectures 7 and 8

Classification methods – k-nearest neighbour

algorithm, decision trees
Outline of the lecture

 Introduction to classification methods

 k-nearest neighbour algorithm for classification

 Decision trees: classification and regression trees

 Model evaluation techniques for classification tasks

– classification matrix and measures of predictive
accuracy

3
Learning objectives
Upon completing this lecture, you should be able to:

 identify the types of research problems and questions

addressed by classification algorithms,

 understand the key terms and basics of k-nearest

neighbour algorithm and decision trees,

 evaluate accuracy of classification methods and

algorithms.
WHAT TASKS CAN CLASSIFICATION METHODS ACCOMPLISH?

 Classification task
 One of the common data mining task is that of
classification.

 Classification differs from clustering in that there is a target

variable for classification.

 In classification, there is a target categorical variable, which

is partitioned into predetermined classes or categories.

 Classification is similar to estimation, except that the target

variable is categorical rather than numeric.
WHAT TASKS CAN CLASSIFICATION METHODS ACCOMPLISH?

 Classification task
 The classification model examines a large set of records, each
record containing information on the target variable as well as a
set of input or predictor variables.
WHAT TASKS CAN CLASSIFICATION METHODS ACCOMPLISH?

 Classification task
 Suppose that there is a target categorical variable, such as
income bracket, which, for example, could be partitioned
into three classes or categories: high income, middle income,
and low income.

 Suppose the researcher would like to be able to classify the

income bracket of new individuals, not currently in the
above database, based on the other characteristics
associated with that individual, such as age, gender, and
occupation.

 This task is a classification task, very nicely suited to

multivariate or data mining methods and techniques.
WHAT TASKS CAN CLASSIFICATION METHODS ACCOMPLISH?

 Examples of classification tasks

 Data analysts are frequently faced with the need to predict a
categorical outcome from a set of predictor variables.
 Examples of classification tasks may be found in nearly every field
of endeavour.
 Examples of classification tasks include, for instance:
 Determining whether a particular credit card transaction is
fraudulent.
 Assessing whether a mortgage application is a good or bad
credit risk.
 Predicting whether an individual will repay a loan, given their
demographics and financial history.
 Deciding whether an email is spam, given the presence of key
words, images, hypertext, header information, and origin.
WHAT TASKS CAN CLASSIFICATION METHODS ACCOMPLISH?

 Examples of classification tasks

 Examples of classification tasks include, for instance:
 Placing a new student into a particular track with regard to
special needs.

 Diagnosing whether a particular disease is present;

 In security field identifying whether or not certain financial or

personal behaviour indicates a possible terrorist threat.

 Determining whether a patient is having a heart attack, based

on their symptoms and vital signs.
WHAT TASKS CAN CLASSIFICATION METHODS ACCOMPLISH?

 Examples of classification tasks

 Each of these cases involves the prediction of a binary categorical
outcome (good credit risk/bad credit risk, heart attack/no heart
attack, spam/not spam) from a set of predictors (also called
features).

 The goal is to find an accurate method of classifying new cases

into one of the two groups.
 WHAT TASKS CAN CLASSIFICATION METHODS ACCOMPLISH?
 The field of multivariate data analysis and supervised machine
learning offers numerous classification methods that can be used to
predict categorical outcomes.

 Common multivariate and data mining methods and techniques

used for classification are:
 k-Nearest Neighbour (kNN) algorithm,
 classification and regression trees,
 logistic regression,
 linear discriminant analysis,
 neural networks,
 support vector machines,
 random forests.
WHAT TASKS CAN CLASSIFICATION METHODS ACCOMPLISH?

 How does classifier work?

 Given: dataset of instances with known categories
 Goal: using the “knowledge” in the dataset, classify a given
instance; predict the category of the given instance that is
rationally consistent with the dataset
WHAT TASKS CAN CLASSIFICATION METHODS ACCOMPLISH?

 How does classifier work?

 The classification algorithm would proceed roughly as follows.

 The procedure starts with a set of observations containing values

for both the predictor variables and the outcome.

 First, examine the data set containing both the predictor

variables and the (already classified) target variable.

 The dataset is then divided into a training sample and a validation

sample.
WHAT TASKS CAN CLASSIFICATION METHODS ACCOMPLISH?

 How does classifier work?

 The algorithm “learns about” which combinations of variables are
associated with which categories of target (categorical) variable.
This data set is called the training set.

 Then the algorithm would look at new records for which no

information about category of target variable is available.

 Based on the classifications in the training set, the algorithm

would assign classifications to the new records.

 Thus, a classification (predictive) model is developed using the

data in the training sample and tested for accuracy using the data
in the validation sample.
 WHAT TASKS CAN CLASSIFICATION METHODS ACCOMPLISH?

 How does classifier work?

 Usually, the given data set is divided into training

and test sets, with training set used to build the
model.
 A test set is used to determine the accuracy of the
model.

15
 WHAT TASKS CAN CLASSIFICATION METHODS ACCOMPLISH?

Illustrating Classification Task

Tid Attrib1 Attrib2 Attrib3 Class Learning
1 Yes Large 125K No
algorithm
2 No Medium 100K No

3 No Small 70K No

4 Yes Medium 120K No

Induction
5 No Large 95K Yes

6 No Medium 60K No

7 Yes Large 220K No Learn

8 No Small 85K Yes Model
9 No Medium 75K No

10 No Small 90K Yes

Model
10

Training Set
Apply
Tid Attrib1 Attrib2 Attrib3 Class Model
11 No Small 55K ?

12 Yes Medium 80K ?

13 Yes Large 110K ? Deduction

14 No Small 95K ?

15 No Large 67K ?
10

Test Set
16
k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 What is kNN?

 One of the top data mining algorithms used today.

 A powerful classification algorithm used in pattern recognition.

 One of the simplest of all machine learning classifiers.

 k-nearest neighbour is an example of instance-based learning.

 In instance-based learning the training data set is stored, so that a

classification for a new unclassified record may be found simply
by comparing it to the most similar records in the training set.
k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 What is kNN?
 kNN is one of the simplest supervised machine learning
algorithm mostly used for

classification.

 It classifies a data point based on how its neighbors are classified.

 What does supervised machine learning mean? We are given

input samples (X) and output samples (y) of a function y = f(X).
We would like to “learn” f, and evaluate it on new data.Types:
 Classification: y is discrete (class labels).
 Regression: y is continuous, e.g. linear regression.
k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 What is kNN?
Idea of KNN

Given a new unlabeled object (a new data point, record, a

new instance), we find – in the data set – k „closest”
objects (data points, records, instances) to a new
unlabeled object, i.e. the nearest neighbours, and then a
new object is assigned to the most common class
amongst its k nearest neighbours.
k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 What is kNN?
 To run a kNN algorithm, it is required:
 The set of stored objects (records).
 Distance metric to compute
distances between objects (records).
 The value of k – the number of
nearest neighbours.

 To classify a new unknown object

(record), once has to:
 Compute distances to other training
objects (records).
 Identify k nearest neighbours.
 Use class labels of k nearest
neighbours to determine the class
label of a new unknown object
(record).
k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 What is kNN?
 k-nearest neighbours of an object (data point) x are data points
that have the k smallest distance (?) to x
k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 What is kNN?
Distance between objects (neighbours)
 How do we define similar?
 Data analysts define distance metrics to measure similarity.
 The most popular measure of the distance between two objects is the
Euclidean distance, but the Manhattan, Canberra, asymmetric binary,
maximum, and Minkowski distance measures are also available.
 Euclidean distance is the most commonly recognized measure of
distance.
 The Euclidean distance between two observations is given by:

where i and j are observations and P is the number of variables.

k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 What is kNN?
Distance between objects (neighbours)

 An example of how Euclidean distance is obtained in the case of two

dimensions is shown geometrically in the figure below:
k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 What is kNN?
The kNN classification algorithm
 All the objects (instances) correspond to points in an
n-dimensional variable (feature) space.
 Each object (instance) is represented with a set of numerical
attributes.
 Each object of the training data consists of a set of vectors and a
class label associated with each vector.
 Classification is done by comparing feature vectors of different k
nearest points.
 Select the k-nearest objects (neighbours) to a new unlabeled
object in the training set.
 Assign a new unlabeled object to the most common class among
its k-nearest neighbours.
k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 How to determine the class of a new object from
nearest neighbour list?

 Now that we have a method of determining which records are

most similar to the new, unclassified record, we need to establish
how these similar records will combine to provide a classification
decision for the new record.

 One approach is to take the majority vote of class labels among

the k-nearest neighbours (a new point is assigned the most
frequent label of its k nearest neighbours)
k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 How to determine the class of a new object from
nearest neighbour list?
k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 How to determine the class of a new object from
nearest neighbour list?

 However, one may feel that neighbours that are closer or more
similar to the new record should be weighted more heavily than
more distant neighbours.

 Approach: weigh the vote according to distance

 In weighted voting, the influence of a particular record is

inversely proportional to the distance of the record from the
new record to be classified.
k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 How to choose k?
 If k is too small, algorithm is sensitive to „noise” objects
(observations). The classification may be affected by outliers or
unusual observations (“noise”).

 If k is too large, the neighbourhood may include objects

(observations) from other classes.

 The data analyst needs to balance these considerations when

choosing the value of k.

 Once would try various values of k with different randomly

selected training sets and choose the value of k that minimizes
the classification error.
k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 What is kNN?
 Advantages:

 Non-complex and very easy to understand and implement.

 Useful for non linear data because there are no assumptions

about data.

 High accuracy (relatively) if the sample is large enough.

 It can be used both for classification or regression.

 Best used where the probability distribution is unknown

k-NEAREST NEIGHBOUR (kNN) ALGORITHM
 What is kNN?
 Disadvantages:

 Computationally expensive.

 Lot of space is consumed because all the data points are

stored.

 Sensitive to irrelevant features and the scale of the data.

 Output purely depends on k value chosen by user which can

reduce accuracy for some values.

 Choosing k may be tricky.

DECISION TREES
 Many decisions are tree structures
DECISION TREES
 What is decision tree?
 A decision tree is a flexible data-driven method that can be used
for both classification (called classification tree) and prediction
(called regression tree).
 Among the data-driven methods, trees are the most transparent
and easy to interpret.
 Trees are based on separating records into subgroups by creating
splits on predictors.
 These splits create logical rules that are transparent and easily
understandable, for example:
“IF Age < 55 AND Education > 12 THEN class = 1.”
DECISION TREES
 What is decision tree?

 A decision tree is a hierarchically organized structure, with each

node splitting the data space into pieces based on value of a
feature.

 Equivalent to a partition of the data space into K disjoint

feature subspaces.

 On each feature subspace, the same decision / prediction is

made for all objects (items).
DECISION TREES
 Terminology
 Parent of a node c is the immediate predecessor node.
 Children of a node c are the immediate successors of c,
equivalently nodes which have c as a parent.
 Root node is the top node of the tree; the only node without
parents.
 Decision nodes are nodes that have successors.
 Leaf nodes (or terminal nodes or leaves) are nodes which
do not have children (successors).
 A K-tree is a tree where each node (except for leaf nodes) has K
children. Usually working with binary trees (K = 2).
 Depth of a tree is the maximal length of a path from the root
node to a leaf node.
DECISION TREES
 Terminology
DECISION TREES
 What is decision tree?
 A classical decision tree classifies data cases using a conjunction of
rules organized into a binary tree structure.
 Each internal node in a classical decision tree contains a rule of
the form (xd < t) or (xd = t) that tests a single data dimension d
against a single value t and assigns the data case to it’s left or right
sub-tree according to the result.
 A data item is routed through the tree from the root to a leaf.
Each leaf node is associated with a class, and a data item is
assigned the class of the leaf node it is routed to.
DECISION TREES
 Type of trees

 Classification trees – if the response variable is

categorical (nominal).

 Regression trees – if the response variable is

continous.
DECISION TREES
 Type of trees
 Classification trees try to predict the class probabilities at
the leaves, such as
 the probability that there will be rain, the probability of
defaulting on a loan, or the preference for one of five
different types of movie genres.

 Regression trees try to predict a (numeric) mean response at

the leaves of the tree, such as
 the expected amount of rain in inches, or the expected
default rate on loans.
 DECISION TREES
 Type of trees
Moreover, we will only consider:
 binary trees where nodes have only 2 children (vs multiway trees
where nodes can have more than 2 children)
 decisions at each node involve only a single feature (i.e. input
coordinate)
DECISION TREES
 Illustration
DECISION TREES
 Learning a tree model
 The structure of the tree
 The threshold values
 The values for the leafs (A,B,...)
 DECISION TREES
 Example
Let us start the discussion with a very simple example of a decision
tree. Here, the decision is whether or not to take an umbrella. For
example, if the forecast predicts rain with a probability between 30%
and 70% and if it is cloudy when you leave the house, you better take
an umbrella.Tree logic uses a series of steps to come to a conclusion.
DECISION TREES

 Two key ideas underlie decision trees:

 The first is the idea of recursive partitioning of the

space of the predictor variables.

 The second is the idea of pruning using validation

data.
DECISION TREES
 Recursive partitioning
 Regression and classification trees partition a data set into
exhaustive and nonoverlapping nodes.
 The data partitioning is
 nonoverlapping (every element of the data set is part of just
one node)
 and exhaustive (every element of the data set must fall into
one of the nodes).
 At each node of the tree, the response of interest is
summarized by
 an average response (for regression trees)
 a frequency distribution (for classification trees).
DECISION TREES
 Recursive partitioning
 The approach is also referred to as divide-and-conquer
partitioning.
 At each stage, the data set is split into two nonoverlapping,
smaller data sets.
 The objective of a split is to increase the homogeneity (!) of
the resulting smaller data sets with respect to the target
variable (alternatively, we can think of this as decreasing the
impurity (!), or the disorder).
 We continually divide the data set by creating node splits of
smaller and smaller data sets.
DECISION TREES

 Classification trees

 Given the data set with categorical response variable minimize

the classification error in each leaf.
 The estimated probability of each class k in region Rj is simply
the frequency in which label k occurs in the leaf Rj .
 These estimates can be regularized as well.
DECISION TREES

 What is the optimal split? How to evaluate the split?

 Split points are ranked according to how much they reduce

impurity (heterogeneity) in the resulting rectangle.

 We need a measure of node impurity (disorder) that

guides our splitting of the data set and that tells us whether it
makes sense to split a certain node even further.
DECISION TREES
 Measures of impurity (uncertainty)

 There are a number of ways to measure impurity. The

three most popular measures are:

 Misclassification error,
 Entropy,
 Gini index.
DECISION TREES
 Measures of impurity (uncertainty)

 Misclassification error:

1- max (pk)

where pk is the proportion of records in rectangle A

that belong to class k.
DECISION TREES
 Measures of impurity (uncertainty)
 Entropy:
 Let us denote the m classes of the response variable by
k = 1, 2, …, m.
 The entropy for a rectangle A is defined by:

where pk is the proportion of records in rectangle A that belong to

class k.
 This measure ranges between 0 (most pure, all records belong to
the same class) and log2(m) (when all m classes are represented
equally).
DECISION TREES
 Measures of impurity (uncertainty)
 Entropy:

 It measures the information content of a random

variable, with small values of entropy reflecting
homogeneity or little impurity (disorder).
DECISION TREES
 Measures of impurity (uncertainty)
 Gini index:
 Let us denote the m classes of the response variable by
k = 1, 2, …, m.
 The Gini impurity index for a rectangle A is defined by:

where pk is the proportion of records in rectangle A that belong to

class k.
 This measure takes values between 0 (when all the records belong
to the same class) and (m-1)/m (when all m classes are equally
represented).
 DECISION TREES
 Measures of impurity (uncertainty)
 The values of uncertainty measures for a two-class case as
a function of the :proportion of records in class 1
DECISION TREES
 Measures of impurity (uncertainty)

 The decision tree construction algorithms

recursively search for optimal (variable,
threshold) pairs according to a given criteria
down to a specified depth.
DECISION TREES

 What is the optimal tree?

 We need to be careful to pick an appropriate tree depth.

 If the tree is too deep, we can overfit.
 If the tree is too shallow, we underfit.
 Max depth is a hyper-parameter that should be tuned by the
data.
 Alternative strategy is to prune the full-grown tree.
DECISION TREES
 Pruning

 The idea behind pruning is to recognize that a very large tree is

likely to overfit the training data, and that the weakest branches,
which hardly reduce the error rate, should be removed.

 Pruning consists of successively selecting a decision node and

redesignating it as a terminal node (lopping off the branches
extending beyond that decision node (its subtree) and thereby
reducing the size of the tree).
DECISION TREES
 Classification trees
DECISION TREES
 Regression trees
 The tree method can also be used for a numerical outcome
variable.
 Regression trees for prediction operate in much the same
fashion as classification trees.
 The outcome variable (Y ) is a numerical variable in this case,
but both the principle and the procedure are the same.
 Many splits are attempted, and for each, we measure “impurity”
in each branch of the resulting tree.
 The tree procedure then selects the split that minimizes the
sum of such measures.
DECISION TREES
 Regression trees – example
DECISION TREES
 Regression trees
 Given the data set with categorical response variable minimize
the error (e.g. square loss error) in each leaf.
 the parameterized function is

 Using squared loss, optimal parameters are:

which is sample mean.

DECISION TREES
 Regression trees

 For each leaf, we need to assign the prediction y which

minimizes the loss for the regression problem
DECISION TREES
 Algorithm for Regression Trees
DECISION TREES
 Example of Regression Trees
DECISION TREES
 Example of Regression Trees
DECISION TREES
 Example of Regression Trees
DECISION TREES
 Regression trees – prediction (!)
 In classification trees, the value of the terminal
node (which is one of the categories) is determined
by the “voting” of the training records that were in
that terminal node.

 In regression trees, the value of the terminal node

is determined by the average outcome value of the
training records that were in that terminal node.
DECISION TREES
 Regression trees – measures of impurity (!)
 In regression trees, a typical impurity measure is the
sum of the squared deviations from the mean of
the terminal node.

 This is equivalent to the sum of the squared errors,

because the mean of the terminal node is exactly the
prediction.
DECISION TREES
 Pruning rule
 Stop when one instance in each leaf (regression problem)
 Stop when all the instance in each leaf have the same label
(classcation problem)
 Stop when the number of leaves is less than the threshold
 Stop when the leaf's error is less than the threshold
 Stop when the number of instances in each leaf is less than the
threshold
 Stop when the p-value between two divided leafs is larger than
the certain threshold (e.g. 0.05 or 0.01) based on chosen
statistical tests.
Evaluating classification accuracy
Model evaluation techniques for the
classification tasks
 How do we assess how well our classification algorithm is
functioning?

 What are the ways to evaluate the predictive accuracy of the

classification methods or algorithms?
Evaluating classification accuracy
Model evaluation techniques for the
classification tasks
 To determine the predictive ability of a classification method,
the researcher must construct classification matrix.

 The classification matrix procedure provides a perspective on

practical significance rather than statistical significance.

 The classification matrix, also referred to the confusion

matrix, is the matrix of the correct and incorrect
classifications made by a classification algorithm.
Evaluating classification accuracy
Model evaluation techniques for the
classification tasks
 The classification matrix is a two-by-two contingency table in
which the columns represent the predicted classifications, and
the rows represent the actual (true) classifications.

 The entries on the diagonal of the matrix represent the

number of individuals correctly classified. The numbers off the
diagonal represent the incorrect classifications.
Evaluating classification accuracy
Classification matrix –
general form of the contingency table of correct and incorrect
classifications

Actual Predicted category/group

category/ Total
group Group 0 Group 1

True negatives (TN) False positives (FP) Total

Group 0 Predicted group → 0 Predicted group → 1 actually
Actual group → 0 Actual group → 0
negative

False negatives (FN) True positives (TP) Total

Group 1 Predicted group → 0 Predicted group → 1 actually
Actual group → 1 Actual group → 1
positive

Total Total Grand

Total
predicted negative predicted positive total
Evaluating classification accuracy
Measures of predictive accuracy
 Let TN, FN, FP, and TP represent the numbers of true
negatives, false negatives, false positives, and true
positives, respectively, in our classification matrix.

 Then we may define our evaluation measures as

follows:
 Evaluating classification accuracy
Measures of predictive accuracy

 Hit ratio (also called hit rate, overall accuracy, ACC or the
percentage correctly classified)
 Percentage of observations (objects, individuals,
respondents, firms, etc.) correctly classified.
 It is calculated as the number of objects in the diagonal of
the classification matrix divided by the total number of
observations.
𝑇𝑁 + 𝑇𝑃
ℎ𝑖𝑡_𝑟𝑎𝑡𝑖𝑜 =
𝑇𝑁 + 𝐹𝑁 + 𝐹𝑃 + 𝑇𝑃
Note! The higher, the better !
 Evaluating classification accuracy
Measures of predictive accuracy

 Sensitivity (also called recall or true positive rate)

 Probability of getting a positive classification when the true
outcome is positive.
 A good classification model should be sensitive, meaning
that it should identify a high proportion of the observations
which are positive.
 Sensitivity is defined as:
Number of true positives 𝑇𝑃
𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 = =
Number of actual positives 𝑇𝑃 + 𝐹𝑁
Note! The higher, the better !
 Evaluating classification accuracy
Measures of predictive accuracy

 Specificity (also called true negative rate)

 Probability of getting a negative classification when the true
outcome is negative.
 A classification model also needs to be specific, meaning
that it should identify a high proportion of the observations
which are negative.
 Specificity is defined as:
Number of true negatives 𝑇𝑁
𝑆𝑝𝑒𝑐𝑖𝑓𝑖𝑐𝑖𝑡𝑦 = =
Number of actual negatives 𝑇𝑁 + 𝐹𝑃
Note! The higher, the better !
 Evaluating classification accuracy
Measures of predictive accuracy

 Positive predictive value (also called precision or

proportion of true positives)
 Probability that an observation with a positive classification
is correctly identified as positive.
 It is defined as:

𝑇𝑃
𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑣𝑒 𝑣𝑎𝑙𝑢𝑒 = 𝑃𝑃𝑉 =
𝑇𝑃 + 𝐹𝑃

Note! The higher, the better !

 Evaluating classification accuracy
Measures of predictive accuracy

 Negative predictive value (also called proportion of

true negatives)
 Probability that an observation with a negative classification
is correctly identified as negative.
 It is defined as:

𝑇𝑁
𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑝𝑟𝑒𝑑𝑖𝑐𝑡𝑖𝑣𝑒 𝑣𝑎𝑙𝑢𝑒 = 𝑁𝑃𝑉 =
𝑇𝑁 + 𝐹𝑁

Note! The higher, the better!