Tree Based Methods: Classification Trees

Rebecca C. Steorts, Duke University

STA 325, Chapter 8 ISL

1 / 22
Agenda

I What are tree based methods?

I Regression trees
I Classification trees

2 / 22
Regression versus Classification trees

Classification trees are similar to regression trees, except that it is

used to predict a qualitative response rather than a quantitative
one.
For a regression tree, the prediction response for an observation is
given by the mean response of the training observations that belong
to the same terminal node.
In contract, for a classification tree, we predict that each
observation belongs to the most commonly occurring class of
training observations in the region to which it belongs.

3 / 22
Interpretation of classification trees

We are usually interterested in

I Interpreting the results of the classification tree.

I The class prediction corresponding to a particular terminal
node region and
I the class proportions among the training observations that fall
into that region.

4 / 22
How to grow a classification tree?

I Growing a classification tree is very similar to that of a

regression tree.
I We use a recursive binary split to grow the tree, however in the
classification setting, the RSS cannot be used as a criterion for
making splits.
I A natural alternative to the RSS is the classification error rate.

5 / 22
Classification error rate

The classification error rate is simply the fraction of the training

observations in that region that do not belong to the most common
class:

E = 1 − max p̂mk (1)

k

where p̂mk represents the proportion of training observations in the

region m that are from class k.
It turns out that the classification er- ror is not sufficiently sensitive
for tree-growing and two other measures are preferable (Gini-index
and cross-entropy).

6 / 22
The Gini index

Recall that p̂mk represents the proportion of training observations in

the region m that are from class k.
The Gini index is defined by

K
X
G= p̂mk (1 − p̂mk ) (2)
k=1

which is a measure of the total variance across the K classes.

7 / 22
The Gini index

The Gini index takes on a small value if all of the p̂mk are close to 0
or 1.
Because of this, the Gini index is called a measure of node purity.
For example, a small value of p̂mk indicates indicates that a node
contains predominantly observations from a single class.

8 / 22
The cross entropy

An alternative to the Gini index is the cross entropy

K
X
D=− p̂mk log(p̂mk ) (3)
k=1

Since
0 ≤ p̂mk ≤ 1
this implies that
0 ≤ −p̂mk log(p̂mk ).

Exercise: The cross entropy will take on a value near 0 if the p̂mk ’s
are all near 0 or 1.

9 / 22
Gini index, cross entropy, and classification error rate

Like the Gini index, the cross-entropy will take on a small value if
node m is pure.
In fact, it turns out that the Gini index and the cross-entropy are
quite similar numerically.
When building a classification tree, either the Gini index or the
cross- entropy are typically used to evaluate the quality of a
particular split, since these two approaches are more sensitive to
node purity than is the classification error rate.
Any of these three approaches might be used when pruning the tree,
but the classification error rate is preferable if prediction accuracy of
the final pruned tree is the goal.

10 / 22
Deviance

For classification trees, the deviance is given by the summary()

function and can be calculated via

XX
−2 nmk log(pmk
ˆ ) (4)
m k

where nmk is the number of observations in the mth terminal node

that belongs to class k.
A small deviance indicates a tree that provides a good fit to the
(training) data.
The residual mean deviance reported is simply the deviance divided
by n − |To |.

11 / 22
Application to Carseats Dataset

We use regression and classification trees to analyze the Carseats

data set.
In the Carseats data, Sales is a continuous variable, and so we begin
by recording it as a binary variable.

12 / 22
Task 1

1. First use the ifelse() function to create a variable, called High,

which takes on a value of Yes if the Sales variable exceeds 8,
and takes on a value of No otherwise. This creates a binary
variable.
2. Next, use a data.frame() to merge High with the rest of the
Carseats data.
3. Finally, use the tree() function to fit a classification tree in
order to predict high using all variables but Sales.

13 / 22
Solution Task 1 (a,b,c)

library(ISLR)
library(tree)
attach(Carseats)
# creating a binary variable
High <- ifelse(Sales <= 8, "No", "Yes")
# merge High with the rest of the Carseats
Carseats <- data.frame(Carseats, High)
# fit a regression tree
tree.carseats <- tree(High ~. - Sales, Carseats)

14 / 22
Task 2

How does the tree fit? Plot your tree and explain your explaination.

15 / 22
Solution Task 2

In order to answer Task 2, we will look at the summary of our tree

and also plot the tree. In general, we have just built the tree, so
remember that we will need to prune it back!

16 / 22
Solution Task 2 (continued)

summary(tree.carseats)

##
## Classification tree:
## tree(formula = High ~ . - Sales, data = Carseats)
## Variables actually used in tree construction:
## [1] "ShelveLoc" "Price" "Income" "CompPrice
## [6] "Advertising" "Age" "US"
## Number of terminal nodes: 27
## Residual mean deviance: 0.4575 = 170.7 / 373
## Misclassification error rate: 0.09 = 36 / 400

17 / 22
Solution to Task 2 (Continued) ShelveLoc: Bad,Medium

Price < 92.5 Price < 135

US: No Income < 46

Price < 109

Income < 57 Advertising < 13.5 No Yes

Yes

CompPrice < 110.5 Population < 207.5

Yes No

No Yes Yes Yes CompPrice < 124.5 Age < 54.5

CompPrice < 130.5 CompPrice < 122.5

Price < 106.5 Price < 122.5
Income < 100
Price < 125

Population < 177 Yes

No
No Yes
Income < 60.5 No
ShelveLoc: Bad CompPrice < 147.5
Yes No
No

Price < 109.5

Price < 147
No Yes
No
No CompPrice < 152.5
Age < 49.5

No
Yes
Yes No

Yes No

The most important indicator of Sales appears to be shelving

location, since the first branch differentiates Good locations from
Bad and Medium locations.
18 / 22
Task 3

1. What is the training error?

2. What is the residual mean error or deviance?
3. What must we do to properly evaluate the performance of a
classification tree on the data? (Hint: use the predict function
and be sure to explain any results that you obtain). Note: in
this step break the data into training and testing data. Don’t
prune the data yet!

19 / 22
Solution to Task 3

20 / 22
Task 4

Given Task 3, now prune the tree to see if you get improved results.
Note: use the argument FUN=prune.misclass in order to indicate
that we want the classification error rate to guide the
cross-validation and pruning process, rather than the default for the
cv.tree() function, which is the deviance.

21 / 22
Solution to Task 4

22 / 22