How to Specify Split in a Decision Tree in R Programming?

Decision trees are versatile and widely used machine learning algorithms for both classification and regression tasks. A fundamental aspect of building decision trees is determining how to split the dataset at each node effectively. In this comprehensive guide, we will explore the theory behind decision tree splitting and demonstrate how to specify splits in R Programming Language using a practical dataset.

Understanding Decision Tree Splitting

Decision tree splitting involves partitioning the dataset into subsets based on the values of a chosen feature. The goal is to create splits that result in homogeneous subsets with respect to the target variable. Different splitting criteria are used to evaluate the quality of splits and select the best split at each node.

Splitting Criteria

In decision tree algorithms, the splitting criteria determine how the decision tree selects the best feature and threshold to split the data at each node. There are several common splitting criteria used in decision trees, each aiming to maximize the purity of the resulting child nodes. Here are some commonly used splitting criteria:

• Gini Impurity: Gini impurity measures the probability of misclassifying a randomly chosen element if it were randomly classified according to the distribution of labels in the node. Lower Gini impurity indicates a more homogeneous subset.
• Information Gain: Information gain measures the reduction in entropy (or uncertainty) achieved by splitting the data based on a particular feature. Higher information gain implies a more informative split.
• Chi-Square Test: Chi-square test evaluates the independence between the feature and the target variable. Significant p-values indicate a strong association between the feature and the target variable.
• Gain Ratio: Gain ratio adjusts information gain to account for the number of branches created by the split. It penalizes splits that create many branches.

Specifying Splits in R

In R, decision trees can be built using various packages, with the rpart package being a popular choice. Let's demonstrate how to specify splits in R using the rpart package with a practical dataset.

Suppose we have a dataset containing demographic information (age, income) and a binary target variable (purchase decision). We want to build a decision tree to predict whether a customer will make a purchase based on their demographic attributes.

# Load required libraries
install.packages("rpart")
library(rpart)

# Generate example dataset
set.seed(123)  # Set seed for reproducibility

# Create customer data
customer_data <- data.frame(
  age = round(rnorm(100, mean = 30, sd = 5)),  # Generate random ages
  income = round(rnorm(100, mean = 50000, sd = 10000)),  # Generate random incomes
  gender = sample(c("Male", "Female"), 100, replace = TRUE),  # Sample gender
  purchase = sample(c("Yes", "No"), 100, replace = TRUE)  # Sample purchase
)

# Build decision tree using rpart
tree_model <- rpart(purchase ~ age + income + gender, data = customer_data,
                    method = "class", 
                    control = rpart.control(minsplit = 10, minbucket = 5))
summary(tree_model)

Output:

Call:
rpart(formula = purchase ~ age + income + gender, data = customer_data, 
    method = "class", control = rpart.control(minsplit = 10, 
        minbucket = 5))
  n= 100 

          CP nsplit rel error   xerror      xstd
1 0.04651163      0 1.0000000 1.000000 0.1151339
2 0.03488372      4 0.8139535 1.441860 0.1128806
3 0.02325581      8 0.6744186 1.581395 0.1084828
4 0.01000000     12 0.5581395 1.604651 0.1075566

Variable importance
income    age gender 
    74     22      3 

Node number 1: 100 observations,    complexity param=0.04651163
  predicted class=No   expected loss=0.43  P(node) =1
    class counts:    57    43
   probabilities: 0.570 0.430 
  left son=2 (33 obs) right son=3 (67 obs)
  Primary splits:
      income < 52461.5 to the right, improve=0.9204975, (0 missing)
      age    < 37.5    to the left,  improve=0.6613043, (0 missing)
      gender splits as  LR,          improve=0.5000000, (0 missing)
  Surrogate splits:
      age < 38.5    to the right, agree=0.68, adj=0.03, (0 split).........................................................................

The summary(tree_model) function provides a summary of the decision tree model built using the rpart package in R. Here's an explanation of the typical output:

• Call: This section displays the function call used to create the decision tree model. It shows the formula used for model building, including the response variable and predictor variables.
• Decision Tree: This section provides a textual representation of the decision tree. It shows the splits made at each node of the tree, along with the number of observations and the predicted class at each terminal node (also known as a leaf).
• Variables actually used in tree construction: This part lists the predictor variables (features) used in constructing the decision tree. It shows which variables were included in the final tree model.
• Root node error: This indicates the error rate associated with the root node of the decision tree. It represents the proportion of misclassified observations at the top level of the tree.
• Residual mean deviance: This value represents the mean deviance of the residuals after fitting the tree. It is a measure of the goodness of fit of the model, with lower values indicating a better fit.
• Misclassification error rate: This indicates the overall misclassification error rate of the tree model. It represents the proportion of misclassified observations in the entire dataset.
• Variable importance: This section provides information about the importance of each predictor variable in the decision tree model. It ranks the variables based on their contribution to the model's performance.
• Node number: This column displays the node number in the decision tree. Each node represents a decision point where the data is split based on a certain criterion.
• Splitting criteria: This column specifies the splitting criterion used at each node to partition the data. It may include information such as the variable name, split point, and other relevant details.
• Number of observations in each node: This column shows the number of observations associated with each node of the decision tree.

Visualize decision tree

To visualize the decision tree created using the rpart package in R, you can use the rpart.plot package, which provides functions for plotting decision trees.

# Visualize decision tree
plot(tree_model)
text(tree_model)

Output:

This will generate a visual representation of the decision tree, making it easier to interpret the splits and understand how the model makes predictions.

Conclusion

By mastering the art of decision tree split specification in R, data analysts and machine learning practitioners can build accurate and interpretable models for classification and regression tasks. Experimenting with different splitting criteria and tuning parameters can help optimize the performance of decision tree models and unlock valuable insights from the data. With the tools and techniques discussed in this guide, you're well-equipped to harness the power of decision trees for predictive modeling in R programming.