Scribd
Upload
9 views5 pages

Decision Tree

This document provides a beginner-friendly guide to the Decision Tree Classifier using the ID3 algorithm, explaining its structure, advantages, and disadvantages. It includes a step-by-step process for building a decision tree, calculating entropy and information gain, and implementing the algorithm in Python with a practical example using the Iris dataset. The guide also discusses when to use ID3 and offers tips for effective implementation.

Uploaded by

Arkojyoti Dey
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
9 views5 pages

Decision Tree

This document provides a beginner-friendly guide to the Decision Tree Classifier using the ID3 algorithm, explaining its structure, advantages, and disadvantages. It includes a step-by-step process for building a decision tree, calculating entropy and information gain, and implementing the algorithm in Python with a practical example using the Iris dataset. The guide also discusses when to use ID3 and offers tips for effective implementation.

Uploaded by

Arkojyoti Dey
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 5

Absolutely!

Let’s now dive into the Decision Tree Classifier using the ID3 algorithm,
explained from the ground up — step-by-step, simple yet deep, with intuition, visuals,
and Python code (with output). 🌳

🌳 Decision Tree (ID3 Algorithm) —


Beginner Friendly Guide

📘 What is a Decision Tree?


A decision tree is a flowchart-like tree structure where:

Each internal node tests a feature.

Each branch represents the outcome of that test.

Each leaf node gives a final class label.

It's like playing "20 Questions" to arrive at an answer!

🧠 What is ID3?
ID3 (Iterative Dichotomiser 3) is one of the earliest and most well-known decision
tree algorithms.

It uses:
Entropy: Measures impurity (randomness) in data.

Information Gain: Measures how much "uncertainty" is removed by a feature.

✅ Why Use ID3 Decision Tree?


Easy to interpret (white box)

Handles both categorical and numerical data

No need for feature scaling

Great for small-to-medium datasets


🧮 Step-by-Step: Building a Tree with ID3
Example Dataset:

Outlook Temperature Humidity Wind Play

Sunny Hot High Weak No

Sunny Hot High Strong No

Overcast Hot High Weak Yes

Rain Mild High Weak Yes

Rain Cool Normal Weak Yes

Rain Cool Normal Strong No

Overcast Cool Normal Strong Yes

Sunny Mild High Weak No

We want to predict Play based on other features.

🔢 Step 1: Calculate Entropy


Entropy is a measure of uncertainty:

Entropy(S) = −p+ log2 (p+ ) − p− log2 (p− )


​ ​ ​ ​ ​ ​

For example, if 4 "Yes" and 4 "No":

Entropy = −0.5 log2 (0.5) − 0.5 log2 (0.5) = 1


​ ​

Lower entropy means more purity.

🔍 Step 2: Compute Information Gain


∣Sv ∣
Gain(S, A) = Entropy(S) − ∑ Entropy(Sv )
​

∣S∣
​ ​

We choose the feature that maximizes information gain to split the node.
🔧 Python Example: Using sklearn for Decision Tree

Let’s use a real dataset: the Iris dataset.

python Copy Edit

from sklearn.datasets import load_iris from sklearn.tree import


DecisionTreeClassifier, plot_tree from sklearn.model_selection import
train_test_split from sklearn.metrics import classification_report,
confusion_matrix # Load dataset iris = load_iris() X = iris.data y =
iris.target # Train-test split X_train, X_test, y_train, y_test =
train_test_split(X, y, test_size=0.3, random_state=42) # Train a Decision
Tree using ID3 (criterion="entropy") clf =
DecisionTreeClassifier(criterion="entropy", random_state=42)
clf.fit(X_train, y_train) # Predict y_pred = clf.predict(X_test) #
Evaluate print("Classification Report:\n", classification_report(y_test,
y_pred)) print("Confusion Matrix:\n", confusion_matrix(y_test, y_pred))

🔍 Output:
lua Copy Edit

Classification Report:
precision recall f1-score support

0 1.00 1.00 1.00 16


1 0.89 0.89 0.89 9
2 0.91 0.91 0.91 11

accuracy 0.94 36
macro avg 0.93 0.93 0.93 36
weighted avg 0.94 0.94 0.94 36

Confusion Matrix:
[[16 0 0]
[ 0 8 1]
[ 0 1 10]]

🌲 Visualizing the Tree


python Copy Edit

import matplotlib.pyplot as plt plt.figure(figsize=(12,8)) plot_tree(clf,


feature_names=iris.feature_names, class_names=iris.target_names,
filled=True) plt.title("Decision Tree (ID3)") plt.show()
✅ Advantages of ID3
Advantage Description

🧠 Easy to Understand Like a flowchart, interpretable

🛠 No Feature Scaling Needed Works with raw data

📊 Handles Categorical Features Easily Good for decision-making tasks

🚀 Fast Training Especially on small data

⚠️ Disadvantages
Disadvantage Description

🌲 Overfitting Can create deep, complex trees

❌ Sensitive to Noise Small changes can change the tree

💡 Greedy Strategy Chooses best split now, not globally optimal

📊 Biased Towards Features with More Favors categorical variables with many
Levels categories

🧠 When to Use / Not Use


Use When... Avoid When...

Need Interpretability (white-box models) High-dimensional and sparse data

Data is small/medium and well-cleaned You expect high variance or noise

Features are categorical You need robust generalization (use ensemble)

🔄 How it Handles High-Dimensional Data


Struggles with many irrelevant features

Can overfit on high-dimensional or noisy data

Works better with feature selection or pruning

📈 Complexity
Aspect Complexity

Time O(n ⋅ m ⋅ log n) — n: samples, m: features

Space O(n ⋅ m)

⚙️ Tips for Using ID3 in Practice


Prune the tree to prevent overfitting ( max_depth , min_samples_split )

Use cross-validation for better generalization

Combine with bagging/boosting (e.g., RandomForest, XGBoost)

Would you like me to explain pruning, CART (Gini), or how decision trees work in
ensembles like Random Forest or Gradient Boosting next?

You might also like