MACHINE LEARNING
PROJECT ON DECISION TREES
Project Name :
Name : Suraj Gopal Aher
Roll No : 23/IT/164
Abstract : Decision Trees are one of the most intuitive and widely used algorithms in
Machine Learning for both classification and regression tasks. This project explains the
working of Decision Trees, their applications, and how they help in making accurate and
interpretable decisions based on data. The study involves building a simple Decision Tree
classifier and analyzing its performance on a dataset.
Introduction
Decision Trees are a supervised learning technique used for both classification and regression
problems. They work by splitting the dataset into subsets based on the value of input features.
This structure resembles a tree, where internal nodes represent a test on a feature, branches
represent the outcome, and leaves represent the final decision.
Study Objective
• To understand the concept and working of Decision Trees.
• To build and evaluate a Decision Tree model using Python and Scikit-learn.
• To explore real-world applications and future trends.
Scope & Limitations
Scope:
• Implementation on publicly available datasets.
• Use of Scikit-learn for building and visualizing Decision Trees.
Limitations:
• Limited to small datasets.
• May not generalize well on complex, large-scale problems due to overfitting.
Real-World Application
Decision Trees are used in:
• Medical Diagnosis: To predict diseases based on symptoms.
• Credit Risk Analysis: To evaluate loan applicants.
• Customer Segmentation: To classify customers based on purchasing behavior.
• Fraud Detection: To detect unusual transactions.
• Recommendation Systems: For product recommendations.
Challenges and Future Trends
Challenges:
• Prone to overfitting on training data.
• Can become unstable with small variations in data.
• Not ideal for continuous or high-dimensional data.
Future Trends:
• Use of Ensemble Methods like Random Forests and Gradient Boosting to improve
accuracy.
• Integration with Explainable AI (XAI) for better interpretability.
• Application in areas like autonomous vehicles and personalized medicine.
Conclusion:
Decision Trees are a powerful and easy-to-understand model in Machine Learning. Despite
their limitations, they serve as a fundamental building block for advanced ensemble methods.
With proper tuning and modern techniques, Decision Trees continue to have a significant role
in real-world applications.
References
1. Scikit-learn documentation: https://scikit-learn.org/
2. “Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow” by Aurélien
Géron.
3. Kaggle Datasets: https://www.kaggle.com/
4. Research papers and articles on Decision Trees.
CODE:
# Credit Card Fraud Detection using Decision Trees
# This program builds a Decision Tree classifier to detect credit card fraud
# using the Kaggle Credit Card Fraud Detection dataset.
# Dataset: https://www.kaggle.com/mlg-ulb/creditcardfraud
# Import necessary libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier, plot_tree
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
from sklearn.metrics import confusion_matrix, classification_report
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings('ignore')
# Set random seed for reproducibility
np.random.seed(42)
# Load the dataset
print("Loading the Credit Card Fraud dataset...")
try:
df = pd.read_csv('creditcard.csv')
print("Dataset successfully loaded.")
except FileNotFoundError:
print("Dataset file not found. Please download the dataset from Kaggle and place it in the
current directory.")
print("Download link: https://www.kaggle.com/mlg-ulb/creditcardfraud")
exit()
# Display basic information about the dataset
print("\n=== Dataset Information ===")
print(f"Shape of the dataset: {df.shape}")
print("\nFirst 5 rows of the dataset:")
print(df.head())
print("\nSummary statistics:")
print(df.describe())
# Check for missing values
print("\nMissing values in each column:")
print(df.isnull().sum())
# Display class distribution (fraud vs non-fraud)
print("\n=== Class Distribution ===")
class_counts = df['Class'].value_counts()
print(class_counts)
print(f"Percentage of fraud transactions: {class_counts[1] / len(df) * 100:.4f}%")
# Visualize class distribution
plt.figure(figsize=(10, 6))
sns.countplot(x='Class', data=df)
plt.title('Class Distribution (0: Normal, 1: Fraud)')
plt.ylabel('Count')
plt.yscale('log') # Using log scale for better visualization due to class imbalance
plt.savefig('class_distribution.png')
plt.close()
# Split the data into features and target
X = df.drop('Class', axis=1)
y = df['Class']
# Split the data into training and testing sets (80% training, 20% testing)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42,
stratify=y)
print("\n=== Dataset Split ===")
print(f"Training set size: {X_train.shape[0]} samples")
print(f"Testing set size: {X_test.shape[0]} samples")
# Create and train a Decision Tree Classifier
# Using class_weight='balanced' to handle imbalanced dataset
print("\n=== Training Decision Tree Classifier ===")
dt_classifier = DecisionTreeClassifier(
max_depth=10, # Limit depth to prevent overfitting
min_samples_split=20, # Minimum samples required to split an internal node
min_samples_leaf=5, # Minimum samples required at a leaf node
class_weight='balanced', # Handle class imbalance
random_state=42
# Fit the model to the training data
dt_classifier.fit(X_train, y_train)
print("Decision Tree training completed.")
# Make predictions on the test set
y_pred = dt_classifier.predict(X_test)
# Evaluate the model
print("\n=== Model Evaluation ===")
# Calculate evaluation metrics
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.4f}")
print(f"Precision: {precision:.4f}")
print(f"Recall: {recall:.4f}")
print(f"F1 Score: {f1:.4f}")
# Generate and display confusion matrix
cm = confusion_matrix(y_test, y_pred)
print("\nConfusion Matrix:")
print(cm)
# Display classification report
print("\nClassification Report:")
print(classification_report(y_test, y_pred))
# Visualize the confusion matrix
plt.figure(figsize=(8, 6))
sns.heatmap(cm, annot=True, fmt='d', cmap='Blues',
xticklabels=['Normal (0)', 'Fraud (1)'],
yticklabels=['Normal (0)', 'Fraud (1)'])
plt.xlabel('Predicted Label')
plt.ylabel('True Label')
plt.title('Confusion Matrix')
plt.savefig('confusion_matrix.png')
plt.close()
# Find the most important features based on the Decision Tree
feature_importance = pd.DataFrame({
'Feature': X.columns,
'Importance': dt_classifier.feature_importances_
})
feature_importance = feature_importance.sort_values('Importance', ascending=False)
print("\n=== Feature Importance ===")
print(feature_importance.head(10)) # Top 10 most important features
# Visualize feature importance
plt.figure(figsize=(12, 8))
sns.barplot(x='Importance', y='Feature', data=feature_importance.head(10))
plt.title('Top 10 Feature Importance')
plt.tight_layout()
plt.savefig('feature_importance.png')
plt.close()
# Visualize the Decision Tree (limited to max_depth=3 for clarity)
plt.figure(figsize=(20, 10))
plot_tree(
dt_classifier,
max_depth=3, # Limit visualization depth for clarity
feature_names=X.columns,
class_names=['Normal', 'Fraud'],
filled=True,
rounded=True,
fontsize=10
)
plt.title('Decision Tree Visualization (Limited to Depth 3)')
plt.savefig('decision_tree.png')
plt.close()
print("\n=== Decision Tree Interpretation ===")
print("""
The decision tree diagram shows how the model makes decisions to classify transactions as
fraudulent or normal:
1. Each node represents a decision based on a specific feature.
2. The color intensity indicates the class distribution at that node (darker = more
homogeneous).
3. The tree branches out based on feature thresholds.
4. The deeper the tree, the more complex the decision-making process.
5. Leaf nodes represent the final classification decisions.
The important features identified by the model (shown in the feature importance plot)
are the key indicators the algorithm uses to detect fraudulent transactions.
Time and transaction amount, along with certain V-features (which are PCA-transformed
features for confidentiality), play significant roles in fraud detection.
""")
print("\n=== Analysis Complete ===")
print("Decision Tree model for Credit Card Fraud Detection has been successfully built and
evaluated.")
print("Visualization files have been saved: class_distribution.png, confusion_matrix.png,
feature_importance.png, and decision_tree.png")
OUTPUT:
