Scribd
Upload
6 views4 pages

ML Project

This study compares the noise resilience of decision trees and linear regression in machine learning, hypothesizing that decision trees are more robust to noise in input data. Experiments using the IPARC dataset showed that decision trees maintained higher accuracy and lower error rates compared to linear regression when noise was introduced. The findings suggest that decision trees are better suited for applications involving noisy data, with potential for future research into other models and preprocessing techniques.

Uploaded by

Shiv Patel
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
6 views4 pages

ML Project

This study compares the noise resilience of decision trees and linear regression in machine learning, hypothesizing that decision trees are more robust to noise in input data. Experiments using the IPARC dataset showed that decision trees maintained higher accuracy and lower error rates compared to linear regression when noise was introduced. The findings suggest that decision trees are better suited for applications involving noisy data, with potential for future research into other models and preprocessing techniques.

Uploaded by

Shiv Patel
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

Under review as submission to TMLR

Noise Resilience in Machine Learning Models: A Comparison


of Decision Trees and Linear Regression

Anonymous authors
Paper under double-blind review

Abstract

This study investigates how slight randomness (noise) in input data affects the accuracy of
machine learning models in recognizing patterns. It tests the hypothesis that decision trees,
which utilize rule-based splitting, are more resilient to noise compared to linear regression,
which assumes continuous relationships. By comparing accuracy drops in both models after
introducing noise, the research evaluates their robustness under imperfect data conditions,
providing insights into their suitability for real-world applications. Experiments were con-
ducted using a Python-based framework applied to the IPARC dataset. Models were tested
across various noise levels, and metrics such as MSE, MAE, and R2 were used to quantify
performance.

1 Introduction

Noise in data is a common challenge in machine learning applications, often stemming from measurement
errors or data preprocessing inconsistencies. Understanding how noise affects model performance is critical for
selecting the most appropriate algorithms for real-world scenarios. Decision trees (DTs) and linear regression
(LR) are two popular models with distinct mechanisms: DTs split data based on discrete thresholds, while
LR assumes continuous linear relationships between input and output variables. This study aims to quantify
the resilience of these models to noise in the context of the IPARC dataset.

2 Hypothesis

Introducing slight noise to inputs in the IPARC dataset will reduce the accuracy of ML models trained to
recognize patterns, but decision trees will demonstrate greater resilience to this noise compared to linear
regression.

3 Experimental Setup

The experiments were conducted using the Python-based IPARCExperiment framework, which implements
the following components:

3.1 1. Dataset and Preprocessing

• Dataset: The experiments focused on the "Simple" category of the IPARC dataset. Tasks involve
recognizing patterns in normalized pixel values of images.

• Preprocessing: Each input and output image was normalized to the [0, 1] range. Input images
were flattened for compatibility with regression models.

• Splitting: The dataset was split into training and testing sets using 5-fold cross-validation to ensure
robust evaluation.

1
Under review as submission to TMLR

3.2 2. Noise Levels

The study introduced Gaussian noise to the test data at the following levels: 5%, 10%, 20%, and 30%. Noise
was added as a percentage of the range of pixel values, ensuring the resulting values remained within the
normalized range.

3.3 3. Models
• Decision Tree: Implemented using DecisionTreeRegressor with a maximum depth of 10 to
prevent overfitting.
• Linear Regression: Implemented using LinearRegression, a standard least-squares regression
method.

3.4 4. Evaluation Metrics


• Mean Squared Error (MSE): Measures average squared differences between actual and predicted
values.
• Mean Absolute Error (MAE): Captures the average magnitude of errors.
• R2 : Indicates the proportion of variance in the dependent variable explained by the model.

3.5 5. Framework Implementation

The experiments were conducted using a custom Python class IPARCExperiment. The framework’s key fea-
tures include data normalization, noise generation, cross-validation, and detailed performance visualization.

4 Results and Analysis

The results confirm the hypothesis by demonstrating that noise reduces model accuracy and that decision
trees are more resilient to noise compared to linear regression.

4.1 1. Impact of Noise on Accuracy


• Decision Tree: MSE increased modestly with noise, indicating better noise tolerance.
– MSE rose from 0.1069 (5% noise) to 0.1240 (30% noise).
• Linear Regression: MSE increased significantly, highlighting sensitivity to noise.
– MSE rose from 0.1846 (5% noise) to 0.1938 (30% noise).

4.2 2. Resilience of Decision Trees


• At all noise levels, decision trees outperformed linear regression:
– At 5% noise, decision tree MSE was 42% lower than linear regression MSE.
– At 30% noise, decision tree MSE was 36% lower than linear regression MSE.
• Decision trees exhibited lower variance across folds, further supporting their robustness.

4.3 3. Significance of Metrics

The metrics highlight the strengths of decision trees in handling noisy data:

• MAE: Decision trees consistently showed lower MAE, reflecting smaller average errors.
• R2 : Decision trees maintained higher R2 values, indicating better explanatory power under noise.

2
Under review as submission to TMLR

5 Visualization

Figure 1 presents the performance comparison between decision trees and linear regression across noise levels.

results_plot.png

Figure 1: Model performance (MSE) vs noise levels with error bars representing standard deviation across
folds.

6 Insights from Related Research

Research on crisp and fuzzy decision trees provides additional insights:

• Attribute Noise: Decision trees are robust to small observation errors (o-noise) due to their
discrete splits.

• Class Label Noise: Noise in class labels significantly impacts tree complexity and accuracy, ne-
cessitating clean training data.

• Fuzzy Decision Trees: Studies show that fuzzy decision trees outperform crisp decision trees in
handling ambiguous data, suggesting a potential avenue for future work.

3
Under review as submission to TMLR

7 Conclusion and Future Work


• The results validate the hypothesis:
– Noise reduces accuracy for both models, but decision trees are significantly more resilient.
– Decision trees are suitable for noisy pattern recognition tasks like those in the IPARC dataset.
• Future Work:
– Evaluate other models, such as random forests and fuzzy decision trees, under similar noise
conditions.
– Explore the effects of higher noise levels and real-world noisy datasets.
– Investigate the impact of data augmentation and other preprocessing techniques on model re-
silience.

References
[1] J. Sun and X.-Z. Wang, An Initial Comparison on Noise Resisting Between Crisp and Fuzzy Decision
Trees, Proceedings of the Fourth International Conference on Machine Learning and Cybernetics, pp.
2545–2550, 2005.

[2] François Chollet, On the Measure of Intelligence, CoRR abs/1911.01547, 2019. Available at https:
//arxiv.org/abs/1911.01547.

You might also like