Module code: UFCFMJ-15-M

Module Title: Machine Learning and Predictive Analytics

Student Number: 23064673

Introduction

Credit card fraud has emerged as a significant challenge for financial institutions worldwide,
resulting in substantial financial losses and erosion of customer trust. Detecting fraudulent
transactions promptly and accurately is crucial for safeguarding both the bank and its
customers. To mitigate these risks, robust fraud detection systems are imperative. This report
presents a comprehensive analysis of my machine learning model applied to a dataset of
credit card transactions to identify fraudulent activities.

The dataset employed in this study encompasses credit card transactions conducted by
European cardholders over a two-day period in September 2013. A critical challenge posed
by this dataset is the significant class imbalance, with fraudulent transactions constituting a
mere 0.172% of the total observations. This imbalance necessitates the application of
specialized techniques to ensure model effectiveness.

The primary objective of this study is to develop a high-performing model capable of

accurately distinguishing between legitimate and fraudulent transactions while effectively
addressing the class imbalance issue. By minimizing false positives and maximizing the
detection of fraudulent activities, this research aims to contribute to the development of
robust fraud prevention strategies.

The subsequent sections delve into data exploration, preprocessing, model development,
evaluation, and recommendations for implementation.

Dataset Overview

The dataset utilized in this study was retrieved from Kaggle.com “[1]”, a renowned platform
for data science and machine learning resources. It encompasses credit card transactions
conducted by European cardholders over a two-day period in September 2013. Due to
confidentiality constraints, the dataset features numerical features generated through Principal
Component Analysis (PCA). The target variable, 'Class', indicates whether a transaction is
fraudulent (1) or legitimate (0).

A critical challenge associated with this dataset is the severe class imbalance. Fraudulent
transactions represent a mere 0.172% of the total observations. This imbalance poses a
significant obstacle for traditional classification models, as they tend to prioritize the majority
class (legitimate transactions) and underperform in identifying the minority class (fraudulent
transactions). To overcome this challenge, specialized techniques will be employed during
model development and evaluation.
Figure 1: Pie Plot for showing percentatge of normal and fraudulent transactions
Figure 2: Linear Correlation: Feature vs. Fraudulent or Not

Methodology

To address the challenge of fraud detection, a comprehensive methodology was employed.

The process involved data exploration, preprocessing, model development, evaluation, and
performance analysis.

Data Exploration and Preprocessing

The dataset was meticulously explored to understand its characteristics, identify potential
outliers, and visualize data distributions. Essential preprocessing steps included handling
missing values, which were found to be absent in this dataset. To align the scale of features,
the 'Amount' column was standardized. The 'Time' column to was also dropped for potential
temporal patterns in fraudulent transactions.

Model Development and Evaluation

A comparative analysis of machine learning algorithms was undertaken to identify the most
suitable model for fraud detection. Decision Trees, Random Forest, and Neural Networks
were selected based on their strengths in classification tasks and their ability to handle
imbalanced datasets.

Hyperparameter tuning was employed to optimize model performance, exploring different

configurations for each algorithm. The models were trained on the preprocessed dataset and
evaluated using a comprehensive set of metrics, including accuracy, precision, recall, F1-
score, and the area under the Precision-Recall Curve (AUPRC).

To address the significant class imbalance, techniques such as oversampling (SMOTE) and
class weighting were implemented. These methods aimed to improve the model's ability to
detect fraudulent transactions without compromising overall performance.
Figure 3: True and False Positive Comparison

Model Selection and Refinement

The performance of each model was carefully evaluated based on the chosen metrics. The
neural network demonstrated superior performance in terms of accuracy, precision, recall,
and F1-score. Additionally, the neural network effectively addressed the class imbalance
challenge, resulting in a lower false positive rate.

To further enhance the model, techniques like feature importance analysis and cost-sensitive
learning could be explored. These approaches can provide insights into the most influential
features and help to prioritize the detection of high-impact fraudulent transactions.

By following this structured methodology and leveraging advanced machine learning

techniques, a robust fraud detection model was developed, capable of effectively identifying
fraudulent credit card transactions while minimizing false positives.

Performance Metrics

To comprehensively assess model performance, a combination of metrics was employed:

• Accuracy: While not ideal for imbalanced datasets, it provides a baseline measure of
overall correct predictions.
• Precision: Measures the proportion of positive predictions that were truly positive,
essential for minimizing false positives.
 • Recall (Sensitivity): Indicates the ability of the model to correctly identify positive
cases, crucial for fraud detection.
• F1-Score: Harmonizes precision and recall, providing a balanced metric for
imbalanced datasets.
• AUC-ROC: Assesses the model's ability to distinguish between classes across
different classification thresholds.

By analyzing these metrics, we can gain valuable insights into the model's strengths and
weaknesses. For instance, a high recall with a relatively low precision might indicate a model
that is good at identifying fraudulent transactions but also flags a significant number of
legitimate ones as fraudulent.

Confusion Matrix Analysis and Class Imbalance

Confusion Matrix and Error Analysis

The confusion matrix provides a granular view of model performance, breaking down
predictions into true positives, true negatives, false positives, and false negatives. By
analyzing the matrix, we can identify patterns in errors.

• False Positives: Understanding the characteristics of false positives can help refine
model rules or thresholds. For instance, if a particular type of transaction is frequently
misclassified, it might require additional features or adjustments.
• False Negatives: Analyzing false negatives is crucial to identify fraudulent patterns
that the model is missing. This analysis can guide feature engineering or model
refinement.

Figure 4: Random Forest

 Figure 5: Decision Tree

Impact of Class Imbalance

The class imbalance in the dataset significantly influenced model performance. To address
this, techniques like oversampling (SMOTE) and undersampling were employed.

• Oversampling: By creating synthetic minority class instances, SMOTE helped to

balance the dataset. However, it's essential to evaluate if overfitting is a concern.
• Undersampling: Randomly removing instances from the majority class can also
address imbalance. However, it might lead to loss of information.
• Class Weighting: Assigning different weights to classes during model training can
help prioritize the minority class.

By combining these techniques and carefully evaluating their impact on model performance,
we can effectively address the class imbalance challenge.
Figure 6: Test Data Accuracy
Model Comparison: PlainNeuralNetwork, OversamplingNeuralNetwork,
UndersamplingNeuralNetwork, and ClassWeightedNeuralNetwork

To evaluate the effectiveness of different approaches to address class imbalance, we

compared four neural network models:

1. PlainNeuralNetwork: A baseline model trained on the original imbalanced dataset.

Figure 7: Plain Neural Network

 2. ClassWeightedNeuralNetwork: Assigned different weights to classes during training
to address imbalance.

Figure 8: Weighted Loss Test Accuracy

Figure 9: Weighted Neural Network
 3. UndersamplingNeuralNetwork: Undersampled the majority class to balance the
dataset.

Figure 10: Under Sampling Neural Network

 4. OversamplingNeuralNetwork: Employed SMOTE to oversample the minority class.

Figure 11: Over Sampling Neural Network

Performance Evaluation

The performance of these models was assessed using metrics such as accuracy, precision,
recall, F1-score, and AUC-ROC. The confusion matrices provided insights into error
patterns.

The neural network models, particularly the PlainNeuralNetwork and

OverSampledNeuralNetwork, demonstrated exceptional performance in detecting fraudulent
transactions. The OverSampledNeuralNetwork, specifically, achieved a remarkable balance
between precision and recall, indicating its effectiveness in addressing the class imbalance
challenge. This model successfully identified a high percentage of fraudulent cases while
minimizing false positives.

However, it's crucial to note that the OverSampledNeuralNetwork might be susceptible to

overfitting due to the introduction of synthetic data. Further investigation into techniques like
regularization or more sophisticated oversampling methods could be beneficial.

The PlainNeuralNetwork, while achieving a high recall rate, exhibited a higher false positive
rate compared to the OverSampledNeuralNetwork. This trade-off highlights the importance
of carefully considering the specific business requirements and the cost associated with false
positives and false negatives.

Cost-Benefit Analysis

In the context of fraud detection, the costs associated with false positives and false negatives
are not equal. False positives might lead to unnecessary investigations and customer
inconvenience, while false negatives have direct financial implications. A cost-benefit
analysis can help determine the optimal model threshold by considering the financial impact
of different error types.

By carefully analyzing the confusion matrix, understanding the impact of class imbalance,
and considering the financial implications of errors, we can refine the model and make
informed decisions about its deployment.
Conclusion

The neural network model, particularly the OverSampledNeuralNetwork, demonstrated

exceptional proficiency in detecting fraudulent credit card transactions. This model
effectively balanced precision and recall, leading to a significant reduction in both false
positives and false negatives. By accurately identifying fraudulent cases while minimizing
false alarms, this model offers substantial value to financial institutions in safeguarding
against financial losses and protecting customers.

The application of oversampling techniques proved instrumental in addressing the class

imbalance challenge inherent in the dataset. However, further exploration of hybrid
approaches, such as combining oversampling with class weighting or undersampling, could
potentially yield additional performance improvements.

While the neural network model showcased promising results, ongoing monitoring and
adaptation are crucial to maintain its effectiveness in the face of evolving fraud tactics.
Implementing a robust monitoring system to detect concept drift and retraining the model
with updated data are essential steps. Additionally, exploring explainable AI techniques can
enhance transparency and trust in the model's decision-making process.

In conclusion, the developed fraud detection system represents a significant advancement in

combating financial crime. By effectively identifying fraudulent transactions, financial
institutions can mitigate risks, protect customer assets, and maintain a strong reputation.

References:
1. https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud?resource=download