Sample Phase 4
Sample Phase 4
Overview of Results
In this phase, we summarize the performance of our models used for fraud detection. The
results section showcases key metrics and visualizations, providing a comprehensive
comparison between the models. Furthermore, we discuss the deployment of our AutoAI
model on IBM Cloud and the development of a user-friendly dashboard for monitoring
flagged transactions.
The following table summarizes the evaluation metrics for the models:
2.2 Visualizations
[You have to add all the important visualizations performed, I have inserted
only few in this sample]
● ROC Curve: Displays the trade-off between true positive and false positive rates
across all models.
● Confusion Matrix: Highlights model predictions, differentiating between true
positives, true negatives, false positives, and false negatives.
ROC Curve
Confusion Matrix
A high number of TP (51,411) reflects effective fraud detection. The low FP
(886) minimizes inconvenience to legitimate customers. However, the
moderate FN (4,949) indicates some fraud cases are missed, suggesting a
trade-off. Adjusting the classification threshold can reduce FN at the cost of
slightly increasing FP.
Insights
The AutoAI model outperformed other models with an accuracy of 97.1% and a balanced
precision-recall trade-off, making it the best choice for deployment. This trade-off is
especially critical in fraud detection because identifying fraudulent transactions (recall) often
comes at the expense of mistakenly flagging legitimate ones (precision). The Confusion
Matrix highlights strong performance with high True Positives (51,411), reflecting effective
fraud detection, and low False Positives (886), minimizing customer inconvenience.
Meanwhile, the ROC-AUC Curve demonstrates robust model performance with an AUC of
0.97, indicating excellent discrimination between fraud and non-fraud cases. Striking the
right balance minimizes financial losses and customer dissatisfaction while maximizing
detection efficiency.
<script>
async function fetchLiveTransactions() {
try {
// Prepare the transactions data (JSON)
let transactions = {
input_data: [
{
fields: [
"Time", "V1", "V2", "V3", "V4", "V5", "V6",
"V7",
"V8", "V9", "V10", "V11", "V12", "V13",
"V14", "V15",
"V16", "V17", "V18", "V19", "V20", "V21",
"V22", "V23",
"V24", "V25", "V26", "V27", "V28", "Amount"
],
values: [
[
1, -1.359807134, 1.191857111,
-1.358354062, -0.877680402,
1.537870779, -0.420986105,
-0.720762887, 0.697211109,
-2.064945287, -5.587793782,
2.115795177, -5.417424082,
-1.235122631, -6.665176895,
0.401700687, -2.897825117,
-4.570529434, -1.315147214,
0.391167041, 1.252966735,
0.778583979, -0.319188819, 0.639418961,
-0.29488504,
0.537502536, 0.788395057, 0.292679966,
0.147967929, 390
]
]
}
]
};
tableBody.innerHTML = '';
flaggedList.innerHTML = '';
// Iterate over transactions and update with predictions
transactions.input_data[0].values.forEach((transaction,
index) => {
const status = predictions[index]?.status ||
'Legitimate';
if (flaggedList.children.length === 0) {
flaggedList.innerHTML = '<li class="list-group-item">No
flagged transactions yet</li>';
}
} catch (error) {
console.error('Error fetching or processing transactions:',
error);
}
}
// Quick Actions
function flagAllFraudulent() {
alert('Flagging all fraudulent transactions!');
}
function notifyAdmins() {
alert('Admins have been notified!');
}
function exportReport() {
alert('Exporting report...');
}
function flagTransaction(transactionId) {
alert(`Transaction ${transactionId} flagged for review!`);
}
// Initialize dashboard
function initDashboard() {
fetchLiveTransactions();
fetchMetricScore();
createFraudChart();
setInterval(fetchLiveTransactions, 500000); // Refresh
every 5 seconds
setInterval(fetchMetricScore, 1000000); // Refresh metric
score every 10 seconds
}
initDashboard();
</script>
</body>
</html>
4. Deploying AutoAI Model on IBM Cloud
The AutoAI model was deployed on IBM Cloud using the following steps:
return response.data.access_token;
} catch (error) {
console.error('Error getting auth token:', error);
throw new Error('Failed to retrieve token');
}
}
'https://eu-de.ml.cloud.ibm.com/ml/v4/deployments/testing1/predictions?
version=2021-05-01',
payload,
{
headers: {
'Authorization': `Bearer ${token}`,
'Content-Type': 'application/json',
},
}
);
return response.data.predictions;
} catch (error) {
console.error('Error during prediction:', error);
throw new Error('Failed to make prediction');
}
}
res.json({
predictions,
});
} catch (error) {
res.status(500).json({
error: 'Error processing prediction',
});
}
});
The deployment and operation of the AutoAI model in Watson Studio involved the following
resource usage:
● Compute Hours (CUH): Approximately 14 CUH were utilized across various tasks,
including:
○ Preprocessing data using Jupyter Notebook.
○ Model creation and training through AutoAI.
○ Testing and deployment for evaluation.
● API Requests: Watson Machine Learning on the Lite plan supports up to 50
deployment requests per month. These requests were used during deployment
testing and user interactions.
● Storage: 1 GB allocated for storing the dataset and trained model.
This setup operates within the free tier limits of IBM services, which supports low to
moderate workloads without incurring additional costs.
● Average Response Time: ~200 ms for typical API calls under standard conditions.
● Peak Response Time: ~500 ms during periods of high traffic or resource contention.
These response times indicate that the model is efficient in handling predictions, even during
peak demand, making it suitable for tasks like fraud detection.
The free tier of IBM Cloud services offers limited resources, which may restrict scalability for
larger-scale deployments:
● Compute Hours: The free tier provides 20 CUH/month, leaving limited overhead for
additional experimentation or scaling.
● API Requests: Limited to 50 deployment requests/month, suitable for small-scale
testing but insufficient for high-traffic applications.
● Storage: 1 GB storage is sufficient for small datasets and models but may require
upgrades for larger or more complex projects.
Note: Avoid committing sensitive information (e.g., API keys, passwords). Use a
.gitignore file to exclude such files and manage secrets with environment variables.
7. Future scope
To further enhance the project and expand its scope, the following steps are proposed:
1. Deploying the UI on Vercel: Hosting the user interface on Vercel will provide a
user-friendly and accessible platform to interact with the deployed model on IBM
Cloud.
2. Feature to Upload CSV Files: Adding functionality to upload and analyze custom
CSV datasets will increase the flexibility and practicality of the system for various use
cases.
3. Connecting to a Database: Integrating a database will enable the persistent storage
of transaction records, anomaly scores, and user data, facilitating historical analysis
and scalability.
4. Automated Report Generation: Implementing a feature to generate detailed reports
summarizing detected anomalies and system performance metrics will improve
transparency and usability for stakeholders.
8. Conclusion
This phase successfully integrated advanced machine learning models with a highly intuitive
and user-friendly interface, while also ensuring smooth deployment on IBM Cloud. Through
a series of meticulous steps, the project achieved the critical objectives of providing accurate
fraud detection, facilitating seamless user interaction, and ensuring that the system can
scale effectively in real-world applications. The integration process not only focused on
enhancing the efficiency and reliability of fraud detection algorithms but also prioritized user
experience, making the interface accessible and easy to navigate for both technical and
non-technical users. By leveraging the powerful capabilities of IBM Cloud, the deployment is
both secure and scalable, capable of handling increased demand and adapting to future
advancements in technology. This approach guarantees that the solution will remain
effective and adaptable in dynamic, high-stakes environments where real-time fraud
detection and swift response times are crucial.