Data Analytics 2024 Data Cleaning

How To
Prepare
Messy Data
for Analysis
using SQL?

How to clean data using SQL? Swipe next

Data Analytics 2024 Data Cleaning

You work as a data analyst for a retail company that collects

daily sales data from various stores.

Let’s walk through the process of cleaning a dataset

using SQL with a real-world example.

Below is a sample of the SalesTransactions table:

TransactionID CustomerID TransactionDate ProductID QuantitySold TotalAmount

1001 2001 12/09/2024 P101 5 500

1002 2001 12/09/2024 P102 NULL 1500

1003 2002 11/09/2024 P103 3 450

1003 2002 11/09/2024 P103 3 450

1004 2003 10/09/2024 P104 10000 100000

1005 2004 09/09/2024 XYZ 2 300

Issues in the Data:

There are several issues in this table

Missing values (NULL in QuantitySold)
Duplicate transactions (same TransactionID repeated)
Inconsistent date format (dd/mm/yyyy instead of yyyy-mm-dd)
Outliers in QuantitySold and TotalAmount (e.g., 10,000 units
sold)
Invalid ProductID (e.g., XYZ is not a valid product).
Data Analytics 2024 Data Cleaning

Let’s clean this data using SQL.

Handling Missing Values

Problem :

The QuantitySold column has a missing value (NULL) for

ProductID = P102. We can estimate this missing value based on

the TotalAmount and the product's price.

First, identify rows where QuantitySold is missing (NULL).

SELECT *

FROM SalesTransactions

WHERE QuantitySold IS NULL;

If a price for the product is available in a Products table, calculate

the missing QuantitySold by dividing TotalAmount by the price.

UPDATE SalesTransactions

SET QuantitySold = TotalAmount / (

SELECT PricePerUnit

FROM Products

WHERE Products.ProductID = .ProductID

SalesTransactions

WHERE QuantitySold IS NULL;

Data Analytics 2024 Data Cleaning

If you can’t calculate QuantitySold, you may choose to remove

the row:

DELETE FROM SalesTransactions

WHERE QuantitySold IS NULL;

After Handling Missing Values

We used SQL to calculate the missing QuantitySold by dividing

the TotalAmount by the product's price from another table

(Products).

The missing value in QuantitySold is now replaced by the correct

value 10.

TransactionID CustomerID TransactionDate ProductID QuantitySold TotalAmount

1001 2001 12/09/2024 P101 5 500

1002 2001 12/09/2024 P102 10 1500

Removing Duplicate Records

Problem :

There’s a duplicate record for TransactionID = 1003, where the

exact same transaction is repeated twice.

Data Analytics 2024 Data Cleaning

Duplicate records can lead to inaccurate analysis. We can remove

duplicates by using the ROW_NUMBER() function.

WITH DuplicateCheck AS (

SELECT *, ROW_NUMBER() OVER

(PARTITION BY TransactionID, CustomerID, TransactionDate

ORDER BY TransactionID) AS RowNum

FROM SalesTransactions

DELETE FROM SalesTransactions

WHERE TransactionID IN (

SELECT TransactionID FROM DuplicateCheck WHERE RowNum > 1

);

After Removing Duplicate Records

The duplicate row has been deleted, leaving only one record

for TransactionID = 1003.

TransactionID CustomerID TransactionDate ProductID QuantitySold TotalAmount

1001 2001 12/09/2024 P101 5 500

1002 2001 12/09/2024 P102 10 1500

1003 2002 11/09/2024 P103 3 450

Data Analytics 2024 Data Cleaning

Standardizing Date Formats

Problem :

The TransactionDate is in different formats like dd/mm/yyyy

(e.g., 12/09/2024), which makes it inconsistent for analysis.

Now, let’s convert TransactionDate from the dd/mm/yyyy format

to yyyy-mm-dd for consistency.

UPDATE SalesTransactions

SET TransactionDate = STR_TO_DATE(TransactionDate, '%d/%m/%Y')

WHERE TransactionDate LIKE '%/%';

After Standardizing Date Formats

All transaction dates are now in the same format: 2024-09-12,

making it easier for queries, comparisons, and reporting.

TransactionID CustomerID TransactionDate ProductID QuantitySold TotalAmount

1001 2001 12-09-2024 P101 5 500

1002 2001 12-09-2024 P102 10 5

1 00

1003 2002 11-09-2024 P103 3 5

4 0
Data Analytics 2024 Data C g
leanin

Identify & Handle Outliers

Problem :

There’s an abnormally high QuantitySold of 10,000 units for

TransactionID = 1004, which is likely an error or an extreme outlier.

We can flag or delete such extreme values.

Identify outliers in QuantitySold

SELECT *

FROM SalesTransactions

WHERE QuantitySold > 1000;

Remove the outlier

DELETE FROM SalesTransactions

WHERE QuantitySold > 1000;

After Identify & Handle Outliers

We flagged this as an outlier using a simple rule (e.g., quantities greater

than 1,000), and then removed this transaction from the data.

TransactionID C
ustomerID TransactionDate ProductID Q S
uantity old TotalAmount

1004 3
200 Deleted
Data Analytics 2024 Data Cleaning

Validating Product IDs

Problem :

The ProductID for TransactionID = 1005 is XYZ, which is invalid

because there is no such product in the Products table.

Identify invalid ProductIDs

SELECT *

FROM SalesTransactions

WHERE ProductID NOT IN (SELECT ProductID FROM Products);

Remove rows with invalid ProductIDs

DELETE FROM SalesTransactions

WHERE ProductID NOT IN (SELECT ProductID FROM Products);

After Identify & Handle Outliers

The row with the invalid ProductID = XYZ has been deleted, leaving

only valid product entries.

TransactionID CustomerID TransactionDate ProductID QuantitySold TotalAmount

1005 2004 Deleted

Data Analytics 2024 Data Cleaning

Final Cleaned Data

After performing these steps, your cleaned SalesTransactions

table would look something like this:

TransactionID CustomerID TransactionDate ProductID QuantitySold TotalAmount

1001 2001 12-09-2024 P101 5 500

1002 2001 12-09-2024 P102 10 1500

1003 2002 11-09-2024 P103 3 450

Summary

Handled missing values by filling or removing rows with NULL

values

Removed duplicates to ensure each transaction is unique

Standardized date formats for consistent querying

Handled outliers in QuantitySold and TotalAmount to ensure

realistic data

Validated ProductIDs by removing invalid entries.

By applying these SQL techniques, you ensure the dataset is clean,

accurate, and ready for analysis or reporting.

Data Analytics 2024 Data Cleaning

Dr. Jayen Thakker MetricMinds.in

Was This Helpful?

Be sure to save it so you can
come back to it later

Follow us for more.