How to Find Duplicates Values Across Multiple Columns in SQL?

In SQL, identifying duplicate entries across multiple columns is crucial for ensuring data integrity and quality. Whether we're working with large datasets or trying to clean up existing data, we can efficiently find duplicates using GROUP BY and COUNT() functions.

In this article, we'll focus on how to find duplicate entries across multiple columns in SQL using a practical example. This is particularly helpful when working with datasets that involve multiple attributes, like officer assignments in different locations, which can have duplicate values in more than one column.

Examples of Finding Duplicates Across Multiple Columns

We’ll use a sample table named POSTINGS inside the GeeksForGeeks database. This table tracks postings of officers in different countries, with columns for posting ID, officer name, team size, and location.

Query:

CREATE TABLE POSTINGS(
POSTING_ID INT,
OFFICER_NAME VARCHAR(10),
TEAM_SIZE INT,
POSTING_LOCATION VARCHAR(10));

INSERT INTO POSTINGS VALUES(1,'RYAN',10,'GERMANY');
INSERT INTO POSTINGS VALUES(2,'JACK',6,'ROMANIA');
INSERT INTO POSTINGS VALUES(3,'JANE',4,'HAWAII');
INSERT INTO POSTINGS VALUES(4,'JIM',10,'GERMANY');
INSERT INTO POSTINGS VALUES(5,'TIM',10,'GERMANY');
INSERT INTO POSTINGS VALUES(6,'RYAN',11,'GERMANY');
INSERT INTO POSTINGS VALUES(7,'RYAN',10,'GERMANY');
INSERT INTO POSTINGS VALUES(8,'RYAN',10,'GERMANY');
INSERT INTO POSTINGS VALUES(9,'JACK',6,'CUBA');
INSERT INTO POSTINGS VALUES(10,'JACK',6,'HAITI');

SELECT * FROM POSTINGS;

Output

1. Duplicates in 3 Columns: OFFICER_NAME, TEAM_SIZE, and POSTING_LOCATION

To find duplicates in these three columns, group the rows by OFFICER_NAME, TEAM_SIZE, and POSTING_LOCATION, we can group the data by these columns and filter the results with HAVING clause, using the COUNT() function.

Query:

SELECT OFFICER_NAME, TEAM_SIZE,
POSTING_LOCATION, COUNT(*) AS QTY
FROM POSTINGS
GROUP BY OFFICER_NAME, TEAM_SIZE,
POSTING_LOCATION HAVING COUNT(*)>1;

Output

Explanation:

The output shows that RYAN has been assigned to GERMANY with a team size of 10 on three different occasions, indicating duplicate records.

2. Duplicates in 2 Columns: TEAM_SIZE and POSTING_LOCATION

In this example, we want to find duplicates based only on TEAM_SIZE and POSTING_LOCATION, ignoring the officer's name. This helps to identify if multiple officers were assigned to the same team size and location.

Query:

SELECT TEAM_SIZE, POSTING_LOCATION, 
COUNT(*) AS QTY
FROM POSTINGS
GROUP BY TEAM_SIZE, POSTING_LOCATION
 HAVING COUNT(*)>1;

Output

Explanation:

The output shows that the team size of 10 is assigned to GERMANY four times, indicating duplicate assignments across different officers.

3. Duplicates in 2 Columns: OFFICER_NAME and TEAM_SIZE

In this scenario, we focus on finding duplicates where the same officer has been assigned to the same team size multiple times, possibly in different locations.

Query:

SELECT OFFICER_NAME, TEAM_SIZE, 
COUNT(*) AS QTY
FROM POSTINGS
GROUP BY OFFICER_NAME, 
TEAM_SIZE HAVING COUNT(*)>1;

Output

Explanation:

The result shows that RYAN has been assigned to a team size of 10 three times, confirming a duplicate entry for the officer and team size.

Conclusion

Finding duplicates across multiple columns in SQL is essential for maintaining clean, accurate, and consistent data. By using the GROUP BY and HAVING clauses with COUNT(), SQL allows us to easily identify and filter duplicate entries based on one or more columns. These techniques are highly valuable for database cleanup, reporting, and ensuring data integrity. With the ability to search for duplicates in different combinations of columns, we can quickly analyze our data and pinpoint redundancies.