How to Get the Common Index of Two Pandas DataFrames

When working with large datasets in Python Pandas, having multiple DataFrames with overlapping or related data is common. In many cases, we may want to identify the common indices between two DataFrames to perform further analysis, such as merging, filtering, or comparison.

This article will guide us through the process of finding the common index between two Pandas DataFrames using simple and efficient methods.

Finding the common index is helpful when:

• We want to focus on the shared data between two DataFrames.
• We must perform operations like intersection or filtering based on the indices.
• We want to merge DataFrames on their indices but only for common elements.

Let’s explore different methods to achieve this in Python.

Pandas DataFrames with Common Index

In Pandas, the index represents the labels for rows in a DataFrame. It helps to uniquely identify rows, providing more control during merging, filtering, or aligning data across DataFrames. If two DataFrames share some indices, we may want to retrieve those common indices to perform meaningful operations like merging or analyzing similar subsets.

Here, let us first create Panads DataFrames with common index.

Install Pandas

First, let’s install Pandas by writing the following command in the terminal.

pip install pandas

Import Pandas

Next, import the installed Pandas library.

import pandas as pd

Create DataFrames

Create two data frames that contains common index values

df1 = pd.DataFrame({
    'A': [1, 2, 3, 4],
}, index=['a', 'b', 'c', 'd'])

df2 = pd.DataFrame({
    'B': [5, 6, 7, 8],
}, index=['b', 'c', 'd', 'e'])

Now, let’s see the full code execution. Here, df1 has indices ['a', 'b', 'c', 'd'], while df2 has indices ['b', 'c', 'd', 'e']. The common indices between df1 and df2 are ['b', 'c', 'd'].

# import pandas module
import pandas as pd

# Create the first DataFrame
df1 = pd.DataFrame({
    'A': [1, 2, 3, 4],
}, index=['a', 'b', 'c', 'd'])

# Create the second DataFrame
df2 = pd.DataFrame({
    'B': [5, 6, 7, 8],
}, index=['b', 'c', 'd', 'e'])

print("DataFrame 1:\n", df1)
print("\nDataFrame 2:\n", df2)

Output:

Methods to Get the Common Index of Pandas DataFrame

Now, let us see a few different ways to get common index of Pandas DataFrame in Python.

1. Using intersection()

Pandas provides a straightforward way to get the common indices using the intersection() method. This method returns the intersection of two Index objects, which represents the common elements between them.

# import pandas module
import pandas as pd

# Create the first DataFrame
df1 = pd.DataFrame({
    'A': [1, 2, 3, 4],
}, index=['a', 'b', 'c', 'd'])

# Create the second DataFrame
df2 = pd.DataFrame({
    'B': [5, 6, 7, 8],
}, index=['b', 'c', 'd', 'e'])

# finding common index using intersection()
common_index = df1.index.intersection(df2.index)
print("Common Index:", common_index)

Output:

Common Index: Index(['b', 'c', 'd'], dtype='object')

2. Using Set Operations

Another approach is to treat the indices as Python sets and use set operations to find the common indices. This approach is particularly useful if we’re familiar with Python’s set operations. This approach provides the same result, though the output is a set instead of a Pandas Index object.

# import pandas module
import pandas as pd

# Create the first DataFrame
df1 = pd.DataFrame({
    'A': [1, 2, 3, 4],
}, index=['a', 'b', 'c', 'd'])

# Create the second DataFrame
df2 = pd.DataFrame({
    'B': [5, 6, 7, 8],
}, index=['b', 'c', 'd', 'e'])

# finding common index using
common_index_set = set(df1.index) & set(df2.index)
print("Common Index (Set Operation):", common_index_set)

Output:

Common Index (Set Operation): {'d', 'c', 'b'}

Filtering DataFrames by Common Index

Once we have the common index, we might want to filter the DataFrames to only include the rows that share this index. This can be done by using the df.loc[] label indexer. It selects the rows from df1 where the index values match the common_index. It filters df1 down to only those rows that share the common indices.

# import pandas module
import pandas as pd

# Create the first DataFrame
df1 = pd.DataFrame({
    'A': [1, 2, 3, 4],
}, index=['a', 'b', 'c', 'd'])

# Create the second DataFrame
df2 = pd.DataFrame({
    'B': [5, 6, 7, 8],
}, index=['b', 'c', 'd', 'e'])

# finding common index using intersection()
common_index = df1.index.intersection(df2.index)

# Filter df1 by common index
df1_filtered = df1.loc[common_index]

# Filter df2 by common index
df2_filtered = df2.loc[common_index]

print("Filtered DataFrame 1:\n", df1_filtered)
print("\nFiltered DataFrame 2:\n", df2_filtered)

Output:

Merging on the Common Index

Once we have the common indices, we can perform a merge operation to combine the data from both DataFrames based on these shared indices. This is done by using the Pandas merge() method which combines the two DataFrames based on the common index, keeping only the rows where the index is shared.

# import pandas module
import pandas as pd

# Create the first DataFrame
df1 = pd.DataFrame({
    'A': [1, 2, 3, 4],
}, index=['a', 'b', 'c', 'd'])

# Create the second DataFrame
df2 = pd.DataFrame({
    'B': [5, 6, 7, 8],
}, index=['b', 'c', 'd', 'e'])

# merging dataframe based on common index
merged_df = pd.merge(df1, df2, left_index=True, right_index=True, how='inner')
print("Merged DataFrame:\n", merged_df)

Output:

Merged DataFrame:
    A  B
b  2  5
c  3  6
d  4  7

Conclusion

Identifying the common index between two Pandas DataFrames is a valuable technique when comparing, merging, or aligning data. Whether we want to merge DataFrames based on shared indices or simply retrieve the common indices for further analysis, Pandas provides efficient ways to achieve this using functions like merge() and intersection().