How to select the largest of each group in Python Pandas DataFrame?

Introduction

One of the most basic and common operations to perform during data analysis is to select rows containing the largest value of some columns within a group. In this post, I will show you how to find the largest of each group within a DataFrame.

Problem..

Let us understand the task first, assume you are given a movies dataset and requested to list the most popular film of each year based on popularity.

How to do it..

1.Preparing the data.

Well Google is full of datasets. I often use kaggle.com to get the datasets I need for my data analysis. Feel free to login to kaggle.com and search for movies. Download the movies dataset to the directory and import it into Pandas DataFrame.

If you have downloaded the data just like me from kaggle.com, please like the person who helped you with the data.

import pandas as pd
import numpy as np
movies = pd.read_csv("https://raw.githubusercontent.com/sasankac/TestDataSet/master/movies_data.csv")

# see sample 5 rows
print(f"Output \n\n*** {movies.sample(n=5)} ")

Output

*** budget id original_language original_title popularity \
2028 22000000 235260 en Son of God 9.175762
2548 0 13411 en Malibu's Most Wanted 7.314796
3279 8000000 26306 en Prefontaine 8.717235
3627 5000000 10217 en The Sweet Hereafter 7.673124
4555 0 98568 en Enter Nowhere 3.637857

release_date revenue runtime status title \
2028 28/02/2014 67800064 138.0 Released Son of God
2548 10/04/2003 0 86.0 Released Malibu's Most Wanted
3279 24/01/1997 589304 106.0 Released Prefontaine
3627 14/05/1997 3263585 112.0 Released The Sweet Hereafter
4555 22/10/2011 0 90.0 Released Enter Nowhere

vote_average vote_count
2028 5.9 83
2548 4.7 77
3279 6.7 21
3627 6.8 103
4555 6.5 49

2. Perform some basic data anaysis to understand the data.

# Identify the data-types
print(f"Output \n*** Datatypes are {movies.dtypes} ")

Output

*** Datatypes are budget int64
id int64
original_language object
original_title object
popularity float64
release_date object
revenue int64
runtime float64
status object
title object
vote_average float64
vote_count int64
dtype: object

2. Now, if we want to save bunch of memory usage we can convert the datatypes of float64 and int64. But we have to be careful and do our homework before converting the data types.

# Check the maximum numeric value.
print(f"Output \n *** maximum value for Numeric data type - {movies.select_dtypes(exclude=['object']).unstack().max()}")

# what is the max vote count value
print(f" *** Vote count maximum value - {movies[['vote_count']].unstack().max()}")

# what is the max movie runtime value
print(f" *** Movie Id maximum value - {movies[['runtime']].unstack().max()}")

Output

*** maximum value for Numeric data type - 2787965087.0
*** Vote count maximum value - 13752
*** Movie Id maximum value - 338.0

3. There are columns that need not be represented in 64 bits and can be brought down to 16 bit, so let's do it. 64 Bit int range is from -32768 to +32767. I will do it for vote_count and runtime and you can do it for the columns that requires less memory storage.

4. Now, to identify the most popular film for each year, we need to group by release_date and get the maximum value of popularity. A typical SQL looks some thing like below.

SELECT movie with max popularity FROM movies GROUP BY movie released year

5. Unfortunately our release_date is an Object data-type there are couple of ways to convert them to datetime. I will choose to create a new column with just year so that I can use that column for grouping.

movies['year'] = pd.to_datetime(movies['release_date']).dt.year.astype('Int64')
print(f"Output \n ***{movies.sample(n=5)}")

Output

*** budget id original_language original_title popularity \
757 0 87825 en Trouble with the Curve 18.587114
711 58000000 39514 en RED 41.430245
1945 13500000 152742 en La migliore offerta 30.058263
2763 13000000 16406 en Dick 4.742537
4595 350000 764 en The Evil Dead 35.037625

release_date revenue runtime status title \
757 21/09/2012 0 111.0 Released Trouble with the Curve
711 13/10/2010 71664962 111.0 Released RED
1945 1/01/2013 19255873 124.0 Released The Best Offer
2763 4/08/1999 27500000 94.0 Released Dick
4595 15/10/1981 29400000 85.0 Released The Evil Dead

vote_average vote_count year
757 6.6 366 2012
711 6.6 2808 2010
1945 7.7 704 2013
2763 5.7 67 1999
4595 7.3 894 1981

Method 1 - Without Using Group By

6. We need only 3 columns, movie titles, movie release year and popularity. So we choose those columns and use sort_values on year to see how the results look like.

print(f"Output \n *** Method 1- Without Using Group By")
movies[["title", "year", "popularity"]].sort_values("year", ascending=True)

Output

*** Without Using Group By

4803 rows × 3 columns

8. Now looking at the results, we need to sort the popularity as well to get the most popular movie in a year. Pass the columns in interest as a list. ascending=False will result the sorting results in descending order.

movies[["title", "year", "popularity"]].sort_values(["year","popularity"], ascending=False)

4802 rows × 3 columns

9. Alright, the data is now sorted perfectly. So the next step is to just keep the first value for each year and remove rest. Guess how to do it?.

We will use .drop_duplicates method.

movies[["title", "year", "popularity"]].sort_values(["year","popularity"], ascending=False).drop_duplicates(subset="year")

91 rows × 3 columns

Method 2 - Using Group By

We can acheive the same with groupby as well. The approach is very similar to the SQL shown above.

print(f"Output \n *** Method 2 - Using Group By")
movies[["title", "year", "popularity"]].groupby("year", as_index=False).apply(lambda df:df.sort_values("popularity", ascending=False)
.head(1)).droplevel(0).sort_values("year", ascending=False)

Output

*** Method 2 - Using Group By

90 rows × 3 columns