NumPy | Replace NaN values with average of columns
Last Updated :
09 Feb, 2024
Data visualization is one of the most important steps in machine learning and data analytics.
Cleaning and arranging data is done by different algorithms. Sometimes in data sets, we get NaN (not a number) values that are unusable for data visualization.
To solve this problem, one possible method is to replace NaN values with an average of columns.
Given below are a few methods to solve this problem.
- Using np.colmean and np.take
- Using np.ma and np.where
- Using Naive and zip
- Using list comprehension and built-in functions
- Using zip()+lambda()
Let us understand them better with Python program examples:
Using np.colmean and np.take
We use the colmean() method of the NumPy library to find the mean of columns. We then use the take() method to replace column mean (average) with NaN values.
Example:
Python3
# Python code to demonstrate
# to replace nan values
# with an average of columns
import numpy as np
# Initialising numpy array
ini_array = np.array([[1.3, 2.5, 3.6, np.nan],
[2.6, 3.3, np.nan, 5.5],
[2.1, 3.2, 5.4, 6.5]])
# printing initial array
print ("initial array", ini_array)
# column mean
col_mean = np.nanmean(ini_array, axis = 0)
# printing column mean
print ("columns mean", str(col_mean))
# find indices where nan value is present
inds = np.where(np.isnan(ini_array))
# replace inds with avg of column
ini_array[inds] = np.take(col_mean, inds[1])
# printing final array
print ("final array", ini_array)
Output:
initial array [[ 1.3 2.5 3.6 nan]
[ 2.6 3.3 nan 5.5]
[ 2.1 3.2 5.4 6.5]]
columns mean [ 2. 3. 4.5 6. ]
final array [[ 1.3 2.5 3.6 6. ]
[ 2.6 3.3 4.5 5.5]
[ 2.1 3.2 5.4 6.5]]
Using np.ma and np.where
We use the ma() method, which allows you to create a masked array where NaN values are masked out. We then use the where() method to replace the NaN values with column averages.
Example:
Python3
# Python code to demonstrate
# to replace nan values
# with average of columns
import numpy as np
# Initialising numpy array
ini_array = np.array([[1.3, 2.5, 3.6, np.nan],
[2.6, 3.3, np.nan, 5.5],
[2.1, 3.2, 5.4, 6.5]])
# printing initial array
print ("initial array", ini_array)
# replace nan with col means
res = np.where(np.isnan(ini_array), np.ma.array(ini_array,
mask = np.isnan(ini_array)).mean(axis = 0), ini_array)
# printing final array
print ("final array", res)
Output:
initial array [[ 1.3 2.5 3.6 nan]
[ 2.6 3.3 nan 5.5]
[ 2.1 3.2 5.4 6.5]]
final array [[ 1.3 2.5 3.6 6. ]
[ 2.6 3.3 4.5 5.5]
[ 2.1 3.2 5.4 6.5]]
Using Naive and zip
We use Zip to pair up the elements from the unpacked arrays, effectively giving us pairs of (row, column) indices for each NaN value in the array. We then replace these values with column averages.
Example:
Python3
# Python code to demonstrate
# to replace nan values
# with average of columns
import numpy as np
# Initialising numpy array
ini_array = np.array([[1.3, 2.5, 3.6, np.nan],
[2.6, 3.3, np.nan, 5.5],
[2.1, 3.2, 5.4, 6.5]])
# printing initial array
print ("initial array", ini_array)
# indices where values is nan in array
indices = np.where(np.isnan(ini_array))
# Iterating over numpy array to replace nan with values
for row, col in zip(*indices):
ini_array[row, col] = np.mean(ini_array[
~np.isnan(ini_array[:, col]), col])
# printing final array
print ("final array", ini_array)
Output:
initial array [[ 1.3 2.5 3.6 nan]
[ 2.6 3.3 nan 5.5]
[ 2.1 3.2 5.4 6.5]]
final array [[ 1.3 2.5 3.6 6. ]
[ 2.6 3.3 4.5 5.5]
[ 2.1 3.2 5.4 6.5]]
Using list comprehension and built-in functions
It first computes the column means using a list comprehension with the help of the filter and zip functions. Then, it replaces the NaN values in the array with the corresponding column means using another list comprehension with the help of the enumerate function. Finally, it returns the modified list.
Algorithm:
1. Compute the column means.
2. Replace the NaN values in the array with the corresponding column means using list comprehension and built-in functions.
3. Return the modified list.
Python3
def replace_nan_with_mean(arr):
col_means = [sum(filter(lambda x: x is not None, col))/len(list(filter(lambda x: x is not None, col))) for col in zip(*arr)]
for i in range(len(arr)):
arr[i] = [col_means[j] if x is None else x for j, x in enumerate(arr[i])]
return arr
arr=[[1.3, 2.5, 3.6, None],
[2.6, 3.3, None, 5.5],
[2.1, 3.2, 5.4, 6.5]]
print(replace_nan_with_mean(arr))
Output[[1.3, 2.5, 3.6, 6.0], [2.6, 3.3, 4.5, 5.5], [2.1, 3.2, 5.4, 6.5]]
Using zip()+lambda()
Compute the column means excluding NaN values using a loop over the transposed array zip(*arr). Replace NaN values with column means using map() and lambda functions.
Algorithm
1. Initialize an empty list means to store the column means.
2. Loop over the transposed array zip(*arr) to iterate over columns.
3. For each column, filter out None values and compute the mean of the remaining values. If there are no remaining values, set the mean to 0.
4. Append the mean to the means list.
5. Use map() and lambda functions to replace None values with the corresponding column mean in each row of the array arr.
6. Return the modified array arr.
Python3
# initial array
arr = [[1.3, 2.5, 3.6, None],
[2.6, 3.3, None, 5.5],
[2.1, 3.2, 5.4, 6.5]]
# compute column means
means = []
for col in zip(*arr):
values = [x for x in col if x is not None]
means.append(sum(values)/len(values) if values else 0)
# replace NaN values with column means
arr = list(map(lambda row: [means[j] if x is None else x for j,x in enumerate(row)], arr))
# print final array
print(arr)
Output[[1.3, 2.5, 3.6, 6.0], [2.6, 3.3, 4.5, 5.5], [2.1, 3.2, 5.4, 6.5]]
Similar Reads
How to Replace Numpy NAN with String Dealing with missing or undefined data is a common challenge in data science and programming. In the realm of numerical computing in Python, the NumPy library is a powerhouse, offering versatile tools for handling arrays and matrices. However, when NaN (not a number) values appear in your data, you
2 min read
How to Remove columns in Numpy array that contains non-numeric values? Many times we have non-numeric values in NumPy array. These values need to be removed, so that array will be free from all these unnecessary values and look more decent. It is possible to remove all columns containing Nan values using the Bitwise NOT operator and np.isnan() function. Example 1: Pyth
2 min read
Numpy MaskedArray.average() function | Python numpy.MaskedArray.average() function is used to return the weighted average of array over the given axis. Syntax : numpy.ma.average(arr, axis=None, weights=None, returned=False) Parameters: arr :[ array_like] Input masked array whose data to be averaged. Masked entries are not taken into account in
3 min read
Averaging over every N elements of a Numpy Array In this article, we will learn how to find the average over every n element of a NumPy array. For doing our task, we will some inbuilt methods provided by NumPy module which are as follows: numpy.average() to calculate the average i.e the sum of all the numbers divided by the number of elementsnumpy
3 min read
How to Remove rows in Numpy array that contains non-numeric values? Many times NumPy arrays may contain NaN values that need to be removed to ensure the array is free from unnecessary or invalid data. This can be achieved using the np.isnan() function along with the Bitwise NOT operator. Note that this approach specifically targets NaN values and may not handle othe
2 min read
NumPy - Arithmetic operations with array containing string elements Numpy is a library of Python for array processing written in C and Python. Computations in numpy are much faster than that of traditional data structures in Python like lists, tuples, dictionaries etc. due to vectorized universal functions. Sometimes while dealing with data, we need to perform arith
2 min read