PANDAS in Python:

Handling missing values and duplicate data is a common data preprocessing task when
working with datasets in Python, and the Pandas library is a powerful tool for these tasks.
Here's how you can handle missing values and duplicate data with Pandas:

Handling Missing Values:

Missing values in a dataset can be represented in various ways, such as NaN, None, or other
custom placeholders. Pandas provides several methods to handle missing values:

1. Detect Missing Values: You can use the isna() or isnull() methods to detect
missing values in a DataFrame.

import pandas as pd

df = pd.read_csv('your_data.csv')

missing_values = df.isnull() # or df.isna()

 Drop Missing Values: To remove rows or columns containing missing values, you can
use the dropna() method.

df.dropna() # Removes rows with missing values

df.dropna(axis=1) # Removes columns with missing values

 Fill Missing Values: You can fill missing values using the fillna() method with a
specified value.

df.fillna(value) # Fill missing values with a specific value

 Interpolate Missing Values: You can use interpolation methods to estimate missing
values.

df.interpolate() # Interpolate missing values

Handling Duplicate Data:

Duplicate data can lead to incorrect analysis and results. Pandas provides methods to handle
duplicate data:

1. Detect Duplicates: Use the duplicated() method to identify duplicate rows and the
drop_duplicates() method to remove them.
duplicated_rows = df[df.duplicated()]
df.drop_duplicates()

2. Keep the First or Last Occurrence: When removing duplicates, you can specify
whether to keep the first or last occurrence of a duplicate row using the keep
parameter.

df.drop_duplicates(keep='first') # Keep the first occurrence (default)

df.drop_duplicates(keep='last') # Keep the last occurrence

By using these methods, you can effectively handle missing values and duplicate data in your
Pandas DataFrame. It's important to choose the method that best suits your data and analysis
needs.

Handling Outliers:

Outliers are data points that significantly differ from the rest of the data and can affect the
analysis. Here's how you can detect and handle outliers:

1. Detect Outliers: Use statistical methods like z-scores or IQR (Interquartile Range) to
identify outliers in your dataset.

from scipy import stats

z_scores = stats.zscore(df)
abs_z_scores = np.abs(z_scores)
outlier_rows = (abs_z_scores > 3).all(axis=1)
outliers = df[outlier_rows]

Data Imputation:

Data imputation is the process of filling in missing values with estimates to make the dataset
complete. Here are some methods for data imputation using Pandas:

import pandas as pd

# Sample DataFrame with missing values

data = {'A': [1, 2, 3, None, 5],

'B': [4, 5, None, 7, 8],
'C': [None, 2, 3, 4, 5]}
df = pd.DataFrame(data)

# Imputation with mean, median, and mode

mean_imputed = df.fillna(df.mean())

median_imputed = df.fillna(df.median())

mode_imputed = df.fillna(df.mode().iloc[0]) # mode() returns a DataFrame, so we use iloc[0]

to select the first mode

# Display the original and imputed DataFrames

print("Original DataFrame:")
print(df)
print("\nImputed with Mean:")
print(mean_imputed)
print("\nImputed with Median:")
print(median_imputed)
print("\nImputed with Mode:")
print(mode_imputed)

In this code, we first create a sample DataFrame with missing values. Then, we use the
fillna() method to perform mean, median, and mode imputation on the DataFrame,
creating three separate DataFrames for each imputation method.

Remember that mean imputation fills missing values with the mean of the respective column,
median imputation uses the median, and mode imputation uses the mode (most frequent
value). The code showcases how to apply each of these imputation techniques to handle
missing values in your data.

Data Normalization and Standardization:

Data normalization and data standardization are two common techniques used in data
preprocessing to prepare data for analysis or machine learning tasks. They are used to
transform the data in a way that makes it more suitable for modeling, improving the
performance and interpretability of machine learning algorithms. These techniques are often
used when working with numerical data.

1. Data Normalization:

Data normalization is the process of rescaling data to a common scale without

distorting differences in the ranges of values. It is particularly useful when you have
 features with different units or different scales. Normalization typically scales the data
to a range between 0 and 1.

It scales the data so that it falls within a specified range, typically 0 to 1.

o Formula:

X_normalized = (X - X_min) / (X_max - X_min)

 Data Standardization:

Data standardization, also known as feature scaling, is a specific type of data normalization
that transforms data to have a mean of 0 and a standard deviation of 1. Standardization is
particularly useful when you are dealing with algorithms that assume your data is normally
distributed, as it ensures that the data conforms to a standard Gaussian distribution.

Advantages of standardization:

 It makes the data more suitable for algorithms like principal component analysis
(PCA) and linear regression.
 It helps in comparing features with different units or scales.

Standardization formula:

X_standardized = (X - mean) / standard deviation

When to use normalization vs. standardization:

 Use normalization when you have features with different ranges or when you want to
scale the data to a specific range.
 Use standardization when you want to transform the data to have a mean of 0 and a
standard deviation of 1, which is often a requirement for certain statistical and
machine learning models.

The choice between normalization and standardization depends on the nature of your data
and the requirements of your specific machine learning algorithm. In some cases, it's also a
good practice to try both methods and see which one works better for your particular
problem.
Interview Questions:

1. How do you handle missing values in a Pandas DataFrame?

 Answer: You can use the fillna() method to fill missing values with a specific
value or use the dropna() method to remove rows or columns with missing values.

2. What is the function of the drop_duplicates() method in Pandas?

 Answer: drop_duplicates() is used to remove duplicate rows from a DataFrame,

keeping only the first occurrence by default.

3. How can you handle outliers in a DataFrame using Pandas?

 Answer: You can identify outliers using methods like z-scores or IQR, and then
decide to either remove them or transform them to reduce their impact on the analysis.

4. Explain the concept of data imputation in Pandas.

 Answer: Data imputation is the process of filling missing values with estimated or
substituted values. Pandas provides functions like fillna() for this purpose.

5. What are some common techniques for data imputation in Pandas?

 Answer: Common techniques include mean, median, mode imputation, forward or

backward fill, interpolation, and machine learning-based imputation.

6. How do you normalize data in a Pandas DataFrame?

 Answer: You can normalize data by scaling it to a specific range, typically [0, 1],
using techniques like Min-Max scaling with the MinMaxScaler from the
sklearn.preprocessing library.

7. What is data standardization, and how can you achieve it using Pandas and scikit-
learn?

 Answer: Data standardization (or z-score standardization) scales data to have a mean
of 0 and a standard deviation of 1. You can use the StandardScaler from scikit-learn
or manually calculate it with Pandas.

8. Can you demonstrate how to remove all rows with missing values in a Pandas
DataFrame?

 Answer: You can use the dropna() method with how='any' to remove all rows
containing any missing values.

9. Explain the difference between imputing missing values with the mean and median.
  Answer: Imputing with the mean replaces missing values with the average of the
available data, while imputing with the median replaces them with the middle value,
making it less sensitive to outliers.

Practice Problems & Solutions With Employee

Information Dataset
First load dataset
import pandas as pd
# Load the dataset
df = pd.read_csv('employee_data.csv')

Problem 1: Identify and count the number of missing values in each column of the dataset.
Problem 2: Find and remove duplicate records from the dataset.
Problem 3: Identify outliers in the 'Salary' column using the Z-score method (threshold: z-score >
3 or < -3).
Problem 4: Normalize the 'Age' column using the min-max normalization technique.
Problem 5: Standardize the 'Salary' column using the z-score standardization technique.
Problem 6: Fill in missing values in the 'Gender' column with the mode value.