### Objective Overview:
The goal of this assignment is to guide you through the process of data preprocessing using Python
libraries like pandas, numpy, scikit-learn, and seaborn. You will apply techniques for data cleaning,
transformation, and visualization, ultimately preparing the dataset for further analysis or machine
learning.
### Step-by-Step Breakdown:
---
### 1. **Dataset Selection**:
Choose a dataset that fits the criteria:
- At least 500 rows and multiple columns of varying data types (numerical, categorical, text, etc.).
- Suitable open data sources include:
- **Kaggle**: Provides datasets on diverse topics (e.g., health, finance, sports).
- **UCI Machine Learning Repository**: Offers datasets used for machine learning tasks.
- **Open Data Portals**: Many governments and organizations release datasets for public use.
**Dataset Example**: Suppose we select the **"Titanic: Machine Learning from Disaster" dataset** from
Kaggle (contains 891 rows, with both numerical and categorical data).
---
### 2. **Data Cleaning**:
#### Missing Values:
- **Step 1**: Identify missing values.
‘‘‘python
import pandas as pd
# Load the dataset
data = pd.read_csv(’titanic.csv’)
# Identify missing values
missing_values = data.isnull().sum()
print(missing_values)
‘‘‘
- **Step 2**: Handle missing values. Depending on the column type and context, you can:
- Impute numerical values (e.g., mean, median).
- Impute categorical values (e.g., mode or constant).
- Drop rows or columns with excessive missing data.
‘‘‘python
# Example of imputing missing ’Age’ with the median
data[’Age’].fillna(data[’Age’].median(), inplace=True)
# Example of imputing missing ’Embarked’ with the mode
data[’Embarked’].fillna(data[’Embarked’].mode()[0], inplace=True)
‘‘‘
#### Duplicates:
- **Step 3**: Detect and remove duplicate rows.
‘‘‘python
# Check for duplicates
duplicates = data.duplicated().sum()
print(f"Number of duplicate rows: {duplicates}")
# Remove duplicates
data = data.drop_duplicates()
‘‘‘
#### Outliers:
- **Step 4**: Identify outliers using the **Z-score** or **IQR (Interquartile Range)** method.
‘‘‘python
import numpy as np
from scipy.stats import zscore
# Calculate Z-scores for numerical columns
data_zscore = data.select_dtypes(include=[np.number])
z_scores = np.abs(zscore(data_zscore))
# Threshold for identifying outliers
threshold = 3
outliers = (z_scores > threshold).sum()
print(f"Outliers detected: {outliers}")
‘‘‘
- **Step 5**: Handle outliers by removing or capping.
‘‘‘python
# Remove outliers (Z-score > 3)
data_clean = data[(z_scores < threshold).all(axis=1)]
‘‘‘
---
### 3. **Data Transformation**:
#### Normalization/Standardization:
- **Step 6**: Normalize or standardize numerical features.
‘‘‘python
from sklearn.preprocessing import MinMaxScaler, StandardScaler
# Example of Min-Max Scaling
scaler = MinMaxScaler()
data_scaled = data.copy()
data_scaled[’Age’] = scaler.fit_transform(data[[’Age’]])
# Example of Z-score Standardization
standardizer = StandardScaler()
data_standardized = data.copy()
data_standardized[’Age’] = standardizer.fit_transform(data[[’Age’]])
‘‘‘
#### Encoding Categorical Variables:
- **Step 7**: Convert categorical variables into numerical formats using encoding.
‘‘‘python
# One-Hot Encoding (e.g., ’Sex’ and ’Embarked’ columns)
data_encoded = pd.get_dummies(data, columns=[’Sex’, ’Embarked’])
# Label Encoding (e.g., ’Survived’ column)
from sklearn.preprocessing import LabelEncoder
label_encoder = LabelEncoder()
data[’Survived’] = label_encoder.fit_transform(data[’Survived’])
‘‘‘
#### Date and Time Features:
- **Step 8**: Extract useful features from date columns (if applicable).
‘‘‘python
# Example: Convert ’Date’ column into year, month, day features
data[’Year’] = pd.to_datetime(data[’Date’]).dt.year
data[’Month’] = pd.to_datetime(data[’Date’]).dt.month
‘‘‘
#### Text Data Preprocessing:
- **Step 9**: If text data is available, preprocess it using tokenization, stop words removal, and
stemming/lemmatization.
‘‘‘python
from sklearn.feature_extraction.text import CountVectorizer
# Example of text tokenization
vectorizer = CountVectorizer(stop_words=’english’)
X = vectorizer.fit_transform(data[’TextColumn’])
# Optionally, apply stemming/lemmatization using libraries like NLTK
‘‘‘
---
### 4. **Data Visualization**:
Visualize the dataset to understand its distribution and relationships.
#### Histograms:
- **Step 10**: Create a histogram for numerical features.
‘‘‘python
import seaborn as sns
import matplotlib.pyplot as plt
sns.histplot(data[’Age’], kde=True)
plt.title(’Age Distribution’)
plt.show()
‘‘‘
#### Box Plots:
- **Step 11**: Visualize outliers with box plots.
‘‘‘python
sns.boxplot(x=data[’Age’])
plt.title(’Box Plot of Age’)
plt.show()
‘‘‘
#### Heatmap (Correlation Matrix):
- **Step 12**: Visualize correlations between numerical features.
‘‘‘python
corr_matrix = data.corr()
sns.heatmap(corr_matrix, annot=True, cmap=’coolwarm’)
plt.title(’Correlation Heatmap’)
plt.show()
‘‘‘
#### Scatter Plot:
- **Step 13**: Visualize relationships between features using scatter plots.
‘‘‘python
sns.scatterplot(x=data[’Age’], y=data[’Fare’])
plt.title(’Age vs Fare’)
plt.show()
‘‘‘
---
### 5. **Feature Engineering**:
- **Step 14**: Create new features based on existing data. For example, combine ’SibSp’ and ’Parch’ into
a new feature, ’FamilySize’.
‘‘‘python
data[’FamilySize’] = data[’SibSp’] + data[’Parch’]
‘‘‘
- **Step 15**: Perform feature selection to identify the most important features.
‘‘‘python
from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import SelectFromModel
X = data.drop(’Survived’, axis=1)
y = data[’Survived’]
# Use Random Forest to rank features by importance
rf = RandomForestClassifier(n_estimators=100)
rf.fit(X, y)
# Select important features
selector = SelectFromModel(rf, threshold="mean")
X_selected = selector.transform(X)
‘‘‘
---
### 6. **Documentation**:
- **Code Documentation**: Add comments and explanations to clarify the rationale behind each
preprocessing step.
- **Preprocessing Impact**:
- **Missing Value Handling**: Imputing or removing missing data can improve model performance by
ensuring no incomplete rows or columns.
- **Outlier Removal**: Identifying and removing outliers ensures the model is not unduly influenced by
extreme values.
- **Encoding**: Converting categorical data into numerical values makes it compatible with machine
learning algorithms.
- **Feature Engineering**: Creating new features helps enhance model accuracy by providing additional
information for the algorithm.
---
### Final Thoughts:
After completing these preprocessing steps, your dataset will be clean, transformed, and ready for
machine learning or further analysis. Keep in mind that data preprocessing is a crucial step, as it directly
impacts the quality of insights and predictions generated by your models.
