Intro to
Data
Visualizatio
n
Python Version
What is Data Visualization For?
Exploratory Data
Analysis Benchmarking
Find out important Generating reports to
features and identify any present to
anomalies stakeholders/team
Use Cases
Evaluating Business
Problems Dashboarding
Identify any areas of Keeps Track of KPIs in
improvement by looking multiple important
at data business functions
Machine Learning Pipeline
Data Preparation & Exploratory Data Analysis
01
Feature Engineering & Feature Selection
02
Model Selection & Testing
03
Hyper-parameter Tuning & Overall Evaluation
04
Deployment
05
Basic Packages for Data Visualization
Data Manipulation:
pandas
Cool Charts:
Matplotlib
Seaborn (nicer graphs)
Interactive Charts:
Plotly
Why do EDA?
● Sanity Checks to look for missing data, outliers, redundant
information, etc
● Understand the distribution of the data and their relationships
● Look out for data leakage!
● Start Feature Selection
Installing/
Importing
Packages
Installing Packages
!pip install numpy==1.18.5
● Run this line of code to install numpy
● Packages must be installed before you can import them
● You can specify the version you want to install, in this case,
v1.18.5
Importing
import XX as YY
● XX refers to the package you want to import
● YY refers to how you will refer to the package as
Pandas
Functions
pd.read_csv()
Reading CSV file
● Data is often stored in the form of CSV
● Allows us to read the CSV, and put it into a tabular format, stored as a dataframe
● Link to CSV here
df.head() & df.tail()
Reading top & bottom rows
● Allows you to do a sanity check for the first few or last few rows of your dataframe
● Can specify the number of rows you want, default set to 5 rows
● e.g. df.head(10) for first 10 rows
● e.g. df.tail(10) for last 10 rows
df.info()
df.info()
● Gives us a quick summary of all the columns
● How do we deal with Null values if any?
df.describe()
df.describe()
● Gives us a quick summary of all the numerical
columns
Value Counts
df[‘Gender’].describe()
● Gives us a quick summary of the categorical variable
Unique Values
df['Season'].unique()
● Obtain all the unique values in a column
Dropping Columns
df.drop(['Gender'], axis = 1, inplace = True)
● Axis = 0 represents Row, Axis = 1 represents Column
● inplace = True to mutate (and save) the data
Dropping NA Values
df.dropna()
● Remove all rows with null or NaN (Not a Number) values
● Any row with a null/NaN value in any column will be dropped
Customer ID Age Payment Method
1 55 Venmo Customer ID Age Payment Method
2 NaN Cash 1 55 Venmo
3 50 NaN
4 21 PayPal
4 21 PayPal
Filling NA Values
df.fillna()
● Fill all rows with null values with a specified values
● e.g., df_filled = df_simple.fillna({'Age': 0, 'Payment Method': 'Unknown'})
Customer ID Age Payment Method Customer ID Age Payment Method
1 55 Venmo 1 55 Venmo
2 NaN Cash 2 0 Cash
3 50 NaN 3 50 Unknown
4 21 PayPal 4 21 PayPal
Sorting Values
df['Season'].value_counts().sort_values()
● Sorts the values of a column
● Smallest to Largest by default
● Use ascending = False sort by largest first
Datetime Manipulation
df[‘year_month’] = pd.to_datetime(df[‘order_date’])
● Converts a column (order_date) to datetime format
● Required for time-series analysis
Charts &
Plots
Types of Charts
Categorical Variables Numerical Data Time-Series Data
Visualization
- Bar charts - Time-series line charts
- Pie charts - Scatter plots - Area charts
- Stacked bar charts - Line charts - Gantt charts
- Radar charts - Histograms
- Box plots
Geospatial Data Others
-Choropleth map -Correlation Matrix
-Heatmap -Waterfall chart
-Many more!
Pie Charts
plt.pie(gender_counts, labels=gender_counts.index,
colors=colors, autopct='%1.1f%%')
● Useful for very little categories
● Very intuitive
● Used to quickly check for class imbalances /
category imbalances
Box Plots
sns.boxplot(x=col, y=counts)
● Quick overview of data distribution of all numerical features
● Can easily identify outliers
● Question: What do we do if we see any extreme outliers here?
Range
Q3 Quartile
outliers Median
Q1 Quartile
Bar Plots
sns.barplot(data=df, x='Category', y='Purchase Amount (USD)', errorbar=None, palette="pastel")
● Can accommodate for more categories
● Very intuitive, can add in a lot of other details i.e., counts
● Used to check for distribution of categories
Correlation Matrix Heatmap
sns.heatmap(correlation_matrix, annot=True, cmap=cmap, linewidths=0.5, square = True)
● Looks at correlation between all numerical values
● Extremely important to understand collinearity
● Useful to determine whether we need to remove highly correlated variables
Density Plot
sns.kdeplot(age_data, fill=True, legend=False)
● Look at the density distribution of continuous variables
● Useful for checking if the distribution is normal
● Easy to look for extreme outliers (e.g., high density of age > 80)
Choropleth Map
merged_data.plot(column='count', cmap='coolwarm', linewidth=0.8, ax=ax, edgecolor='0.8')
● Easily the coolest map (+10 style points)
● I added this map for fun
● Very useful in showing data distribution for geographical data
Data
Cleaning
Dealing with NA Values
df.dropna()
● Easiest method, just remove unclean data!
● What if NA values make up >20% of the data?
df[column].fillna()
● Fill NA values with something else
● Mean? Median?
● What about for categorical values?
What if data is not missing at random?
Dealing with NA Values
● In this example, all females’ Group is unknown
○ Data is not missing at random
○ Fill with ‘Unknown’
● What about Salary?
○ Impute with mean/median
○ Fill with 0
Gender Group Salary
Male A 50
Female ? 100
Male A ?
Female ? 50
Dealing with Extreme Values
Extreme outliers can skew models!
Some outliers may even be due to human error
Some data inherently will be extremely skewed: Salary, Age
df.drop()
● Easiest method also, just remove extreme outliers!
● What if it is not a normal distribution?
What if extreme data points tell us something important?
E.g., using salary to predict spending
Binning Numerical Data
Changing numerical features to categorical features
● Create groups for extreme values
○ <20: Age Group 0-20
○ >60: Age Group 61+
Learning from Documentation
SEABORN API : https://seaborn.pydata.org/api.html
● In Built examples of how you can use their plot methods
● Detailed explanation of customisations/parameters
Exercise
