Data Analytics with Python

Data Analytics Overview

 Learning Objectives

By the end of this lesson, you will be able to:

Describe the process of data analytics and its steps

List the skills and tools required for data analysis

Understand the challenges of data analytics

Explain exploratory data analysis technique

Illustrate data visualization techniques

Describe hypothesis testing

Data Analytics
 Why Data Analytics?

Data, by itself, is just an information source. Unless you understand it, you will not be able to use it effectively.

Date Description Deposit Withdrawal Balance

Apr 1 ATM Post Debit 100 $200,000

Apr 2 PayPal Transfer 231054 200 $202,000

Apr 3 Simplilearn course fee 150 $200,500

Overall patterns not
Apr 4 Starbucks Café 210 $198,400 clearly visible
Apr 5 Walmart TX 230 $196,100

Apr 6 eBay swiss watch 239 250 $193,600

Apr 7 Caterpillar black boots men 270 $190,900

Apr 8 Halo blue shirt 831 160 $189,300

 Why Data Analytics?

When the transaction details are presented as a line chart, the deposit and withdrawal patterns become apparent.

Overall pattern
 Why Data Analytics?

When the transaction details are presented as a line chart, the deposit and withdrawal patterns become
apparent. It helps view and analyze general trends and discrepancies.

Discrepancy
 Introduction to Data Analytics

Data analytics is a combination of processes to extract information from datasets.

Data Analytics Process
Data Analytics Process
 Data Analytics: Skills and Tools

Question or Business Data Acquisition Data Wrangling Data Exploration Conclusion or Communication or
Problem Predictions Data Visualization
• Ability to ask • Beautiful Soup • CSV or other file • NumPy • Scikit-Learn, the • Pandas
appropriate for web scraping knowledge • SciPy main machine • Database
questions and • CSV or other file • NumPy • Pandas learning library • Matplotlib
know the knowledge • Pandas • Matplotlib • CSV or other file • PPT
business • NumPy • Database knowledge • CSV or other file
• Domain • Pandas • SciPy • NumPy knowledge
knowledge • Database • Pandas
• Passion for data • Database
• Analytical • SciPy
approach
 Business Problem

The process of analytics begins with questions or business problems of the stakeholders.

Sales Inventory

Customers Traffic volume

Business problems trigger the need to

analyze data and find answers.
 Data Acquisition

Collect data from various sources for analysis to answer the question raised in step 1.

Twitter, Facebook,
LinkedIn, and other social
media and information
sites provide streaming
APIs.

Data Scientist expertise:

File handling
• Database Skills Data
• Web scraping
Server logs can be
extracted from enterprise
system servers to analyze
and optimize application
performance.
 Data Wrangling and Exploration

Data wrangling is the most important phase of the data analytics process.

Data cleansing Data Data pattern

Data discovery
manipulation

Data Wrangling Data Exploration

 Data Wrangling: Challenges

This phase includes data cleansing, data manipulation, data aggregation, data split, and reshaping of data.

Causes of challenges in the data wrangling phase:

• Unexpected data format

• Erroneous data
• Voluminous data to be manipulated
• Classifying data into linear or clustered
• Determining relationship between observation, feature, and response

Data wrangling is the most challenging phase and takes upto 70% of the Data Scientist’s time.
 Data Exploration: Model Selection

Model selection is based on the overall data analysis process to draw conclusions and make accurate predictions.

Model selection
• Is based on the overall data analysis process
• Should be accurate to avoid iterations
• Depends on pattern identification and algorithms
• Depends on hypothesis building and testing
• Leads to building mathematical statistical functions
 Exploratory Data Analysis (EDA)

Approach Focus Assumptions EDA Techniques

Quantitative:
EDA techniques
The focus is on data Provides numeric
EDA approach make minimal or no
and its structure, outputs for the
studies the data to assumptions. They
outliers, and models inputted data
recommend suitable present all the
suggested by the Graphical:
models that best fit underlying data
data. Uses statistical
the data. without any data
functions for
loss.
graphical output
 EDA: Quantitative Technique

EDA quantitative technique has two goals: measurement of central tendency and spread of data.

Measurement of Central Tendency

Mean Mean is the point which indicates how centralized the data points are.
• Suitable for symmetric distributions

Median Median is the exact middle value.

• Suitable for skewed distributions and for catching outliers in the
dataset

Mode Mode is the most common value in the data (frequency).

Measurement of Spread
Variance Variance is approximately the mean of the squares of the deviations.

Standard deviation Standard deviation is the square root of the variance.

Interquartile range Interquartile range is the distance between the 75th and 25th percentile.
It’s essentially the middle 50% of the data.
 EDA: Graphical Technique

Histograms and scatter plots are two popular graphical techniques to depict data.

Histogram graphically summarizes the distribution of

25

a univariate dataset.

It shows:
Frequency
20

• the center or location of data (mean, median, or mode)

• the spread of data
15

• the skewness of data

• the presence of outliers
• the presence of multiple modes in the data
10

1 1 2 2
0 5 Per 0
Miles 5
Gallon
EDA: Graphical Technique

Scatter plot represents relationships between two variables.

It can answer these questions visually:

• Are variables X and Y related?

• Are variables X and Y linearly related?
• Are variables X and Y nonlinearly related?
• Does change in variation of Y depend on X?
• Are there outliers?
Hypothesis
 Conclusion or Prediction

This step involves reaching a conclusion and making predictions based on the data analysis.

This phase:

• Involves heavy use of mathematical and statistical functions

• Requires model selection, training, and testing to help in forecasting

• Is called “machine learning” as data analysis is fully or semi-automated with

minimal or no human intervention

 Meaning of Hypothesis

Hypothesis is used to establish the relationship between dependent and independent variables.

Hypothesis building begins in the data exploration stage but

becomes more mature in the conclusion or prediction phase.

Data Exploration Stage Conclusion and Prediction

Key Considerations of Hypothesis Building

Testable explanations of a problem or observation
Used in quantitative and qualitative analyses to provide research solutions
Involves two variables, one dependent on another
Independent variable manipulated by the researcher
Dependent variable changes when the independent variable changes
 Hypothesis Building Using Feature Engineering

Domain knowledge leads to hypothesis building using feature engineering.

Feature engineering involves domain expertise to:

• Make sense of data
• Construct new features from raw data automatically
• Construct new features from raw data manually
 Hypothesis Building Using a Model
There are three phases to hypothesis building which are model building, model evaluation, and model deployment.

Phase 1: Model Building

• Identify best input variables
• Evaluate the model’s capacity to forecast with these
variables

Phase 2: Model Evaluation

• Train and test the model for accuracy
• Optimize model accuracy, performance, and
comparison with other models

Phase 3: Model Deployment

• Use the model for prediction
• Use the model to compare actual outcome with
expectations
 Hypothesis Testing

Draw two samples from the population and calculate the difference between their means.

μ1 Calculating the
difference
S1 between the two
means is called
hypothesis
testing.
μ2
S2
Hypothesis Testing

Alternative Hypothesis
• Proposed model outcome is
accurate and matches the data
• There is a difference between the
means of S1 and S2

Null Hypothesis
• Opposite to the alternative
hypothesis
• There is no difference between
the means of S1 and S2
 Hypothesis Testing Process

Choosing the training and test dataset and evaluating them with the null and alternative hypothesis.

Usually the training dataset is between 60% to 80% of the big dataset and the test dataset is between
20% to 40% of the big dataset.
Data Visualization
 Communication

The last step of data analysis is communication, where the analyzed data is formally presented to stakeholders.

Forms of data analysis presentations:

• Visual graphs
• Plotting maps
• Reports
• White paper reports
• PowerPoint presentations
 Data Visualization

Data visualization techniques are used for effective communication of data.

Benefits of data visualization: Examples of data visualization:

• Simplifies quantitative information through visuals • Presenting information about new and existing
• Shows the relationship between data points and customers on the website and their behavior when
variables they access the website
• Identifies patterns • Representing web traffic pattern for the website,
• Establishes trends for example, more activity on the website in the
morning than in the evening
 Plotting

Plotting is a data visualization technique used to represent underlying data through graphics.

Features of plotting:

• Plotting is like telling a story about data using

different colors, shapes, and sizes.
• Plotting shows the relationship between variables.
• Example:
o Change in value of Y results in change in
value of X
o X is independent of y
 Data Types for Plotting

There are two types of numerical data:

Discrete Data: Distinct or counted values
Example: Number of employees in a company or number of students in a class

Continuous Data: Values within a range that can be measured

Numerical Data Example: Height can be measured in feet or inches and weight can be measured in pounds or kilograms

There are two types of categorical data:

Cluster or group: Grouped values
Example: Students can be divided into different groups based on height: tall, medium, and short

Ordinal: Grouped values as per ranks

Categorical Data Example: A ranking system; a three-point scale with ranks like agree, strongly agree, and disagree

Data is measured in time blocks such as date, month, year, and time (hours, minutes, and seconds)

Time Series
 Types of Plot

Different data types can be visualized using various plotting techniques.

Knowledge Check
Knowledge
Check What is the goal of data acquisition?
Select all that apply.
1

a. Collect data from various data sources

b. Answer business questions through graphics

c. Collect web server logs

d. Scrape the web through web APIs

 Knowledge
Check What is the goal of data acquisition?
Select all that apply.
1

a. Collect data from various data sources

b. Answer business questions through graphics

c. Collect web server logs

d. Scrape the web through web APIs

The correct answer is a, c, d

Data acquisition is a process to collect data from various data sources, such as RDBMS and NoSQL databases, collect web
server logs, and also scrape the web through web APIs.
Knowledge
Check What is exploratory data analysis technique?
Select all that apply.
2

a. Analysis of data using quantitative techniques

b. Conducted only on a small subset of data

c. Analysis of data using graphical techniques

d. Suggests models that best fit the data

 Knowledge
Check What is exploratory data analysis technique?
Select all that apply.
2

a. Analysis of data using quantitative techniques

b. Conducted only on a small subset of data

c. Analysis of data using graphical techniques

d. Suggests models that best fit the data

The correct answer is a, c, d

Most EDA techniques are graphical in nature with a few quantitative techniques and also, suggest models that best fit the data.
They use almost the entire data with minimum and no assumptions.
Knowledge
Check Which plotting technique is used for continuous data?
Select all that apply.
3

a. Regression plot

b. Line chart

c. Histogram

d. Heat map
 Knowledge
Check Which plotting technique is used for continuous data?
Select all that apply.
3

a. Regression plot

b. Line chart

c. Histogram

d. Heat map

The correct answer is b, c

Line charts and histograms are used to plot continuous data.

Knowledge
Check
Which Python library is the main machine learning library?
4

a. Pandas

b. Matplotlib

c. Scikit-learn

d. NumPy
 Knowledge
Check
Which Python library is the main machine learning library?
4

a. Pandas

b. Matplotlib

c. Scikit-learn

d. NumPy

The correct answer is c

Scikit-learn is the main machine learning library in Python.

Knowledge
Check Which of the following includes data transformation, merging, aggregation, group by operation, and
reshaping?
5

a. Data acquisition

b. Data visualization

c. Data wrangling

d. Machine learning
 Knowledge
Check Which of the following includes data transformation, merging, aggregation, group by operation, and
reshaping?
5

a. Data acquisition

b. Data visualization

c. Data wrangling

d. Machine learning

The correct answer is c

Data wrangling includes data transformation, merging, aggregation, group by operation, and reshaping.
 Key Takeaways

You are now able to:

Describe the process of data analytics and its steps

List the skills and tools required for data analysis

Understand the challenges of data analytics

Explain exploratory data analysis technique

Illustrate data visualization techniques

Describe hypothesis testing

Thank You