Exploratory Data Analysis: INTRODUCTION TO

DATA SCIENCE

Presenter’s Name
Dr. Shital Bhatt
Associate Professor
School of Computational and Data Sciences

www.vidyashilpuniversity.com www.vidyashilpuniversity.com
Raw Data Implications and
Data from Multiple Sources
Introduction

 Definition of Raw Data

 - Unprocessed data collected from various sources

 Importance of Raw Data

 - Foundation for data analysis and decision-making

 Definition of Data from Multiple Sources

 - Data collected from various systems, formats, and locations
Implications of Raw Data - Benefits

 Complete and Unbiased Data

 - Provides a comprehensive view

 Flexibility in Analysis
 - Allows custom transformations and aggregations

 Richness
 - Contains detailed information
Implications of Raw Data - Challenges

 Volume
 - Large amounts of data to process

 Variety
 - Different formats and structures

 Veracity
 - Quality and accuracy of data

 Velocity
 - Speed at which data is generated and needs to be processed
Implications of Raw Data - Use Cases

 Big Data Analytics

 - Analyzing large datasets for insights

 Machine Learning
 - Training models on comprehensive datasets

 Business Intelligence
 - Driving strategic decisions with complete data
Processing Raw Data - Data Cleaning

 Removing Duplicates
 - Ensuring data uniqueness

 Handling Missing Values

 - Imputation, deletion, or ignoring missing data

 Correcting Errors
 - Fixing incorrect or inconsistent data
Processing Raw Data - Data Transformation

 Normalization
 - Scaling data to a standard range

 Aggregation
 - Summarizing data at different levels

 Encoding
 - Converting categorical data to numerical format
Processing Raw Data - Data Integration

 Combining Data from Different Sources

 - Merging datasets into a unified view

 Data Warehousing
 - Centralized storage for integrated data

 ETL (Extract, Transform, Load) Processes

 - Standard method for integrating data
Handling Data from Multiple Sources - Types
of Data Sources
 Structured Data
 - Databases, spreadsheets

 Semi-Structured Data
 - XML, JSON

 Unstructured Data
 - Text, images, videos
Handling Data from Multiple Sources - Data
Integration Techniques
 Batch Processing
 - Integrating data at scheduled intervals

 Real-Time Processing
 - Continuous integration as data is generated

 Data Virtualization
 - Accessing data without moving it
Handling Data from Multiple Sources - Data
Consistency and Quality
 Data Consistency
 - Ensuring data remains consistent across sources

 Data Quality
 - Accuracy, completeness, reliability

 Data Governance
 - Policies and procedures for managing data
Tools and Technologies - Data Integration
Tools
 Apache NiFi
 - Data routing and transformation

 Talend
 - Data integration platform

 Informatica
 - Enterprise data integration
Tools and Technologies - Data Cleaning Tools

 OpenRefine
 - Tool for cleaning messy data

 Trifacta
 - Data preparation platform

 DataWrangler
 - Interactive tool for data cleaning
Tools and Technologies - Data Transformation
Tools
 Apache Spark
 - Unified analytics engine

 Pandas (Python Library)

 - Data manipulation and analysis

 Alteryx
 - Data blending and advanced analytics
What is Raw Data?

 Definition: Unprocessed, unorganized data collected from its original source.

 Examples: Sensor readings, survey responses, website clickstream data.

 Emphasize the contrast between raw data and processed/structured data.

 Why is Raw Data Important?

 Foundation for all data analysis: Provides the building blocks for insights.

 Enables transparency and reproducibility: Allows others to verify and replicate findings.

 Potential for uncovering hidden patterns: Raw data may contain nuances lost in processing.
 Implications of Raw Data: Volume and Variety

 Volume: The sheer amount of raw data can be overwhelming (Big Data).

 Variety: Raw data comes in diverse formats (text, images, sensor data).

 Discuss challenges associated with storing, managing, and processing large, varied datasets.

 Implications of Raw Data: Quality and Security

 Quality: Raw data may contain errors, inconsistencies, and missing values.

 Security: Protecting raw data from unauthorized access or manipulation is crucial.

 Highlight the importance of data cleaning, validation, and security measures.

 Taming the Beast: Processing Raw Data

 Data Preprocessing: Cleaning, transforming, and organizing raw data for analysis.

 Data Transformation: Techniques like normalization, scaling, and feature engineering.

 Ethical Considerations of Raw Data

 Privacy: Protecting individual privacy when working with raw data, especially personal
information.

 Bias: Potential biases introduced during data collection or processing can skew results.

 Emphasize the importance of ethical data collection, handling, and analysis practices.
 The Future of Raw Data

 Advancements in data storage and processing technologies.

 Increased focus on real-time analysis of raw data streams (e.g., Internet of Things).

 The growing importance of data governance and ethical frameworks.

 Conclusion

 Raw data holds immense potential, but requires careful handling and processing.

 By understanding its implications, we can unlock valuable insights and make informed decisions.

 End with a call to action: responsible management and utilization of raw data for a better future.
Handling Data from Multiple
Sources
Introduction

 Definition of Data from Multiple Sources

 - Data collected from various systems, formats, and locations

 Importance of Integrating Data from Multiple Sources

 - Comprehensive insights, improved decision-making, and enhanced data quality
Types of Data Sources

 Structured Data
 - Databases, spreadsheets

 Semi-Structured Data
 - XML, JSON

 Unstructured Data
 - Text, images, videos
Challenges of Integrating Data from Multiple
Sources
 Data Variety
 - Different formats and structures

 Data Volume
 - Large amounts of data to process

 Data Quality and Consistency

 - Ensuring accuracy and reliability

 Data Security
 - Protecting data during integration
Data Integration Techniques - Batch Processing

 Definition and Process

 - Integrating data at scheduled intervals

 Use Cases
 - Periodic data consolidation
Data Integration Techniques - Real-Time
Processing
 Definition and Process
 - Continuous integration as data is generated

 Use Cases
 - Time-sensitive applications
Data Integration Techniques - Data
Virtualization
 Definition and Process
 - Accessing data without moving it

 Use Cases
 - Federated data access
Data Integration Techniques - ETL (Extract,
Transform, Load)
 Definition and Process
 - Standard method for integrating data

 Use Cases
 - Data warehousing
Tools and Technologies - Data Integration
Tools
 Apache NiFi
 - Data routing and transformation

 Talend
 - Data integration platform

 Informatica
 - Enterprise data integration
Tools and Technologies - Data Cleaning Tools

 OpenRefine
 - Tool for cleaning messy data

 Trifacta
 - Data preparation platform

 DataWrangler
 - Interactive tool for data cleaning
Tools and Technologies - Data Transformation
Tools
 Apache Spark
 - Unified analytics engine

 Pandas (Python Library)

 - Data manipulation and analysis

 Alteryx
 - Data blending and advanced analytics
Best Practices for Data Integration

 Ensuring Data Quality

 - Regular data validation and cleaning

 Maintaining Data Consistency

 - Using consistent data formats and standards

 Implementing Data Governance

 - Policies and procedures for managing data

 Data Security and Privacy

 - Protecting sensitive data during integration
Data Preprocessing: Numeric
and Non-Numeric
A comprehensive guide to preprocessing techniques in data science
Introduction to Data Preprocessing

 Importance of Data Preprocessing

 Types of Data: Numeric and Non-Numeric
Numeric Data Preprocessing

 Handling Missing Values

 Normalization and Standardization
 Binning
 Transformation Techniques
Handling Missing Values in Numeric Data

 Techniques:
 1. Mean/Median/Mode Imputation
 2. Forward/Backward Fill
 Example:
 Consider a dataset with missing values in the 'Age' column. Fill missing values using the
mean of the column.
Normalization and Standardization

 Definitions:
 Normalization: Scaling data to a range of [0, 1].
 Standardization: Scaling data to have a mean of 0 and a standard deviation of 1.
 Techniques:
 1. Min-Max Scaling
 2. Z-score Standardization
 Example:
 Consider normalizing the 'Income' column to the range [0, 1].
Binning

 Purpose of Binning: Grouping numeric data into bins to reduce noise and handle outliers.
 Techniques:
 1. Fixed-width Binning
 2. Quantile Binning
 Example:
 Bin the 'Age' column into intervals of 10 years each.
Transformation Techniques

 Transformations:
 1. Log Transformation: Applying log function to skewed data.
 2. Box-Cox Transformation: Stabilizing variance and making the data more normal
distribution-like.
 Example:
 Apply log transformation to the 'Income' column.
Non-Numeric Data Preprocessing

 Handling Missing Values

 Encoding Categorical Data
 Text Data Preprocessing
 Feature Extraction
Handling Missing Values in Non-Numeric Data

 Techniques:
 1. Imputation with Most Frequent Value
 2. Removing Rows/Columns
 Example:
 Fill missing values in the 'City' column with the most frequent city name.
Encoding Categorical Data

 Techniques:
 1. One-Hot Encoding: Creating binary columns for each category.
 2. Label Encoding: Converting categories to numerical labels.
 Example:
 One-hot encode the 'Gender' column.
Text Data Preprocessing

 Techniques:
 1. Tokenization: Splitting text into words or tokens.
 2. Stemming: Reducing words to their root form.
 3. Lemmatization: Reducing words to their base form.
 Example:
 Tokenize and lemmatize the 'Review' column.
Feature Extraction

 Techniques:
 1. Bag of Words: Creating a matrix of word counts.
 2. TF-IDF: Creating a matrix of term frequency-inverse document frequency.
 Example:
 Apply TF-IDF to the 'Review' column.
Tools and Libraries for Data Preprocessing

 Pandas: Data manipulation and analysis.

 NumPy: Numerical operations.
 Scikit-learn: Preprocessing module with various utilities.
Numeric Data Preprocessing
An overview of techniques and methods
Introduction

 Data preprocessing is a crucial step in data analysis and machine learning. It involves
transforming raw data into a clean and usable format. This presentation covers the main
techniques for preprocessing numeric data.
Handling Missing Values

 1. Remove missing values

 - Drop rows or columns with missing values.
 2. Impute missing values
 - Replace with mean, median, mode, or a constant value.
 - Use advanced techniques like KNN imputation or regression imputation.
Normalization

 Normalization scales numeric data to a standard range, typically [0, 1]. Common methods
include:
 1. Min-Max Scaling
 - Formula: (X - min(X)) / (max(X) - min(X))
 2. Z-Score Standardization
 - Formula: (X - mean(X)) / std(X)
Handling Outliers

 Outliers can distort statistical analyses and models. Techniques to handle outliers include:
 1. Removing outliers
 - Identify and drop outlier values.
 2. Transforming data
 - Use log, square root, or other transformations to reduce impact.
 3. Imputation
 - Replace outliers with mean, median, or a constant value.
Binning

 Binning involves grouping numeric values into bins or categories. This can help reduce the
impact of minor observation errors.
 1. Equal-width binning
 - Divide the range of data into intervals of equal size.
 2. Equal-frequency binning
 - Divide the data into intervals that contain approximately the same number of
observations.
Feature Engineering

 Feature engineering involves creating new features from existing data to improve model
performance.
 1. Polynomial features
 - Generate polynomial terms (e.g., X^2, X^3) from existing features.
 2. Interaction features
 - Create features that represent interactions between existing features (e.g., X1 * X2).
Conclusion

 Numeric data preprocessing is essential for improving the quality of data and the
performance of machine learning models. Techniques such as handling missing values,
normalization, handling outliers, binning, and feature engineering are crucial steps in this
process. Properly preprocessed data leads to more accurate and reliable analyses.
Numeric Data Preprocessing
An overview of techniques and methods
Introduction

 Data preprocessing is a crucial step in data analysis and machine learning. It involves
transforming raw data into a clean and usable format. This presentation covers the main
techniques for preprocessing numeric data.
Handling Missing Values

 1. Remove missing values

 - Drop rows or columns with missing values.
 2. Impute missing values
 - Replace with mean, median, mode, or a constant value.
 - Use advanced techniques like KNN imputation or regression imputation.
Normalization

 Normalization scales numeric data to a standard range, typically [0, 1]. Common methods
include:
 1. Min-Max Scaling
 - Formula: (X - min(X)) / (max(X) - min(X))
 2. Z-Score Standardization
 - Formula: (X - mean(X)) / std(X)
Handling Outliers

 Outliers can distort statistical analyses and models. Techniques to handle outliers include:
 1. Removing outliers
 - Identify and drop outlier values.
 2. Transforming data
 - Use log, square root, or other transformations to reduce impact.
 3. Imputation
 - Replace outliers with mean, median, or a constant value.
Binning

 Binning involves grouping numeric values into bins or categories. This can help reduce the
impact of minor observation errors.
 1. Equal-width binning
 - Divide the range of data into intervals of equal size.
 2. Equal-frequency binning
 - Divide the data into intervals that contain approximately the same number of
observations.
Feature Engineering

 Feature engineering involves creating new features from existing data to improve model
performance.
 1. Polynomial features
 - Generate polynomial terms (e.g., X^2, X^3) from existing features.
 2. Interaction features
 - Create features that represent interactions between existing features (e.g., X1 * X2).
Conclusion

 Numeric data preprocessing is essential for improving the quality of data and the
performance of machine learning models. Techniques such as handling missing values,
normalization, handling outliers, binning, and feature engineering are crucial steps in this
process. Properly preprocessed data leads to more accurate and reliable analyses.
Numeric Data Preprocessing
An overview of techniques and methods
Introduction

 Data preprocessing is a crucial step in data analysis and machine learning. It involves
transforming raw data into a clean and usable format. This presentation covers the main
techniques for preprocessing numeric data.
Handling Missing Values

 1. Remove missing values

 - Drop rows or columns with missing values.
 2. Impute missing values
 - Replace with mean, median, mode, or a constant value.
 - Use advanced techniques like KNN imputation or regression imputation.
Normalization

 Normalization scales numeric data to a standard range, typically [0, 1]. Common methods
include:
 1. Min-Max Scaling
 - Formula: (X - min(X)) / (max(X) - min(X))
 2. Z-Score Standardization
 - Formula: (X - mean(X)) / std(X)
Handling Outliers

 Outliers can distort statistical analyses and models. Techniques to handle outliers include:
 1. Removing outliers
 - Identify and drop outlier values.
 2. Transforming data
 - Use log, square root, or other transformations to reduce impact.
 3. Imputation
 - Replace outliers with mean, median, or a constant value.
Binning

 Binning involves grouping numeric values into bins or categories. This can help reduce the
impact of minor observation errors.
 1. Equal-width binning
 - Divide the range of data into intervals of equal size.
 2. Equal-frequency binning
 - Divide the data into intervals that contain approximately the same number of
observations.
Feature Engineering

 Feature engineering involves creating new features from existing data to improve model
performance.
 1. Polynomial features
 - Generate polynomial terms (e.g., X^2, X^3) from existing features.
 2. Interaction features
 - Create features that represent interactions between existing features (e.g., X1 * X2).
Conclusion

 Numeric data preprocessing is essential for improving the quality of data and the
performance of machine learning models. Techniques such as handling missing values,
normalization, handling outliers, binning, and feature engineering are crucial steps in this
process. Properly preprocessed data leads to more accurate and reliable analyses.
Numeric Data Preprocessing
An overview of techniques and methods
Introduction

 Data preprocessing is a crucial step in data analysis and machine learning. It involves
transforming raw data into a clean and usable format. This presentation covers the main
techniques for preprocessing numeric data.
Handling Missing Values

 1. Remove missing values

 - Drop rows or columns with missing values.
 2. Impute missing values
 - Replace with mean, median, mode, or a constant value.
 - Use advanced techniques like KNN imputation or regression imputation.
Normalization

 Normalization scales numeric data to a standard range, typically [0, 1]. Common methods
include:
 1. Min-Max Scaling
 - Formula: (X - min(X)) / (max(X) - min(X))
 2. Z-Score Standardization
 - Formula: (X - mean(X)) / std(X)
Handling Outliers

 Outliers can distort statistical analyses and models. Techniques to handle outliers include:
 1. Removing outliers
 - Identify and drop outlier values.
 2. Transforming data
 - Use log, square root, or other transformations to reduce impact.
 3. Imputation
 - Replace outliers with mean, median, or a constant value.
Binning

 Binning involves grouping numeric values into bins or categories. This can help reduce the
impact of minor observation errors.
 1. Equal-width binning
 - Divide the range of data into intervals of equal size.
 2. Equal-frequency binning
 - Divide the data into intervals that contain approximately the same number of
observations.
Feature Engineering

 Feature engineering involves creating new features from existing data to improve model
performance.
 1. Polynomial features
 - Generate polynomial terms (e.g., X^2, X^3) from existing features.
 2. Interaction features
 - Create features that represent interactions between existing features (e.g., X1 * X2).
Conclusion

 Numeric data preprocessing is essential for improving the quality of data and the
performance of machine learning models. Techniques such as handling missing values,
normalization, handling outliers, binning, and feature engineering are crucial steps in this
process. Properly preprocessed data leads to more accurate and reliable analyses.