Week 2 - LAQ's

Explain in detail about the steps in data preparation process?

Data preparation is the process of making raw data ready for after processing
and analysis. The key methods are to collect, clean, and label raw data in a format
suitable for machine learning (ML) algorithms, followed by data exploration and
visualization. The process of cleaning and combining raw data before using it for
machine learning and business analysis is known as data preparation, or sometimes
“pre-processing.” But it may not be the most attractive of duties, careful data
preparation is essential to the success of data analytics. Clear and important ideas from
raw data require careful validation, cleaning, and an addition. Any business analysis
or model created will only be as strong and validating as the very first information
preparation.

Data Preparation Process

There are a few important steps in the data preparation process, and each one is
essential to making sure the data is prepared for analysis or other processing. The
following are the key stages related to data preparation:

Step 1: Describe Purpose and Requirements

Identifying the goals and requirements for the data analysis project is the first step in
the data preparation process. Consider the followings:
 What is the goal of the data analysis project and how big is it?
 Which major inquiries or ideas are you planning to investigate or evaluate using
the data?
 Who are the target audience and end-users for the data analysis findings? What
positions and duties do they have?
 Which formats, types, and sources of data do you need to access and analyze?
 What requirements do you have for the data in terms of quality, accuracy,
completeness, timeliness, and relevance?

Pragalath EA2252001010013 1
 What are the limitations and ethical, legal, and regulatory issues that you must take
into account?
With answers to these questions, data analysis project’s goals, parameters, and
requirements simpler as well as highlighting any challenges, risks, or opportunities
that can develop.

Step 2: Data Collection

Collecting information from a variety of sources, including files, databases, websites,
and social media, to conduct a thorough analysis, providing the usage of reliable and
high-quality data. Suitable resources and methods are used to obtain and analyze data
from a variety of sources, including files, databases, APIs, and web scraping.

Step 3: Data Combining and Integrating Data

Data integration requires combining data from multiple sources or dimensions in order
to create a full, logical dataset. Data integration solutions provide a wide range of
operations, including combination, relationship, connection, difference, and join, as
well as a variety of data schemas and types of architecture.
To properly combine and integrate data, it is essential to store and arrange information
in a common standard format, such as CSV, JSON, or XML, for easy access and
uniform comprehension. Organizing data management and storage using solutions
such as cloud storage, data warehouses, or data lakes improves governance, maintains
consistency, and speeds up access to data on a single platform.
Audits, backups, recovery, verification, and encryption are all examples of strong
security procedures that can be used to make sure reliable data management. Privacy
protects data during transmission and storage, whereas authorization and
authentication.

Step 4: Data Profiling

Data profiling is a systematic method for assessing and analyzing a dataset, making
sure its quality, structure, content, and improving accuracy within an organizational
context. Data profiling identifies data consistency, differences, and null values by
analyzing source data, looking for errors, inconsistencies, and errors, and
understanding file structure, content, and relationships. It helps to evaluate elements
including completeness, accuracy, consistency, validity, and timeliness.

Pragalath EA2252001010013 2
Step 5: Data Exploring
Data exploration is getting familiar with data, identifying patterns, trends, outliers,
and errors in order to better understand it and evaluate the possibilities for analysis.
To evaluate data, identify data types, formats, and structures, and calculate descriptive
statistics such as mean, median, mode, and variance for each numerical variable.
Visualizations such as histograms, boxplots, and scatterplots can provide
understanding of data distribution, while complex techniques such as classification
can reveal hidden patterns and show exceptions.

Step 6: Data Transformations and Enrichment

Data enrichment is the process of improving a dataset by adding new features or
columns, enhancing its accuracy and reliability, and verifying it against third-party
sources.
 The technique involves combining various data sources like CRM, financial, and
marketing to create a comprehensive dataset, incorporating third-party data like
demographics for enhanced insights.
 The process involves categorizing data into groups like customers or products
based on shared attributes, using standard variables like age and gender to describe
these entities.
 Engineer new features or fields by utilizing existing data, such as calculating
customer age based on their birthdate. Estimate missing values from available data,
such as absent sales figures, by referencing historical trends.
 The task involves identifying entities like names and addresses within unstructured
text data, thereby extracting actionable information from text without a fixed
structure.
 The process involves assigning specific categories to unstructured text data, such
as product descriptions or customer feedback, to facilitate analysis and gain
valuable insights.
 Utilize various techniques like geocoding, sentiment analysis, entity recognition,
and topic modeling to enrich your data with additional information or context.
 To enable analysis and generate important insights, unstructured text data is
classified into different groups, such as product descriptions or consumer
feedback.

Pragalath EA2252001010013 3
Use cleaning procedures to remove or correct flaws or inconsistencies in your data,
such as duplicates, outliers, missing numbers, typos, and formatting difficulties.
Validation techniques like as checksums, rules, limitations, and tests are used to
ensure that data is correct and complete.

Step 8: Data Validation

Data validation is crucial for ensuring data accuracy, completeness, and consistency,
as it checks data against predefined rules and criteria that align with your
requirements, standards, and regulations.
 Analyze the data to better understand its properties, such as data kinds, ranges, and
distributions. Identify any potential issues, such as missing values, exceptions, or
errors.
 Choose a representative sample of the dataset for validation. This technique is
useful for larger datasets because it minimizes processing effort.
 Apply planned validation rules to the collected data. Rules may contain format
checks, range validations, or cross-field validations.
 Identify records that do not fulfill the validation standards. Keep track of any flaws
or discrepancies for future analysis.
 Correct identified mistakes by cleaning, converting, or entering data as needed.
Maintaining an audit record of modifications made during this procedure is critical.
 Automate data validation activities as much as feasible to ensure consistent and
ongoing data quality maintenance.

Tools for Data Preparation

The following section outlines various tools available for data preparation, essential
for addressing quality, consistency, and usability challenges in datasets.
1. Pandas: Pandas is a powerful Python library for data manipulation and analysis.
It provides data structures like DataFrames for efficient data handling and
manipulation. Pandas is widely used for cleaning, transforming, and exploring data
in Python.
2. Trifacta Wrangler: Trifacta Wrangler is a data preparation tool that offers a
visual and interactive interface for cleaning and structuring data. It supports

Pragalath EA2252001010013 4
 various data formats and can handle large datasets.
3. KNIME: KNIME (Konstanz Information Miner) is an open-source platform for
data analytics, reporting, and integration. It provides a visual interface for
designing data workflows and includes a variety of pre-built nodes for data
preparation tasks.
4. DataWrangler by Stanford: DataWrangler is a web-based tool developed by
Stanford that allows users to explore, clean, and transform data through a series of
interactive steps. It generates transformation scripts that can be applied to the
original data.
5. RapidMiner: RapidMiner is a data science platform that includes tools for data
preparation, machine learning, and model deployment. It offers a visual workflow
designer for creating and executing data preparation processes.
6. Apache Spark: Apache Spark is a distributed computing framework that includes
libraries for data processing, including Spark SQL and Spark DataFrame. It is
particularly useful for large-scale data preparation tasks.
7. Microsoft Excel: Excel is a widely used spreadsheet software that includes a
variety of data manipulation functions. While it may not be as sophisticated as
specialized tools, it is still a popular choice for smaller-scale data preparation tasks.

Challenges in Data Preparation

Now, we have already understood that data preparation is a critical stage in the
analytics process, yet it is fraught with numerous challenges like:
1. Lack of or insufficient data profiling:
 Leads to mistakes, errors, and difficulties in data preparation.
 Contributes to poor analytics findings.
 May result in missing or incomplete data.
2. Incomplete data:
 Missing values and other issues that must be addressed from the start.
 Can lead to inaccurate analysis if not handled properly.
3. Invalid values:
 Caused by spelling problems, typos, or incorrect number input.

Pragalath EA2252001010013 5
 Must be identified and corrected early on for analytical accuracy.
4. Lack of standardization in data sets:
 Name and address standardization is essential when combining data sets.
 Different formats and systems may impact how information is received.
5. Inconsistencies between enterprise systems:
 Arise due to differences in terminology, special identifiers, and other factors.
 Make data preparation difficult and may lead to errors in analysis.
6. Data enrichment challenges:
 Determining what additional information to add requires excellent skills and
business analytics knowledge.
7. Setting up, maintaining, and improving data preparation processes:
 Necessary to standardize processes and ensure they can be utilized repeatedly.
 Requires ongoing effort to optimize efficiency and effectiveness.

Pragalath EA2252001010013 6