Module 5

Understanding the
Data Science Methodology
Data Science
CRISP-DM Methodology
• CRISP-DM stands for The CRoss Industry Standard Process for Data Mining
• CRISP-DM is a process model with six phases that naturally describes the
data science life cycle. It’s like a set of guardrails to help you plan, organize,
and implement your data science (or machine learning) project.
• The process consists of the following steps:
• Business understanding – What does the business need?
• Data understanding – What data do we have / need? Is it clean?
• Data preparation – How do we organize the data for modeling?
• Modeling – What modeling techniques should we apply?
• Evaluation – Which model best meets the business objectives?
• Deployment – How do stakeholders access the results?
CRISP-DM Methodology Diagrams
Business Understanding The Business Understanding phase
focuses on understanding the objectives
and requirements of the project.

• Determine business objectives: You should first “thoroughly

understand, from a business perspective, what the customer really
wants to accomplish.” (CRISP-DM Guide) and then define business
success criteria.
• Assess situation: Determine resources availability, project
requirements, assess risks and contingencies, and conduct a cost-
benefit analysis.
• Determine data mining goals: In addition to defining the business
objectives, you should also define what success looks like from a
technical data mining perspective.
• Produce project plan: Select technologies and tools and define
detailed plans for each project phase.
Data Understanding
• Adding to the foundation of Business Understanding, it drives the focus
to identify, collect, and analyze the data sets that can help you
accomplish the project goals.
• Collect initial data: Acquire the necessary data and (if necessary) load it into
your analysis tool.
• Describe data: Examine the data and document its surface properties like data
format, number of records, or field identities.
• Explore data: Dig deeper into the data. Query it, visualize it, and identify
relationships among the data.
• Verify data quality: How clean/dirty is the data? Document any quality issues.
Data visualization can provide
insight that traditional descriptive
statistics cannot. A perfect
example of this is Anscombe’s
Quartet, created by Francis
Anscombe in 1973. The
illustration includes four different
datasets with almost identical
variance, mean, correlation
between X and Y coordinates, and
linear regression lines. However,
the patterns are clearly different
when plotted on a graph. Below,
you can see a linear regression
model would apply to graphs one
and three, but a polynomial
regression model would be ideal
for graph two. This illustration
highlights why it’s important to
visualize data and not just rely on
descriptive statistics.
7
Data Preparation This phase, which is often referred to as
“data munging”, prepares the final data
set(s) for modeling. It has five tasks:

• Select data: Determine which data sets will be used and document reasons
for inclusion/exclusion.
• Clean data: Often this is the lengthiest task. Without it, you’ll likely fall
victim to garbage-in, garbage-out. A common practice during this task is to
correct, impute, or remove erroneous values.
• Construct data: Derive new attributes that will be helpful. For example,
derive someone’s body mass index from height and weight fields.
• Integrate data: Create new data sets by combining data from multiple
sources.
• Format data: Re-format data as necessary. For example, you might convert
string values that store numbers to numeric values so that you can perform
mathematical operations.
Modeling Here you’ll likely build and assess
various models based on several
different modeling techniques.

• Select modeling techniques: Determine which algorithms to try (e.g.

regression, neural net).
• Generate test design: Pending your modeling approach, you might
need to split the data into training, test, and validation sets.
• Build model: As glamorous as this might sound, this might just be
executing a few lines of code like “reg = LinearRegression().fit(X, y)”.
• Assess model: Generally, multiple models are competing against each
other, and the data scientist needs to interpret the model results
based on domain knowledge, the pre-defined success criteria, and
the test design.
Evaluation
• Evaluate results: Do the models meet the business success criteria?
Which one(s) should we approve for the business?
• Review process: Review the work accomplished. Was anything
overlooked? Were all steps properly executed? Summarize findings
and correct anything if needed.
• Determine next steps: Based on the previous three tasks, determine
whether to proceed to deployment, iterate further, or initiate new
projects. Whereas the Assess Model task of
the Modeling phase focuses on technical
model assessment, the Evaluation phase
looks more broadly at which model best
meets the business and what to do next.
 A model is not particularly useful unless
Deployment the customer can access its results. The
complexity of this phase varies widely.

• Plan deployment: Develop and document a plan for deploying the

model.
• Plan monitoring and maintenance: Develop a thorough monitoring
and maintenance plan to avoid issues during the operational phase
(or post-project phase) of a model.
• Produce final report: The project team documents a summary of the
project which might include a final presentation of data mining
results.
• Review project: Conduct a project retrospective about what went
well, what could have been better, and how to improve in the future.
Beyond CRISP-DM
• Your organization’s work might not end there.
• As a project framework, CRISP-DM does not outline what to
do after the project (also known as “operations”).
• But if the model is going to production, be sure you maintain
the model in production. Constant monitoring and
occasional model tuning is often required.
Benefits
• Generalize-able: Although designed for data mining, William Vorhies, one of the creators of CRISP-DM, argues that
because all data science projects start with business understanding, have data that must be gathered and cleaned,
and apply data science algorithms, “CRISP-DM provides strong guidance for even the most advanced of today’s
data science activities” (Vorhies, 2016).
• Common Sense: When students were asked to do a data science project without project management direction,
they “tended toward a CRISP-like methodology and identified the phases and did several iterations.” Moreover,
teams which were trained and explicitly told to implement CRISP-DM performed better than teams using other
approaches (Saltz, Shamshurin, & Crowston, 2017).
• Adopt-able: Like Kanban, CRISP-DM can be implemented without much training, organizational role changes, or
controversy.
• Right Start: The initial focus on Business Understanding is helpful to align technical work with business needs and
to steer data scientists away from jumping into a problem without properly understanding business objectives.
• Strong Finish: Its final step Deployment likewise addresses important considerations to close out the project and
transition to maintenance and operations.
• Flexible: A loose CRISP-DM implementation can be flexible to provide many of the benefits of agile principles and
practices. By accepting that a project starts with significant unknowns, the user can cycle through steps, each time
gaining a deeper understanding of the data and the problem. The empirical knowledge learned from previous
cycles can then feed into the following cycles.
Weaknesses & Challenges
• Rigid: On the other hand, some argue that CRISP-DM suffers from the same weaknesses
of Waterfall and encumbers rapid iteration.
• Documentation Heavy: Nearly every task has a documentation step. While documenting
one’s work is key in a mature process, CRISP-DM’s documentation requirements might
unnecessarily slow the team from actually delivering increments.
• Not Modern: Counter to Vorheis’ argument for the sustaining relevance of CRISP-DM,
others argue that CRISP-DM, as a process that pre-dates big data, “might not be suitable
for Big Data projects due its four V’s” (Saltz & Shamshurin, 2016).
• Not a Project Management Approach: Perhaps most significantly, CRISP-DM is not a true
project management methodology because it implicitly assumes that its user is a single
person or small, tight-knit team and ignores the teamwork coordination necessary for
larger projects (Saltz, Shamshurin, & Connors, 2017).
 KDnuggets Polls
• KDnuggets is a common
source for data mining
methodology usage. Each
of the polls
in 2002, 2004, 2007 posed
the question: “What main
methodology are you using
for data mining?”, and
the 2014 poll expanded the
question to include “…for
analytics, data mining, or
data science projects.” 150-
200 respondents answered
each poll.
 2020 Poll
• For a more current look into
the popularity of various
approaches, we conducted
our own poll on this site in
August and September 2020.
• Note the response options
for our poll were different
from the KDnuggets polls
and our site attracts a
different audience.
• CRISP-DM was the clear
winner, garnering nearly half
of the 109 votes.
Recommendations
• Iterate quickly: Don’t fall into a waterfall trap by working thoroughly across layers of the project.
Rather, think vertically and deliver thin vertical slices of end-to-end value. Your first deliverable
might not be too useful. That’s okay. Iterate.
• Document enough…but not too much: If you follow CRISP-DM precisely, you might spend more
time documenting than doing anything else. Do what’s reasonable and appropriate but don’t go
overboard.
• Don’t forgot modern technology: Add steps to leverage cloud architectures and modern software
practices like git version control and CI/CD pipelines to your project plan when appropriate.
• Set expectations: CRISP-DM lacks communication strategies with stakeholders. So be sure to set
expectations and communicate with them frequently.
• Combine with a project management approach: As a more generalized statement from the
previous bullet, CRISP-DM is not truly a project management approach. Thus combine it with
a data science coordination framework. Popular agile approaches include:
• Kanban
• Scrum
• Data Driven Scrum