A Simple Framework for Building

Predictive Models

A Little Data Science Business Guide

Authored by:
Hal Kalechofsky, Ph.D.

September 2016
 Table of Contents

1. Introduction............................................................................................................................................3

1.1 Working Definition of Predictive Models....................................................................4

1.2 Problem Statement.............................................................................................................4

1.3 Goal of this Publication.....................................................................................................4

2. Audience..................................................................................................................................................5

3. Business Usage of Predictive Models............................................................................................5

3.1 Use Cases................................................................................................................................6

4. Modelling Framework.........................................................................................................................6

4.1 Recommended Predictive Modelling Business Requirements...........................6

4.2 The Modelling Process.......................................................................................................7

4.2.1 Plan the model...........................................................................................................7

4.2.2 Build the model.........................................................................................................7

4.2.3 Implement the model.............................................................................................8

4.3 Modelling Longevity and Considerations...................................................................8

4.3.1 Types of Predictive Models....................................................................................9

4.4 Supervised and Unsupervised Learning................................................................... 10

4.5 Practical Considerations When Working with the Data....................................... 10

5. Cognitive Architecture..................................................................................................................... 13

5.1 Data Management Strategy.......................................................................................... 13

5.2 Basic Model Architecture............................................................................................... 14

6. How Do You Know Your Model is Successful?......................................................................... 14

6.1 Statistical Uplift.................................................................................................................. 14

6.2 Business Success Measures........................................................................................... 15

7. Appendix A: References.................................................................................................................. 16

8. About the Author.............................................................................................................................. 17

A Simple Framework for Building Predictive Models | 2

1. Introduction
“Statistics are like a bikini: What they reveal is interesting, but what they conceal is vital.”
Most humans seem to want to predict the future. It is a natural human desire. To know in advance
about the weather, the stock market, the next card coming up in blackjack - what a power to
have over the natural world!
After decades of research and
development, computer science and
information technologies are now
reaching a point where predictive models
are an important, if not indispensable,
part of the business ecosystem. Many
technologies, including fast computing
power, inexpensive storage, cloud
computing, voice recognition, mobile
devices, artificial intelligence (cognitive
computing), and advanced application
software are combining to make this
possible. The world is generating vast
amounts of data. To cite just one example,
the amount of data that will be generated
by the Internet of Things (IoT) in the years
to come will dwarf what we today call
“Big Data.” Recent trends are allowing
predictive analytics and models to be democratized and spread to smaller organizations and
individual users, avoiding the need for large software budgets or armies of data scientists to
create and analyze the insights generated.
Business leaders in all industries will want to not only be aware of the data science forces shaping
our future economy, but also to be well versed in how best to use and to make the most of these
coming opportunities.
As human beings, unlike most other animals, we have the power to shape our environment.
One main way of doing this is to assess situations based on data and evidence and then plan
and strategize for the future. One might go so far as to say this is an evolutionary necessity for
survival, or at least for improved chances for survival. It is interesting how our tools (of which
information technology is one powerful example) reflect what we need (or think we need)
to survive throughout history. Just think back through the wheel, fire, steel, weapons, travel,
medicine and information technology.
Viewed in this way, predictive computing models are an extension of our natural survival
instinct. In this data-intensive world, predictive models are more important than ever in order
to make sense out of what is around us and to estimate, assess, or plan for what might happen
in the future.

A Simple Framework for Building Predictive Models | 3

1.1 Working Definition of Predictive Models • Risk Management
“It’s hard to make predictions, especially about the - How can I identify fraud or increase trust in
future.” financial transactions?
- Which business scenario is the likeliest to
A short definition of a predictive model is: win out?
• Using data to make decisions. - How can I build and maintain advantage
A longer definition might be: over my competitors?
• Operations
• Using data to make decisions and to take actions - How can I save money, resources and time
using models that are empirically derived and by anticipating when and how physical
statistically valid. assets are likely to break down, and
Predictive modelling is a commonly used statistical preventing it?
technique to predict future behavior. Predictive • Customer Relationships
modelling solutions are a form of data-mining - What will my customers want in the future?
technology that works by analyzing historical and - How can I solve customer problems before
current data and generating a model to help predict they happen?
future outcomes. It is sometimes said that decisions made on wrong data
Predictive models are created whenever data is used are better than decisions made on no data at all. This is
to train a predictive modelling technique. To put it because even if the data are wrong, at least one can use
another way: data + predictive modelling technique = human skepticism and questioning techniques to learn
predictive model. from it. Without data, this is not possible.
A predictive model is the result of combining data and Business leaders often use intuition, a “gut feeling” or
mathematics, where learning can be translated into the revenue streams to forecast future market conditions.
creation of a mapping function between a set of input Sometimes they are right, sometimes not. This could
data fields and a response or target variable. be called an example of an emotional or gut-level
predictive model. Given all the data we have gathered
1.2 Problem Statement in our high-tech world, it can be useful to supplement
“I know that half my marketing spend works ... I just this instinct with predictable information so that one
don’t know which half.” can evaluate the decisions to make.

Business metrics do a great job at summarizing the 1.3 Goal of this Publication
past. However, if you want to predict how customers
will respond in the future, there is one place to turn — The goal of this paper is to present a simple framework
predictive analytics. for developing predictive or statistical models for
modern business purposes.
There are many business reasons why it is important to
make use of predictive models. Any time you would like Modern means the first few decades of the 21st
to know something about the future, it is useful to have century, with all of our high-tech measurements and
a predictive model. Although an educated guess is fine, computational apparatus brought to bear. What we
imagine being right more than half the time! Imagine were able to do with predictive modelling a generation
being right most of the time or all of the time. or more ago and what we can do at the present time
may share mathematical techniques, yet are different in
Here are just a few reasons to consider using predictive their historical anchor points.
analytics and models in your business: This is not intended to be an academic or university
• Future Potential Revenue or Cost Drivers research type of paper; this paper is presenting a
- How can I know my product is right for framework, not an exposition of deep data science
this market? techniques. In the modern world, we have educated
- What is the best way to continuously lower many competent analysts and data scientists and have
my cost of goods sold? built many software systems to apply the necessary

A Simple Framework for Building Predictive Models | 4

predictive modelling techniques, so it is not needed to
cover that here.
This paper is meant to be a primer, not a detailed essay.
It does not instruct people on how to build models,
but covers the steps involved and the practical issues
to consider. This paper is designed to be relatively short
and to deliver information efficiently in a short amount
of the reader’s time.

2. Audience
This paper is written for a variety of audiences. The
reader should be familiar with modern computer
systems and have some level of curiosity. Many
business people these days are well versed in technology
and technical reasoning. Trained statisticians and data
scientists who are practitioners of predictive models
may know the theory and the mathematics, but may
get something out of the business discussions.
A list of role/resource types that may benefit from this
paper are listed below:
• Project Leaders: who desire to have a more
detailed understanding of predictive modelling
methods and techniques to better manage and
interact with their practitioners
• Business Analysts: who must develop and 3. Business Usage of Predictive Models
interpret the models, communicate the results
“If you predict it, you own it.”
and make actionable recommendations
• Big Data Analysts: who are under increasing This section describes some of the business usages of
pressure to transform their deluge of data from predictive analytics and predictive models.
a liability to an asset Predictive analytics encompasses a variety of statistical
• Functional Analytic Users: Customer Relationship techniques from predictive modelling, machine
Managers, Risk Analysts, Business Forecasters, learning, and data mining that analyze current and
Statistical Analysts, Social Media and Web Data historical facts to make predictions about future or
Analysts, Fraud Detection Analysts, Audit otherwise unknown events. When most lay people
Selection Managers, Direct Marketing Analysts, discuss predictive analytics, they are usually discussing
Medical Diagnostic Analysts, Market Timers it in terms of predictive modelling. Indeed, predictive
• Data Scientists: who desire to extend the modelling is at the heart of predictive analytics.
scientist aspect of the role with formal process
Predictive analytics is used in marketing, ecommerce,
and hands-on methodological practice
financial services, actuarial science and insurance,
• IT Professionals: who wish to gain a better
telecommunications, internet security, retail,
understanding of the data preparation, analytics
travel, medical and healthcare, child protection,
and analytic sandbox development requirements
pharmaceuticals, capacity planning, supply chain, and
to more fully support the growing demand for
other fields.
analytic IT support
• Anyone overwhelmed with data and starved for
actionable insights

A Simple Framework for Building Predictive Models | 5

3.1. Use Cases 4. Modelling Framework
The table below lists predictive analytics business “The best way to predict the future is to create it.”
applications. The first column names the type of
business benefit, and the second column identifies the 4.1 Recommended Predictive Modelling
type of customer prediction required – that is, which Business Requirements
behavior or action must be predicted to undertake
While they may share many characteristics and
each business application. As there are many such
techniques, building statistical or predictive models
applications, this list includes only the most pervasive
for a scientific or university research context can be
in commercial deployment to date.
different than a similar exercise in a business context.
Reference or For business-oriented projects, here are some basic
Business What is
Company core modelling requirements and considerations.
Application Predicted
Case Study
Customer Reed Elsevier,
• Formulate a clear and transparent framework:
Customer retention defection/churn/ and Telenor, - What are the objectives? What are we
attrition Published article modelling for?
How to sell - Predictive models alone do not create
eCommerce Amazon.com
successfully online business value, but rather need to be
Direct marketing Customer response
Charles Schwab effectively deployed either into a decision-
Published article making or business process.
Netflix Prize • Make room for using business knowledge in the
Product Customer
leader, HSBC &
recommendations wants/likes process:
Amazon.com
- Involve business users in the modelling
Google, Yahoo!
Which ad customer and Click process to ensure clear requirements,
Behavior-based communication and purpose verification.
will Forensics,
advertising
click “$1 million” - Business experience and heuristics may
case study contain valuable nuggets for how to use the
Message that will data.
World Wildlife
Email targeting generate customer
response
Fund - Do solutions pass the “Does it make
business sense” test?
Credit scoring Debtor risk or fraud Wells Fargo • Make sure that data are appropriate for
Fundraising for modelling:
Donation amount NRA - Quality, scope and quantity suitable to meet
nonprofits
Insurance objectives
Insurance pricing Applicant response,
and selection insured risk
Journal, Pinnacol - Structured in line with underlying processes
Assurance being modeled to maximize the signal
• Don’t just make best statistical fits to historical
There are many more applications of predictive analytics, data:
including collections, supply chain optimization, human - Test predictive power and consistency of
resource decision support for recruitment and human patterns over time and across validation
capital retention, and market research survey analysis. samples
The way predictive scores help your business depends • Think about managing the model:
on the customer behavior predicted – just aim predictive - How to integrate the model into existing
analytics towards the right customer prediction business systems
goal and fire away. This is why it is sometimes said, - After implementing, how to track and
“If you predict it, you own it!” manage the interactions, and how to
maintain, tweak, or update the model
going forward in an efficient manner

A Simple Framework for Building Predictive Models | 6

4.2 The Modelling Process Scope
“The devil is in the details.” A clear specific objective for the model is required.
Let’s take a look at the process of predictive modelling. Each model is developed for its own specific purpose
There is a wealth of information on the Web about this. and cannot be used effectively in another situation.
However, to understand the strategic areas, it is useful For example a model that predicts ecommerce online
to break down the process of prediction into just a few customer churn cannot be used to predict credit card
essential components. churn. An example of a clearly defined model objective
contains the event or action that the model is to predict
The below diagram is a good depiction of the predictive and the period when it is likely to happen. For example,
modelling process across a wide range of businesses. the objective could be to predict customers that are
likely to miss a credit card payment within the next
month.

Prepare the Dataset

The dataset created to use for the model might broadly
cover diverse information, such as product, case,
behavioral, demographic, geographic, competitor
details or weather. The variables that are not included in
the dataset will not form part of the prediction. Variables
cover both static fields such as income and triggers
such as change in spend. Both technical and business
people need to be involved in the decisions relating
to the contents of the dataset. Focus needs to be on
the behavioral information as this is more powerful for
predictions than demographic data. It is useful to give
consideration to which customers to exclude from the
model build process.
Customers need to be excluded if they are going
to impair the performance of the model. Potential
exclusions include things like bad debt, staff and new
customers as they have insufficient history. The actual
Broadly speaking, predictive analysis and modelling can exclusions applied relates to the specific purpose of the
be divided into three parts: Plan, Build, and Implement. model. For example, if the model is to predict customers
that are likely to carry bad debts, bad debt customers
4.2.1 Plan the Model would be included in the model. In practice, exclusions
are applied as filters when creating the dataset and
The next step includes scoping and planning. In the should be noted down.
Planning section, the sub-phase activities here are
called Scope and Prepare. This part of the exercise 4.2.2 Build the Model
might take around 40% of the total time.
Here you will write model code, build the model,
To build a predictive model, you first need to assemble calculate scores, and validate the data. In the Build
the datasets that will be used for training. You must section, the Sub-phases are known as Model and
formulate clear objectives, cleanse and organize the Validate. This might take around 20% of the overall time.
data, perform data treatment including missing values This is the part that can be left with the data scientists
and outlier fixing, make a descriptive analysis of the or technical analysts. The technical aspects of building
data with statistical distributions, and create data sets models are not covered in this paper.
used for the model-building.

A Simple Framework for Building Predictive Models | 7

The model will be built using a sample from the data set Deploy the Model
created. The resulting model will contain a subset of the It’s time to apply the model. The model will be built
original list of variables considered for the model. This is on a subset of data. Once the model is complete and
acceptable and happens because some of the variables has been validated, it will be run over the customer,
considered for the model will be correlated with each product, or case base.
other, for example, a product type and product family,
or the floor number and height of building. Other Assess
variables will have been discarded as they add little or
nothing to the model’s predictive power. It is typical to generate rankings from your model
– depending what you are modelling – customers
Calculate a Score or products. You will want to understand the
performance of your model. The ranking scores allow
The model developed will be an equation that, when a customer or product base to be ranked in order of
applied to the customer base, will allocate a score the predicted score, such as from most likely to least
to each customer. The score represents a customer’s likely to churn or from most popular product to least. In
likelihood ‘to do’ whatever the model is predicting, reality, some customers will churn and some customers
such as predict a customer’s expected order value or will not. Therefore, in absolute terms, these predictions
likelihood to churn within a month. If you are building will not be accurate. The ranked list provides a superb
a model for product failures, you may have a score that base upon which to vary the treatment, and therefore
calculates the likelihood of a product to fail in a given the level of service or marketing spend, to groups of
time frame. customers.

Validate the Model Monitor

Typically models are validated against a hold out group. You want to consider how often to run the model across
This group contains customers that have not been the base and generate scores. You will want to monitor
included in the development of the model. As such, the performance of the model on an ongoing basis, and
they represent a group of previously unseen perhaps take advantage of new data or techniques as
customers that are representative of the customer they become available.
base. To achieve an accurate prediction of lift, the hold
out group must not be made up of the customers 4.3 Modelling Longevity and Considerations
that have been excluded from the model “The simplest solution is the best.”
development process. The same idea applies if
you are building a product failure score model. It is useful to think of the deployment of predictive
models in a continuous improvement scenario. You
4.2.3 Implement the Model don’t just build the model and leave it alone. The model
may take nourishment, watering, monitoring, and care
Here you will deploy and apply the model, rank and can improve over time as conditions change and
customers or products, use the model outcomes for new data come in.
some business purpose, estimate model performance, A model does not have to be extremely complicated
assess and monitor the model, and drive initiatives in order to make good predictions. Like in coding,
based on the model. The Sub-phases here are known simplicity is a good guideline. It is useful to keep
as Deploy, Assess, and Monitor. Occam’s Razor in mind when considering how to build a
You need to think about accessing, storing, and using model: Some people say Occam’s Razor is “the simplest
the data. This might take around 40% of the overall solution is the best.” However, if you look at the original
time, require IT department work, and can be an text, it says “Do not multiply entities beyond necessity.”
ongoing operational exercise when the model is used Generally, the analyst may make assumptions when
continuously for the business. building a model. However, nature does not assume
anything before forcing an event to occur. The fewer
assumptions there are in a predictive model, the greater

A Simple Framework for Building Predictive Models | 8

will be the predictive power. Clearly, attributes that are values, where the class labels are drawn from some
not in the model will have no effect on the model’s finite set.
predictions.
The speed of changing business conditions is a factor in 4. Linear Regression
how well the model will perform over time. For example, In statistics, regression analysis is a statistical process
information technology is a faster-moving industry than for estimating the relationships among variables. Linear
insurance. Once a model is in use and driving actions, regression is an approach for modelling the relationship
there is a requirement for tracking and managing the between a scalar dependent variable Y and one or
interactions. It is a good idea to refresh model scores more explanatory variables (or independent variables),
over time, sometimes frequently depending on the denoted X. The case of one explanatory variable is
industry. called simple linear regression. More than one variable
Control files will be required to test initiatives and to is called multivariate.
test the model performance. The results from this will
help determine how frequently the model needs to
5. Logistic Regression
be refreshed. As a rule of thumb, a model needs to be In statistics, logistic regression, or logit regression, or
reviewed and possibly rebuilt annually. logit model is a regression model where the dependent
variable (DV) is categorical or binary.
4.3.1 Types of Predictive Models
“There are three types of lies -- lies, damn lies, and 6. Neural Networks (NNs)
statistics.” An Artificial Neural Network (ANN) is an information
processing paradigm that is inspired by the way
Many types of predictive models have been developed
biological nervous systems, such as the brain, process
over the years that are useful for different classes of
information.
problems. Applying these techniques is the domain of
statisticians and data scientists, and is intentionally not
7. Machine Learning
covered in this short paper.
Machine learning is a type of artificial intelligence (AI)
1. Business Rules that provides computers with the ability to learn without
A business rule is a rule that defines or constrains some being explicitly programmed. Machine learning focuses
aspect of business and always resolves to either true on the development of computer programs that can
or false. Business rules are intended to assert business teach themselves to grow and change when exposed
structure or to control or influence the behavior of the to new data.
business.
8. Support Vector Machines (SVMs)
2. Classification and Decision Trees In Machine Learning, an SVM is a supervised learning
A decision tree is a decision support tool that uses a model with associated learning algorithms that analyze
tree-like graph or model of decisions and their possible data used for classification and regression analysis. An
consequences, including chance event outcomes, SVM maps input data vectors into a higher dimensional
resource costs, and utility. It is one way to display an space, where an “optimal hyperplane” that separates
algorithm. the data is constructed. An SVM is a discriminative
classifier formally defined by a separating hyperplane.
3. Naive Bayes In other words, given labeled training data (supervised
learning), the algorithm outputs an optimal hyperplane
In machine learning, naive Bayes classifiers are a family which categorizes new examples.
of simple probabilistic classifiers based on applying
Bayes’ theorem with strong (naive) independence 9. Natural Language Processing (NLP)
assumptions between the features. The technique
Since NLP (content analytics) feeds predictive and
constructs classifiers: models that assign class labels to
prescriptive analytics, it is included here. NLP fact
problem instances, represented as vectors of feature
A Simple Framework for Building Predictive Models | 9
extraction can be used with descriptive statistical 4.5 Practical Considerations When Working with
formulae to make use of the wealth of unstructured the Data
data. “Correlation does not imply causality.”

4.4 Supervised and Unsupervised Learning 1) A

sking the Right Questions
“Statistics can be made to prove anything - even the What are the business objectives of the project? Does it
truth.” make business sense?
SVMs, decision trees, NNs and regression models use Awash in reams of data, it is critical that companies
supervised learning to create the mapping function ask the right questions. Being specific is important.
between a set of input data fields and a target variable. Predictive analytics is most efficient when used to
The known outcome is then used as a teacher who determine the answer to a narrow inquiry, such as the
supervises the learning of her pupil. Whenever the pupil likelihood of customer A to buy product X at time Y for
makes a mistake, the teacher provides her with the right price Z, rather than the likelihood of customers buying
answer in the hopes that the pupil will eventually get it product X.
right. For instance, when presented with a specific set of
inputs, her output will match the target. 2) I dentify Available Data
Unsupervised learning requires no teacher or target. It is one thing to have someone say, “Build me a
Clustering techniques fall into this category. Data points predictive model to mint money.” Is that realistic? If the
are simply grouped together based on their similarity. In goal looks realistic, one must find out if there are even
an ecommerce website analysis, online shoppers might data available to do something reasonable. Sometimes
be grouped into window shoppers or power buyers. In the data will not be accessible, or it will not be ordered
case of customer churn, a clustering technique could or structured well enough.
potentially assign different clusters to churners and
non-churners even though the outcome is not available 3) Data Accuracy and Cleanup
during model training.
Data scientists must be aware that not all data is
Black-box is a term used to identify certain predictive accurate, arrive at an estimate of bad data, and correct
modelling techniques that are not capable of explaining for it in their studies. Data can be bad for any number
their reasoning because you can’t really know what of reasons, including self-reporting errors, corrupted
is happening in the Black-box. Although extremely files, poorly phrased questions, incomplete data
powerful, techniques such as NNs and SVMs fall into aggregation, and poor standardization methods.
this category. Very often, an explanation or reason for
the model decision is required; for example, when a risk It is critical that data scientists quickly recognize and
score is used to decline a loan application or a credit filter bad data from their data sets. They must also
card transaction. make sure they do not create bad data themselves,
i.e., through an imperfectly calculated transformation
Whenever explaining is a must, you need to consider function. Since bad data can play through the model-
using a predictive modelling technique that clearly building process, you have to have a feel for what will
describes the reasons for its decisions. Scorecards fit cause problems and what you can accept. In general, it
such criteria very well. Based on regression models, is useful to have procedures in place to both filter out
scorecards are a popular technique used by financial bad data and to prohibit data pollution in the future.
institutions to assess risk. With scorecards, all data fields
in an input record are associated with specific reason 4) What Assumptions are Being Made?
codes. Thus, it is possible to explain to the customer
Generally speaking, the more assumptions one makes,
why a given decision was rendered.
the less predictive the model may be. For example, if
you assume that every customer is from an urban area,

A Simple Framework for Building Predictive Models | 10

what will that tell you about people who live in the • Are the data symmetric in any way?
countryside? • Are there upper or lower limits on the data?
Big Data is messy, consisting of everything from social • What is the likelihood of observing extreme values in
media mentions to traffic camera images to website the distribution?
logs. Predictive analytics, being a set of statistical
techniques, requires all data to be standardized and 7) Determine Predictors
quantified. Quantifying non-numeric data has its own It is common sense that one should identify as many of
risks and creates uncertainty. the main predictive variables as possible. It is important
Further, data is unpredictable, especially dynamic data. to distinguish dependent and independent variables,
A model that accurately forecasts future events could as well as correlations between variables that could
be thrown into disarray by a sudden unanticipated skew model results. Additionally, it is good to find if
cascade of events, which were not initially estimated. In there is an identifiable signal to noise ratio in the data.
2007, the failure the majority of financial services firms For example, if you are trying to predict incomes in a
to incorporate the possibility of sudden credit defaults demographic population, geolocation and job type
triggered a series of other events that prior to 2007 may be important predictors.
would have been improbable.
8) W
hat Model to Select and Why
5) P
rivacy and Security There are many statistical, data mining, and machine-
It is always crucial to respect customer data privacy and learning algorithms available for use in your predictive
data security. Predictive models bear the added weight analysis model, often available in enterprise analytics
of collecting and using information about individuals or open-source software. Once you have defined the
in order to predict future behavior. Some privacy objectives of your model and selected the data you’ll
advocates find such data usage invasive and alarming. work on, you should be in a good position to choose
Many people feel there is intrusive about firms collecting which algorithms might apply best.
information about individuals in order to predict Here are a few general rules of thumb to decide which
their behavior. Advocacy efforts include lobbying algorithms can address various business concerns.
for limitations to data collection types, amounts and • For customer segmentation and/or community
methods in nations across the globe. Executives and detection in the social sphere, you will need
data managers must be aware of the ever-changing Big clustering algorithms.
Data regulatory landscape.
• For customer retention or to develop a
Privacy is a huge concern for another reason – security. recommender system, you’d use classification
Hackers may target companies for financial gain. If algorithms.
hackers can glean customer or product information,
• You can use decision trees when you have a
this can lead to data compliance issues, customer dis-
linear decision boundary, for example,
satisfaction, or unfavorable press.
classifying people on the basis of their IQ.
Knowing your company’s data privacy and data security
• For credit scoring or predicting the next outcome
policies in advance is a big help when doing predictive
of time-driven events, you’d use a regression
analysis of any type.
algorithm. Use regression when you want to
6) P
re-model Exploration – Statistical Distributions predict continuous values, instead of classifying.
Regression can be used for traffic prediction, for
This typically involves data mining: univariate, bivariate, instance.
sets, graphs, and so forth. There are generally four basic
questions that can help in the characterization of the • You can use the Naive Bayes classifier when the
data and constructing data distributions: features are conditionally independent. For
example, it has been used for really simplistic
• Are the data discrete values or whether the data is object recognition in RGB where the three
continuous? channels were assumed to be uncorrelated.

A Simple Framework for Building Predictive Models | 11

• Machine Learning (ML) is not a solution for every operate on the same data. An ensemble model uses a
type of problem. There are certain cases where predefined mechanism to gather outcomes from all its
robust solutions can be developed without using component models and provide a final outcome for the
ML techniques. For example, you don’t need ML user.
if you can determine a target value by using simple
rules, computations, or predetermined steps that 9) D
eployment Path – Integrated Architecture –
can be programmed without needing any data- Analytic Infrastructure
driven learning. Consider the outputs of your model and how this
- Machine learning can be used for the following will be incorporated into the enterprise or business
situations: infrastructure. It is very advisable to be familiar with the
• It is hard to code the rules or there are data systems architecture and how the model will be
many rule factors: Many human tasks managed over time.
(such as recognizing whether an email is
spam or not spam) cannot be adequately
10) P
erformance - Test , Train, Validate
solved using a simple (deterministic), rule- Validating the predictive model you have built is
based solution. critical. You can generate scores on a test sample
and generalize to a whole similar population, bearing
• The solution cannot scale. You might be
in mind that there will be statistical confidence factors
able to manually recognize a few hundred
to consider.
emails and decide whether they are spam
or not. However, this task becomes Broadly speaking, one can talk about several types of
tedious for millions of emails. ML solutions validation:
are effective at handling large-scale • Apparent: performance on sample used to develop
problems. model
• Machine learning is a great approach for • Internal: performance on population underlying the
many text classification problems, like the sample
email spam case above. However, with
• External: performance on related but slightly
other problems, such as classifying a job
different population
title into a rank, there may be an approach
difficulty in that, unless the training set Good modelers and data scientists are able to explain
is very large and sufficiently diverse, a in detail how the model was validated and can show
machinelearning solution can significantly the statistical performance of the model (Regression
overfit it. The term “overfit” means “the coefficients, chi squares, significance of functions).
learned model does not work adequately Analysis of variance (ANOVA) is often used for this
well on titles not seen during training”. In purpose. ANOVA is a collection of statistical models
other words, it is crucial to measure how used to analyze the differences among group means
the model performs on cases not seen and their associated procedures, such as variation
during the training phase. among and between groups. Below is a quick
overview of the main regression calculations for
As time and resources permit, it is okay to run several
performance purposes.
algorithms of the appropriate type. Comparing different
runs of different algorithms can bring surprising findings
R Square:
about the data or the business intelligence embedded
in the data. Doing so gives you more detailed insight R Square describes the goodness of the fit in terms
into the business problem, and helps you identify which of the independent variables. For example, 96% of
variables within your data have predictive power. the variation in Quantity Sold is explained by the
independent variables Price and Advertising. The closer
Some predictive analytics projects succeed best by
to 1, the better the regression line fits the data.
building an ensemble model, a group of models that

A Simple Framework for Building Predictive Models | 12

Significance F: 5.1 Data Management Strategy
To check if your results are reliable (statistically Building a model by itself is a good exercise. However,
significant), look at Significance F (0.001). If this value the real power comes from when the predictions
is less than 0.05, you’re okay. If Significance F is greater are actually used for business decisions or customer
than 0.05, it’s probably better to stop using this set of touchpoints. As data drives everything in business today,
independent variables. Most or all P values should be you can’t be successful without a data management
below 0.05. Delete a variable with a high P-value (greater strategy. There are five key areas to a successful data
than 0.05) and rerun the regression until Significance F management strategy:
drops below 0.05. • Data quality
- How to keep your data clean, accurate, and
Coefficients: correct
The regression line might take the form: Y = Intercept • Data integration
+- A*X1 +- B*X2. This is an equation in some number - How to integrate your data systems together
of variables; Where Y is the dependent variable, and X1 for optimal usage
and X2 are independent variables. A and B are called • Data federation (including access)
the coefficients. These describe the strength (increase, - How to provide secure, federated access
decrease) in the variable relationships. To use a and tenancy to the data
marketing/advertising example: For each unit increase • Data governance
in price, Quantity Sold might decrease by with some - Having understood policies and procedures
number of units. For each unit increase in Advertising, for data storage and usage
Quantity Sold increases by some number of units. This is • Master data management
valuable information. You can also use these coefficients - How to efficiently warehouse and manage
to do a forecast. For example, if you know the price and your data
Advertising dollars, you could tell how much Quantity In particular, when using predictive models in a
Sold you could achieve. corporate environment, one must consider a data
architecture that allows the model to be transactionally
Residuals: implemented and used and tuned, configured, or
The residuals show how far away the actual data points updated over time. There should be simple plug-ins
are from the predicted data points (using the equation). or API hooks that can use the predictive model output
in a technology application. Enabling easy A/B testing
5. Cognitive Architecture in a live environment, for example, can be a powerful
“I never predict anything, and I never will.” technique.
Today’s enterprises have many touchpoints It should be said that many enterprises are not really
where predictive analytics and modelling can be prepared for this kind of big data architecture, even
brought to bear. It is increasingly important in if they have modern data warehouses or business
today’s data-intensive world that the business’ intelligence systems in place. The reason for this is
system architecture supports predictive modelling that big data involves not only data volumes, but
development, processes, and outcomes. The term also data velocity and data variety. These are known
cognitive architecture identifies the technology as the three “V’s” of big data. Many companies have
infrastructure and architecture as well as executive simply not invested in an architecture that enables
support and proper investment that enables this modern view of data management.
and supports data science practices, predictive
analytics, and modelling processes.

A Simple Framework for Building Predictive Models | 13

5.2 Basic Model Architecture empirical and statistical measures of success, and also
more subjective measures of success.
The picture below shows a high-level data architecture
that builds in predictive models and analytics as Some principal success metrics or indicators of model
a foundational element of the business systems. success are described below.
There are three layers: core data, which stores all
the essential data for the corporation; models/ 6.1 Statistical Uplift
analytics, which performs the predictive aspects;
and transactions, which is everything that touches 1) Uplift from your model
customers or end-users. Dev, QA, and Production • Compare predictive model performance against
environments are important to consider across random results with lift charts and decile tables.
these three basic layers.
- Using lift charts, hold out samples, and decile
A rough mapping back to the model process framework tables are a basic way to look at predictive
as described earlier is shown as well (Plan, Build, model performance. Typically models are
Implement). It is important to consider how to implement validated against a hold out group. This
model inputs and outputs in a comprehensive manner, group contains data that have not been
ideally without silos that would preclude certain kinds included in the development of the model.
of data access or usage.It is important to see the model For example, in a target lead model, you can
in this architecture as a growing element. The model look at how much more successful the model
may be turned on or off, upgraded, or replaced over is likely to be than if no predictive model
time, and the same basic architecture can still power was used to target leads.
the business moving forward.
• Evaluate the validity of your discovery with target
shuffling.
6. How Do You Know Your Model is
Successful? This method is particularly useful for identifying false
positives, or when two events or variables occurring
“Not everything that can be counted counts, and not
together are perceived to have a cause-and-effect
everything that counts can be counted.”
relationship, as opposed to a coincidental one.
To an important extent, of course, the determination This may happen because the more variables you
of what is considered a good model depends on the have, the more things may seem to have a causal
particular interests of the organization and is specified relationship and you may “oversearch.”
as the business success criterion. Generally, one sees

B asic M odel A rchitecture

A Simple Framework for Building Predictive Models | 14

• Test predictive model consistency using boot-
strap sampling.
This method tests a model’s performance on certain
subsets of data over and over again to provide
an estimate of accuracy, not just for statistical
significance, but also for operational significance.
2) If the purpose of the model is to provide highly
accurate predictions or decisions to be used by
the business, empirical measures of accuracy
will be used, such as confidence levels or other
statistical quantities.

6.2 Business Success Measures

“Success means going from failure to failure with
enthusiasm.”
Business success measures are important to keep in
mind. Often, executives have a good feel for what is
helping to drive the business. An analyst might build
an elegant model that works and saves the business
$80,000 a year, but that is really about the salary of the • Implementation efficiency: The ease of use,
analyst, so is there really an overall benefit? immediacy, and practicality of model
1)
Is the model intended to lower costs, is this implementation is important. If it is costly to deploy
really helping to do that? If the model is supposed a smart model, that could be a downside.
to enhance revenue or cross-sell, can this be • Time-to-capability (or value): The best idea in the
checked independently? Often, other effects have world has no real business value unless it hits
to be disentangled from the model performance the street and people are using the results.
itself. Spending years on a huge abstract idea may be
2)
If interpretation of the business is of most less valuable than quick low-hanging fruit.
interest, accuracy measures will not be used; • End-user satisfaction: If internal users, sponsors,
instead, subjective measures of what provides or end customers are not happy with what they
maximum insight may be most desirable. experience, this can be an uphill battle, regardless
3)
Some projects may use a combination of both of how good the model is.
empirical and subjective measures so that the
most accurate model is not selected if a less
accurate but more transparent model with nearly
the same accuracy is available.
4) The project nature of the exercise carries its own
success measures as well.

A Simple Framework for Building Predictive Models | 15

7. Appendix A: References

1. http://www.theanalysisfactor.com/7-guidelines-model-building/

2. https://en.wikipedia.org/wiki/Predictive_modelling

3. https://the-modelling-agency.com/model-development/

4. http://math.gmu.edu/~rgoldin/Articles/StatisticalModelsBookReview.pdf

5. Statistical Models: Theory and Practice (Revised Edition). By David A. Freedman.

Cambridge University Press, 2009.

6. https://www3.nd.edu/~steve/Rcourse/Lecture7v1.pdf

7. http://cdn.oreillystatic.com/en/assets/1/event/85/Best%20Practices%20for%20
Building%20and%20Deploying%20Predictive%20Models%20over%20Big%20
Data%20Presentation.pdf

8. http://www.jerrydallal.com/lhsp/LHSP.HTM

9. http://www.gartner.com/it-glossary/predictive-modelling/

10. http://www.analyticbridge.com/profiles/blogs/the-8-worst-predictive-modelling-
techniques

11. https://www.analyticsvidhya.com/blog/2015/09/perfect-build-predictive-model-10-
minutes/

12. http://www.predictiveanalyticsworld.com/businessapplications.php

13. http://www.predictionimpact.com/predictive-analytics-training.html

14. http://www.datamine.com/site/datamine/files/White%20Papers/predictive_
modelling_process_whitepaper.pdf

15. http://www.plottingsuccess.com/3-predictive-model-accuracy-tests-0114/

16. https://www.cleverism.com/predictive-analytics-forecast-future/

17. http://www.dummies.com/how-to/content/how-to-choose-an-algorithm-for-a-
predictive-analys.html

18. http://www.ibm.com/developerworks/library/ba-predictive-analytics2/

19. http://www.kdnuggets.com/

A Simple Framework for Building Predictive Models | 16

8. About the Author
Dr. Hal Kalechofsky is passionate about using technology innovation to solve new business
problems. With more than 20 years of high-tech experience, he has managed data science
initiatives and has built analytical computing systems and
processes at eBay, CERN and Wells Fargo Bank. As co-Founder
of Appiom, Inc., Kalechofsky invented new home networking
techniques for parental internet applications. Kalechofsky has
led service and business innovation teams at Cisco Systems,
eBay, and Coremetrics (acquired by IBM), and has consulted
and designed/deployed innovative technology solutions at
several Fortune 500 companies. Dr. Hal Kalechofsky

Dr. Kalechofsky earned a Ph.D. in high energy physics from the University of Pittsburgh and a
Bachelor of Arts in philosophy from Tufts University.

LinkedIn Profile:

www.linkedin.com/in/halkalechofsky

A Simple Framework for Building Predictive Models | 17

Where strategy meets execution