Ten Common Analytic Mistakes

Ten Common Analytic Mistakes
CHAPTER 19
1. Optimizing around the Wrong Metric
Metrics exist for just about anything in an organization and most
probably are collected for a good reason. Be sure the metric you want to
optimize will achieve not just your goals, but also your customers’ goals.
If airlines optimize around on-time departure instead of on-time arrival, an
airplane that pulls away from the gate and sits on the tarmac is a metric
success even though the customers feel the experience is disappointing as
they arrive at their destination an hour late. If you optimize around the
number of calls answered in one hour at a call center, you are placing
quantity over quality. While customers generally want to get resolution
quickly, are their issues being properly addressed?
Be sure your metrics are meaningful to your customer and that optimizing
those metrics makes for a better experience.
2. Relying Too Much on Behavioral or Attitudinal
Data
Mining customer transactions can reveal a lot of patterns in

things like what products customers purchase together or the
average time between purchases. But this behavioral data doesn’t
necessarily help you understand the attitudes and motivations
behind why customers purchase things together. This attitudinal
data can more easily be collected using surveys or other methods
of asking customers.
3. Not Having a Large Enough Sample Size
If you’re looking to detect small differences in metrics, like
conversion rates or customer attitudes, and you’re measuring a
sample of customers or data, be sure your sample size is large
enough to detect that difference. Use the sample size tables in this
book or consult a statistician to know what sample size you’ll
need ahead of time.
A lot of cost and effort are wasted on looking for very small
differences in customer attitudes, such as satisfaction, perception
of usability, or likelihood to recommend after making very small
changes to products or websites with too small of a sample size.
4. Eyeballing Data and Patterns
I call it “eyeballing statistics.” It’s the tendency to think you

can detect patterns from data by examining it without any
statistics. For very large patterns, you can see these easily without
any computations, but these sorts of obvious patterns rarely show
up. To minimize the chance that you’re being fooled by
randomness in data, use statistics and mathematical computations
to differentiate the news from the noise.
5. Confusing Statistical Significance with Practical
Significance
With a large sample size, you’ll be able to detect very small differences
and patterns that are statistically significant. Statistical significance just
means that the pattern or difference is not due to random noise in your data.
But that doesn’t mean that what’s detected will have much practical
importance. Analytics programs will flag different patterns and differences,
but you need to determine if a 1% difference in conversion rates results will
have a major or negligible impact. This depends on the context but means
you’ll need to exercise judgment and not blindly follow the software. Don’t
immediately think every statistically significant result is meaningful. Think
through the business implications of the result carefully. See the appendix
for more of a discussion on statistical versus practical significance.
6. Not Having an Interdisciplinary Team
If you have a stats PhD crunching numbers in your company

basement, it may generate the right insights; but if sales,
marketing, service, or product teams aren’t involved, it’s going to
be difficult to get buy-in and implement the insights. Get the right
people and teams involved in your initiative early and look to
have complementary skills, including mathematical, software,
business, marketing, and product experience .
7. Not Cleaning Your Data First
Garbage In, Garbage Out (GIGO) is a common phrase data

junkies like to use to explain that data that has problems before
analysis will have problems after analysis. This can be anything
from mismatched data pulled from databases (customer names
don’t match transactions) or missing values. If the data is bad
going in, you’ll have bad insights coming out. Before running any
analysis, do a quality check on your data by selecting a sample of
data and auditing it for quality. Corroborate it with other sources
to verify its accuracy.
8. Improperly Formatted Data
When you analyze your data, at least half of the effort is spent
formatting the data so your software can properly analyze it. This
often involves disaggregating and getting customer transactions
or survey data in rows and columns.
Skimping on proper formatting usually means a lot of rework
later, so be sure your data is formatted properly — and early.
9. Not Having Clear Research Questions to Answer
Sometimes it’s fine to have a fishing expedition and examine

patterns in data. But don’t stop with the fishing expedition; use
what you find to form hypotheses about customer behavior and
look to confirm, refine, or reject these hypotheses with additional
data.
10. Waiting for Perfect Data
Every data set tends to have some problem of some sort. Some
are minor, like a few missing fields; others are major, with lots of
missing fields and mismatched data. For survey data, there
always seems to be a concern about how a question was asked
and to whom it was asked. That said, expect some imperfection in
all your datasets and surveys. But don’t let it stop you from
working with what you have. Just be cautious about your
interpretation.
THANK YOU
FOR
LISTENING
GOD BLESS!

Ten Common Analytic Mistakes

Uploaded by

Copyright:

Available Formats

Ten Common Analytic Mistakes

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Ten Common Analytic Mistakes

Uploaded by

Copyright:

Available Formats

Ten Common Analytic Mistakes

Mining customer transactions can reveal a lot of patterns in

I call it “eyeballing statistics.” It’s the tendency to think you

If you have a stats PhD crunching numbers in your company

Garbage In, Garbage Out (GIGO) is a common phrase data

Sometimes it’s fine to have a fishing expedition and examine

You might also like