The Digital

Experimentation Playbook
Everything you need to know to gain value from experimentation

1
CONTENTS
01 Welcome to The Digital
Experimentation Playbook
p3

02 Turn Your Ideas into

Experiments
p5

03 Build a Winning Experiment p7

04 The Vital Role of Sharing

Results
p8

05 Experimentation in Action p10

2
01 Welcome to The Digital
Experimentation Playbook
The Digital Experimentation Playbook provides the practical first
steps to help you get value from digital experimentation. The
playbook will guide you through the core processes, providing you
with five downloadable worksheets and templates to help you build
quality experiments quickly, share results with confidence and
implement experimentation across teams.

Digital experimentation includes A/B testing, multivariate testing, multi-

armed bandit testing, user research, analytics, and personalization in
order to provide the best possible customer experience across all digital
touch points.

3
 Digital Experimentation Benefits
Digital experimentation allows you to A/B/n test across the entire customer
journey to meet your overall business goals, including:

To optimize top-line growth you can test several checkout funnels to see
which drives customer purchases.

To reduce customer churn you can test a variety of site navigation maps to
understand the best actions for a new customer.

To deliver world class customer experiences you can test several digital
products to know which drives customer engagement.

Most importantly, digital experimentation allows you to make data-driven

decisions to meet overall business goals.

The Digital Experimentation Playbook will help you create quality

experiments from start to finish. It will take you through the best practices for
experimentation ideation, creating winning experiments and results sharing.

“Experimentation has been instrumental to our growth.”

Rémy Aldasoro / Associate product director / HelloFresh

4
02 Turn Your Ideas
into Experiments
The digital experimentation process has five key components. Follow
these steps to create quality, data-driven experiments quickly and
ultimately accelerate the achievement of your overall business goals.

Research Collect relevant customer data, both qualitative data

(customer surveys and reviews) and quantitative data (web
analytics and conversations).

Ideate Analyze the customer data collected to identify the problem

and where improvements need to be made. What problem are
you trying to solve for the customer? Determine your key goal
and top metric for this experiment.

Refine Create a problem-centric hypothesis. State what you believe

needs to change, what the outcome will be and why.

Plan & Run Develop an experimentation roadmap that includes key

metrics, duration and target audience, and roll out the
experiment to a small subset of users.

Review Analyze the experiments’ results to see if the outcome has

proven or disproven your hypothesis, share the results internally
5 and define your next steps with data-driven decisions.
 How to create an experiment hypothesis
An experiment hypothesis is a bold statement to predict what you think the
outcome of your experiment will be, not an open-ended question which you
hope the experiment will answer. Above all, an experiment hypothesis is
there to be tested. It can be proven right or wrong.

An experiment hypothesis An experiment hypothesis

looks like: example:

“ If ,

The Result:
The predicted outcome of your If we set the algorithm for featured products
experiment. Ideally your overall on our homepage to display products from
business goal i.e. reduced customer recent catagories the user has visited,
churn or increased conversations.

then

The Variable:
An element which has been then the percentage of users that click
identified as a problem. It can be on the featured products and percentage
modified, added or removed to of users that added a product to cart will
improve the customer experience. increase.

.”

The Rationale:
Demonstrate why you have chosen Customer surveys showed users find
this variable and that your prediction the featured products on the homepage
is built on customer data. irrelevant. 9 out of 11 state they never found
relevant products.

6
03 Build a Winning
Experiment
Now you know the digital experimentation process it’s time to
start building a winning experiment. This section of The Digital
Experimentation Playbook is broken into three key stages with
practical steps and editable worksheets to help you map out your
ideas and create a clear, solid plan to follow.

Put ‘pen to paper’ and bring your digital

experiment ideas to life by filling out the
Experimentation Ideation Framework. This
template will help you ideate those ideas
from start to finish, understand what data
you need to collect and build your experiment
hypothesis. Create your first experiment ideas
Experimentation
by downloading Optimizely’s Experimentation
Ideation Framework
Ideation Framework.

Prioritize your experiment ideas by using our

Prioritization Index. The Prioritization Index
helps you to sort ideas by comparing the
Business Impact (higher conversion rates,
increased revenue or reduced internal costs)
to the Business Effort (technology and teams
needed). Prioritize which experiments you
Experimentation
Prioritization Index will run first by downloading Optimizely’s
Experimentation Prioritization Index.

Now that you’ve prioritized which experiments

you will run first, you need to refine your idea
by creating an experimentation roadmap,
including the top metric you will use to
measure success and the audience which
will take part in the experiment. Plan your first
experiments by downloading Optimizely’s
Experimentation
Roadmap Worksheet Experimentation Roadmap Worksheet.
7
04 The Vital Role of
Sharing Results
When creating a culture of experimentation, securing executive buy-
in is vital. One way to do this is by consistently sharing experiment
results with the whole company.

Optimizely’s customers have proven that organizations which share

experiment results more often are more efficient, and more likely to
achieve business results.

Let’s imagine a situation where you ran a winning experiment which

achieved a high ROI and brilliant customer success. That’s great news! Your
experiment has accelerated you towards your targets. But, you have not
shared your results.

If you do not share your wins with the rest of the organization, how can
they see the benefits of experimentation? More importantly, if you do not
share your wins with executive stakeholders, how can they see the ROI and
customer growth from your experiment?

Sharing results builds a culture of experimentation and secures executive

buy-in. Communicate why you created an experiment hypothesis, how you
defined your key metrics and audiences, and what impact it had on the
business.

“Having data to back up our

ideas gives us a voice to surface
our views up to a higher-up
level. Being able to say that
you improved sign ups by ‘X’
percentage, because of what you
did, that is a powerful thing.”

Team Lead / Growth &

Experimentation / Atlassian

8
 How to share vital results
Present to executive stakeholders and demonstrate your digital
experimentation findings to them, including your experimentation
objectives and goals, results overview and what data-driven
decisions have been made.

Share regular emails with the whole organization outlining your

experiment and your results. Make use of the organization’s
internal newsletters to deliver information and learnings across
departments.

Host ‘Lunch and Learn’ presentations for the whole organization.

Fill your colleagues lunch hours with informal presentations and
discussions on your experiments and the outcomes.

Results Sharing
Presentation Template

Build your first presentation by downloading

Optimizely’s Results Sharing Presentation
Template.

9
 05 Experimentation
in Action
The Digital Experimentation Playbook shares a process for
experimentation. Now it is over to you! You may have already filled
your Experimentation Ideation Framework to the brim with ideas. You
may also be looking for some inspiration to start. Luckily we’re experts
in this area. Here are a few experimentation ideas to kickstart your
brainstorming.

Media Travel Retail B2B

Inform and delight readers and Whisk users away with Help users shop ‘til they Inform and help web visitors
viewers with unique experiences an outstanding customer drop and deliver an in-store with easy-to-use site
to reach key metrics and impress experience, converting experience online with navigation, relevant product
advertisers, including monthly them from daydreamers to personalized, well-optimized recommendations and clear
readership, video engagement, holidaymakers in a few insightful product recommendations and call-to-actions to generate leads
return visits, session duration and well-optimized clicks which seamless checkout experiences more efficiently and effectively.
and ad viewability. enable your booking conversions to increase your revenue per
to take off. customer.

Think outside the box. Do not fly blind. Make confident decisions. Hone in on Results.
Experiment on: Experiment on: Experiment on: Experiment on:

Replacing ‘trending articles’ Pre-populating the search Presenting users with products Highlighting popular plans,
with ‘trending videos’ on the landing page from fields from recent categories they have packages and features on
homepage visited pricing page

Introducing an article’s Presenting users with Adding videos of the models Increasing visibility of products
read time at the beginning of personalized packages on the modelling clothes to product and solutions on the navigation
each article website homepage pictures bar

Presenting users with relevant Simplifying the step-by-step Increasing visibility of customer Reducing lead forms to
content in the sidebar vs the design for bookings reviews and ratings for each the minimum number of
footnote product components

media D O W N LOA D travel D O W N LOA D retail D O W N LOA D b2b D O W N LOA D

MORE IDEAS MORE IDEAS MORE IDEAS MORE IDEAS

10
CONCLUSION Now it’s your turn
The Digital Experimentation Playbook has taken you through the key steps
to create quality experiments quickly, experiments which put the customer
at the center and helped you achieve your overall business goals. Now, it is
your turn to put these steps into action.

If you need inspiration for your experimentation program, check out

these testing ideas in our Experimentation in Action overview for different
industries.

Put those ideas onto paper by filling in the Experimentation Ideation

Framework and create a winning hypothesis. Prioritize those ideas with
the Experimentation Prioritization Index and develop your primary idea on
the Experimentation Roadmap Worksheet. Once you have carried out the
experiment, analyze the results and share the key findings using our
Results Sharing Presentation Template with not only the executive team, but
the whole organization.

Most importantly, repeat.

As you now know, continuous experimentation helps organizations achieve

their business goals.

Want some more information on digital experimentation?

Talk to us today to find out how we can help you move faster, build quality
experiments and drive substantial growth for your business.

Optimizely is here to help you at every step of the way.

We look forward to hearing from you soon.
“The differentiator for
us is ensuring that we The Optimizely Team.
experiment our way
to a better customer
experience.”
Head / Digital Transformation /
Sky

11