Creative Testing Playbook May2020
Creative Testing Playbook May2020
1
Smartly.io: Creative Testing Playbook
Intro-
duction
At Smartly.io, we love exploring new ways to craft
storytelling optimized for mobile consumption, and
we consider technology as the enabler of our creative
practice. As fast movers, our philosophy is to learn,
test, and iterate continuously in an agile manner.
2
Smartly.io: Creative Testing Playbook
Table
Of Contents
09 What To Test?
31 Summary
3
Smartly.io: Creative Testing Playbook
What Is
What Is
Creative
Creative
Testing?
Testing?
Think of creative testing as going to an art gallery:
the art itself and the artist’s execution of the idea will
dictate the experience - this is what we call the creative
concept and its execution.
4
Smartly.io: Creative Testing Playbook
Why Test
Why Test
Creative?
Creative?
5
Smartly.io: Creative Testing Playbook
CON CE P T A C O N C E PT B
11:09 11:09
6
?
Smartly.io: Creative Testing Playbook
CPA
• Are these the most critical creative elements to test? (aka am I testing the right
things?)
Conclusion
Conclusion
Before coming out with a testing plan, you need a testing strategy and a
solid methodology to systematically test and iterate the things that have
the highest possible impact on business outcomes.
7
Smartly.io: Creative Testing Playbook
8
Smartly.io: Creative Testing Playbook
What To
What To
Test?
Test?
9
Smartly.io: Creative Testing Playbook
Start by analyzing your overall creative approach by funnel stage and business
line. Such analysis aims to resurface gaps and challenges in the current approach.
$ 9:16
?
“ 4:5
square
10
Smartly.io: Creative Testing Playbook
Types of creatives
used
Spend distribution
Creative refresh
interval
Individual creatives
or concepts
Branding / DR
creatives ratio
Messaging
Audience
customization
Culture and language
customization
You should also have a good understanding of your current metrics for
each funnel step and what you want to achieve. Once you have gone
through this process, you’ll be able to understand the challenges in your
current set up.
11
Smartly.io: Creative Testing Playbook
Hierarchy of Creative
Challenges
The next step is to take the challenges you have learned from your
Creative Analysis Gap and organize them by prioritization.
3 Test concepts against each other to find what speaks to the chosen
audience
5 Plan on testing the most critical large-scale questions with lift tests
and smaller iterations of variations directionally in the auction
6 Find the optimal cycle for refreshing concepts and restart the process
12
Smartly.io: Creative Testing Playbook
Hierarchy of
Hierarchy of Creative
Creative
Opportunities
Opportunities
How often
Campaign level,
should I refresh somewhat large
my creatives
in evergreen strategice questions Should I use
campaigns? that can be tested with videos or static
creatives or both?
a formal ad study.
13
Smartly.io: Creative Testing Playbook
A creative concept is the approach you take to tackle your business goals:
For example, you want to increase app installs for your business and
want to run a campaign for it. The creative concept will dictate the
messaging and the visual execution behind that campaign.
CR EATI VE OPTIMIZATION L E VE RS
You could have many different variations of a single creative concept - each
performing differently based on placements, formats, or audiences targeting.
You can test the variations using various creative optimization levers.
IT ERATI ON S
14
Smartly.io: Creative Testing Playbook
Generating
Generating
Creative Ideas
Creative Ideas
To help find ideas for creative tests that will address the gaps that you
have identified, tap into as many sources as you can. The more ideas
you come up with, the better - you can always prioritize tests based on
their potential business impact.
• Motivations
• Behavioral
Trends
• Blockers
TAP INTO
MULTIPLE
SOURCES TO
FIND INSIGHTS
• Competition
• Industry
Trends
• Intuition
• Imagination
15
Smartly.io: Creative Testing Playbook
16
Smartly.io: Creative Testing Playbook
...Mapping out data insights from past campaigns, creative tests and
industry trends.
Use previous results to inform the design of new creatives. Is your
audience responding better to direct messaging? Are your lifestyle images
performing better than product shots? Use Smartly.io Pivot Tables to see
what messages have resonated well in the past. Are there any learnings
you could use to fuel future creative planning?
For example: if using influencers on IG Stories helped connect with
Generation Z, you might want to expand on this approach - e.g. appointing
best performing influencers as brand ambassadors.
Identify industry trends that could inspire you to come up with some
interesting hypotheses to validate. Facebook Ad Library is a great tool
offering a comprehensive, searchable collection of all ads currently running
from across all Facebook Products.
For example: if your competitors use customer reviews as ‘social proof’ in
their ads, you could test if this same approach works for your audience
17
Smartly.io: Creative Testing Playbook
How To
How To Test
Test
Creative?
Creative?
18
Smartly.io: Creative Testing Playbook
TRY NOW
19
Smartly.io: Creative Testing Playbook
Testing Methodology
Testing Methodology
Overview
Overview
ST E P 1
A VS
B VS
C
ST E P 2
Iteration 1
A C Iteration 1
Iteration 2 Iteration 2
winner
Iteration 3 Iteration 3
ST E P 3
Learn
20
Smartly.io: Creative Testing Playbook
Smartly.io Creative
Testing Workflow
Convenience-driven
21
Smartly.io: Creative Testing Playbook
11:09
Winning
CPA
22
Smartly.io: Creative Testing Playbook
3
THE LEARNINGS
23
Smartly.io: Creative Testing Playbook
Overall business
impact - Big Lift Test for differences in incrementality
decisions
24
Smartly.io: Creative Testing Playbook
F O RMAL AD STUDI E S
There are two types of Ad Studies you can apply. Use this methodology
when you want to take important decisions that require a ‘scientific’
method.
Lift Test: Execute this type of tests when you have a complex set up or
want to achieve major changes in your creative approach.
For example:
• Testing image vs. image + video
• Creatives that tap into different audience segments that may engage
you at different rates regardless of the ads
Split Test: Use this methodology when you want to inform future creative
production processes, or a default (evergreen) approach for creatives.
For example:
• Localized vs. non-localized creatives
• User generated content vs studio produced videos
both.
2 calculate
There is no need to manually
statistical significance. This
25
Smartly.io: Creative Testing Playbook
Important Considerations
Set up your Ad Study with the Expect a hit in performance.
right budget. Whenever you’re running a
As a general rule, we recommend split test, the audience will be
using 1,000 conversions as the split, likely resulting in a dip in
minimum number needed for performance KPIs. Therefore, any
a 2-cell split test. Multiply this comparisons should be made
time your average CPA for the between the study cells and not
conversion. Example: If CPA Sign against e.g. historical performance.
Up is $10, then 1,000 x $10 =
$10,000. That is how you could Make sure that your events are
calculate the recommended firing properly.
budget needed for a test. All too often, advertisers start
running ad studies and notice
during the study that their pixel
events are not firing, which naturally
harms the study, as we are unable
to measure the results reliably.
$
$
$
26
BU Y NOW
Smartly.io: Creative Testing Playbook
27
Smartly.io: Creative Testing Playbook
… You want to take small decisions or decisions that impact only the
choice of creatives in question.
… You are not ready to sacrifice a lot of the campaign performance during
the test.
Directional Testing: You can always let the Facebook algorithm choose
by running different creatives in parallel and compare their performance
using Smartly.io’s Pivot Tables.
28
Smartly.io: Creative Testing Playbook
• Launch all ads tested against each other at the same time to avoid
learnings or ad fatigue from creating the difference.
• You may use Smartly.io’s feature Predictive Creative Optimization to get
a more even distribution of spend between ads and ensure each ad gets
a fair chance to perform.
• How long to test:
→ Match test duration and budget with test importance
→ A good rule of thumb is to let each and collect at least 5–20
conversions depending on overall volume and CPA
29
Smartly.io: Creative Testing Playbook
Summary
Summary
30
Smartly.io: Creative Testing Playbook
31
Smartly.io: Creative Testing Playbook
Bonus Section
H OW TO STRUCTURE TE ST CAMPAIG N S?
You can simplify things for yourselves and drive more impact by using the
following approach: constant and consistent testing is essential, so having a
separate test campaign can help you with your testing initiatives.
Our recommended campaign structure is in the chart, on the following page.
32
Smartly.io: Creative Testing Playbook
How To
How To Structure
Structure
Test Campaigns?
Test Campaigns?
T E ST CA M PA I G N S
C VS
C VS
C
Iteration 1 Iteration 2 Iteration 3
E V E R G R E E N CA M PA I G N S
A B C C
Iteration 3
Best performers from Test Campaigns are moved to Evergreen – others are paused
33
Smartly.io: Creative Testing Playbook
WHEN TO TEST?
• Most successful brands test continuously - have a test & learn agenda
with your customers and seek to question the status quo and validate new
hypothesis.
Continuous
Creative Create and test
Testing creative concepts
formally with A/B
tests.
Learning
& Iteration
Cycle
Gather learnings from
the informal tests, and
Define other test
apply to similar creatives.
factors on the
Run the top performing
winning concept and
creatives until refreshing
start systematic
is timely.
observational testing
to gather more
insights.
34
Smartly.io: Creative Testing Playbook
• Testing creative formats – especially the mix between static and video
35
Smartly.io: Creative Testing Playbook
Why Partner
With Smartly.io?
Smartly.io automates every step of your social advertising to unlock
greater performance and creativity. We combine creative automation, ad
operations, and outstanding customer service to help 600+ brands scale
their results – not headcount.
INNOVATE FASTER
Access what‘s new before the competition.
36
Smartly.io: Creative Testing Playbook
AMSTERDAM M A DR I D
Mindspace Amsterdam Dam WeWork
Nieuwezijds Voorburgwal 162 Paseo de la Castellana 77
1012 SJ, Amsterdam 28046, Madrid
AUST IN N E W YO R K
WeWork 440 Park Avenue South -
316 W 12th St - 6th Floor 14th Floor
Austin, TX 78701 New York, 10016
BERL IN PA R I S
Schiffbauerdamm 12 Spaces Bonne Nouvelle
10117, Berlin 17-21 Rue Saint-Fiacre
75002, Paris
BUEN OS AIRE S
Torre Bellini Esmeralda SA N F R A N C I S C O
950 Capital Federal 580 Market Street
B, C10007, Buenos Aires Suite 200
San Francisco, CA 94104
DUBAI
Astro Labs Dubai S I N GA PO R E
Parkside Retail Level Cluster R One Raffles Place #34-03
JLT, Dubai Singapore, 048616
DUBL IN SY D N E Y
Dogpatch Labs 175 Pitt St - 15th Floor
The Chq Building, Custom House Quay, NSW, 2000
North Dock, Dublin
SÃO PAU LO
HELSI N K I Rua Prof. Atílio Innocenti
Postitalo 165 - Vila Nova Conceição
Elielinaukio 2G - 4th Floor São Paulo, SP 04538 - 000
00100, Helsinki
TO KYO
LON DON Business-Airport
Business Centre 3F SHIBUYA SOLASTA
2 Stephen Street 1-21-1 Dogenzaka Shibuya-ku
London, W1T 1AN Tokyo, 150-0043
37
Smartly.io: Creative Testing Playbook
Ready to Start
Creating Smartly?
Creating Smartly?
Click here for a demo of our platform or speak
with your Customer Success Manager.
38